Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 43 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,49 @@ func main() {
- **Returns an AST** you can walk or render, with precise, aggregated errors.
- **Concurrency-safe** parsers.

## How it works

Every parser is gated by the same allow-list of fields. Untrusted query-string
parameters go in; validated, structured values come out — an AST for filters,
typed lists for fields and sorting — so nothing outside your allow-list ever
reaches your data layer.

```mermaid
flowchart LR
Q["HTTP query string<br/>?fields=…&filter=…&sort=…"]

Q --> FP["FieldsParser"]
Q --> FI["FilterParser"]
Q --> SP["SortParser"]

AL(["Allow-list of fields"]) -. gates .-> FP
AL -. gates .-> FI
AL -. gates .-> SP

FP --> FN["FieldsNode<br/>[]string"]
FI --> AST["Filter AST<br/>(Node tree)"]
SP --> SN["SortNode<br/>[]SortFieldNode"]

FN --> APP["Your handler / SQL builder"]
AST --> APP
SN --> APP

APP --> DB[("Database")]
```

The filter parser is a hand-written recursive-descent parser: a `Lexer`
tokenizes the input, then the parser builds the AST while honoring operator
precedence (`OR` < `AND` < comparison) and the allow-list.

```mermaid
flowchart LR
IN["input string"] --> LEX["Lexer<br/>tokenize"]
LEX --> TOK["[]Token"]
TOK --> PAR["FilterParser<br/>recursive descent"]
PAR --> AST["AST root (Node)"]
PAR -. "collects every error" .-> ERR["errors.Join(…)"]
```

## Documentation

Full documentation lives in [`docs/`](docs/README.md):
Expand Down
32 changes: 32 additions & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,38 @@

Reference documentation for the **Query Filters Validator** (`github.com/slashdevops/qfv`).

## The three parsers at a glance

```mermaid
flowchart TB
subgraph inputs["Query-string parameters"]
fq["fields=first_name,email"]
flq["filter=age >= 18 AND status IN ('active')"]
sq["sort=created_at DESC"]
end

fq --> FieldsParser
flq --> FilterParser
sq --> SortParser

FieldsParser --> FN["FieldsNode<br/>validated []string"]
FilterParser --> AST["AST<br/>walkable Node tree"]
SortParser --> SN["SortNode<br/>[]SortFieldNode"]

allow(["Shared allow-list<br/>(supports dotted paths)"])
allow -. gates .-> FieldsParser
allow -. gates .-> FilterParser
allow -. gates .-> SortParser
```

| Parser | Input example | Output | Complexity |
| --- | --- | --- | --- |
| `FieldsParser` | `first_name, email` | `FieldsNode` | comma-split validation |
| `SortParser` | `first_name ASC, created_at DESC` | `SortNode` | comma-split + direction check |
| `FilterParser` | `age >= 18 AND status IN ('active')` | AST (`Node`) | full recursive-descent grammar |

## Guides

| Guide | What it covers |
| --- | --- |
| [Getting Started](getting-started.md) | Install, update, and a complete runnable example of the three parsers |
Expand Down
46 changes: 46 additions & 0 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,30 @@ p.Parse("name LIKE '%x%'") // error: operator "LIKE" is not
The two options are **additive** — groups expand to their operators and merge
with any individually listed operators.

```mermaid
flowchart LR
G["WithAllowedOperatorGroups(GroupComparison, GroupLogical)"] --> EXP["expand each group<br/>to its operators"]
O["WithAllowedOperators(OpIn)"] --> SET
EXP --> SET(["allowed-operator set (union)"])
SET --> CHK{"operator seen<br/>while parsing?"}
CHK -->|"in set"| OK["accept"]
CHK -->|"not in set"| ERR["error:<br/>operator is not allowed"]
```

### How gating decides

Every operator the parser encounters is checked against the allowed set. A
**nil** set (neither option supplied) is the default and permits everything.

```mermaid
flowchart TD
T["operator token in input"] --> C{"allow-list<br/>configured?"}
C -->|"no (nil) — default"| OK["allowed<br/>(every operator permitted)"]
C -->|"yes"| M{"operator in set?"}
M -->|"yes"| OK
M -->|"no"| ERR["QFVFilterError:<br/>operator is not allowed"]
```

### Operators

| Operator | Matches |
Expand Down Expand Up @@ -59,6 +83,18 @@ with any individually listed operators.

`OperatorGroup.Operators()` returns the operators a group expands to.

```mermaid
flowchart LR
GroupLogical --> OpAnd & OpOr & OpNot
GroupComparison --> OpEqual & OpNotEqual & OpLessThan & OpLessThanOrEqual & OpGreaterThan & OpGreaterThanOrEqual
GroupPattern --> OpLike & OpILike & OpSimilarTo & OpRegexMatch
GroupMembership --> OpIn
GroupRange --> OpBetween
GroupNull --> OpIsNull
GroupBoolean --> OpBooleanTest
GroupDistinct --> OpIsDistinctFrom
```

### Negation semantics

A negated **predicate** is governed by its base operator, not by `OpNot`:
Expand All @@ -68,6 +104,16 @@ A negated **predicate** is governed by its base operator, not by `OpNot`:
`OpIsDistinctFrom` (respectively) are allowed.
- `OpNot` governs only the **standalone** logical `NOT`, e.g. `NOT (age > 30)`.

```mermaid
flowchart TD
N["NOT appears in input"] --> K{"what follows?"}
K -->|"( expr ) or a comparison"| U["UnaryOperatorNode<br/>governed by OpNot"]
K -->|"IN / LIKE / BETWEEN /<br/>SIMILAR TO / DISTINCT FROM"| P["predicate node with IsNot=true<br/>governed by the base operator"]

U -. "allow-list check" .-> gnot{"OpNot allowed?"}
P -. "allow-list check" .-> gbase{"OpIn / OpLike / OpBetween /<br/>OpSimilarTo / OpIsDistinctFrom allowed?"}
```

So allowing `OpEqual` but not `OpNot` accepts `a = 1` but rejects `NOT (a = 1)`.

## Restricting sort directions
Expand Down
45 changes: 45 additions & 0 deletions docs/error-handling.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,51 @@ if err != nil {
- `*QFVSortError` — sort parser failures
- `*QFVFieldsError` — fields parser failures

```mermaid
classDiagram
class error {
<<interface>>
+Error() string
}
error <|.. QFVFilterError
error <|.. QFVSortError
error <|.. QFVFieldsError

class QFVFilterError {
+Field string
+Message string
}
class QFVSortError {
+Field string
+Message string
}
class QFVFieldsError {
+Field string
+Message string
}
```

## How the filter parser reports errors

Unlike the sort and fields parsers — which fail fast on the first problem — the
**filter** parser makes a single pass and keeps going after each error, so one
call can surface several problems at once. The collected errors are combined
with `errors.Join`.

```mermaid
flowchart TD
IN["FilterParser.Parse(input)"] --> LEX["lex input, then scan<br/>for ILLEGAL tokens"]
LEX --> EMPTY{"empty<br/>expression?"}
EMPTY -->|"yes"| E0["return single QFVFilterError"]
EMPTY -->|"no"| P["single parse pass<br/>(continues past each error)"]
P --> CONS{"entire input<br/>consumed?"}
CONS -->|"no"| ADD["add 'unexpected token' error"]
CONS -->|"yes"| Q
ADD --> Q{"len(errors) &gt; 0?"}
Q -->|"yes"| J["return errors.Join(…)<br/>of *QFVFilterError"]
Q -->|"no"| OK["return AST, nil"]
```

## Aggregated filter errors

`FilterParser.Parse` collects **every** problem in a single pass and returns
Expand Down
142 changes: 141 additions & 1 deletion docs/filtering.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,48 @@ The parser guarantees that:
- the syntax is valid and the **entire** input is consumed,
- the result is a well-formed AST.

## How the filter parser works

Parsing is a two-stage pipeline. The `Lexer` first turns the raw string into a
flat list of tokens; the recursive-descent parser then consumes those tokens to
build the AST, checking each field against the allow-list and each operator
against the (optional) operator allow-list as it goes.

```mermaid
flowchart LR
IN["first_name = 'John' AND age >= 18"] --> LEX["Lexer.Parse()"]

subgraph tokens["[]Token"]
direction LR
t1["IDENT<br/>first_name"] --- t2["=<br/>OPERATOR"] --- t3["STRING<br/>'John'"] --- t4["AND"] --- t5["IDENT<br/>age"] --- t6[">="] --- t7["INT<br/>18"]
end

LEX --> tokens
tokens --> PAR["parseExpression()"]
PAR --> AST["AST root (Node)"]
PAR -. "every problem collected" .-> ERR["errors.Join(…)"]
```

### Operator precedence

The parser encodes precedence as a ladder of methods. `OR` binds loosest and is
tried first; each level descends to the next tighter-binding one, and a
parenthesized group re-enters the ladder at the top.

```mermaid
flowchart TD
OR["parseLogicalOr()<br/>OR — lowest precedence"]
AND["parseLogicalAnd()<br/>AND"]
CMP["parseComparison()<br/>NOT · ( ) · field OP value · IS/IN/BETWEEN/LIKE/…"]
PRI["parsePrimary()<br/>string · int · float · bool — highest"]

OR --> AND --> CMP --> PRI
CMP -->|"( expr )"| OR
```

So `a = 1 OR b = 2 AND c = 3` parses as `a = 1 OR (b = 2 AND c = 3)` — `AND`
binds tighter than `OR`.

## Supported operators

| Category | Operators |
Expand Down Expand Up @@ -48,6 +90,25 @@ A dot is only valid **inside** an identifier, so numeric literals such as `3.14`
are unaffected. Unknown dotted fields are rejected by the allow-list like any
other field.

## The `IS` predicate family

`IS` is a single entry point into several predicates. After consuming `IS` (and
an optional `NOT`), the parser branches on the next keyword to build the right
node:

```mermaid
flowchart LR
IS(["field IS"]) --> NOT{"NOT?"}
NOT -->|"optional"| KW{"next keyword"}

KW -->|"NULL"| N["IsNullNode"]
KW -->|"TRUE / FALSE"| B["BooleanTestNode"]
KW -->|"UNKNOWN"| B
KW -->|"DISTINCT FROM value"| D["DistinctNode"]

N -. "shorthands" .- SH["field ISNULL / NOTNULL<br/>→ IsNullNode"]
```

## Worked examples

```go
Expand Down Expand Up @@ -103,7 +164,86 @@ other field.

## AST nodes

`Parse` returns a `qfv.Node`. Common concrete types:
`Parse` returns a `qfv.Node` — the root of a tree you can walk or render.
For example, the expression

```text
(first_name = 'John' OR first_name = 'Jane') AND age > 30
```

parses into this tree (precedence makes `AND` the root, with the parenthesized
`OR` on its left):

```mermaid
graph TD
AND["BinaryOperatorNode<br/>AND"]
AND --> GRP["GroupNode<br/>( … )"]
AND --> GT["BinaryOperatorNode<br/>&gt;"]

GRP --> OR["BinaryOperatorNode<br/>OR"]
OR --> EQ1["BinaryOperatorNode<br/>="]
OR --> EQ2["BinaryOperatorNode<br/>="]

EQ1 --> id1["IdentifierNode<br/>first_name"]
EQ1 --> lit1["LiteralNode<br/>'John'"]
EQ2 --> id2["IdentifierNode<br/>first_name"]
EQ2 --> lit2["LiteralNode<br/>'Jane'"]

GT --> id3["IdentifierNode<br/>age"]
GT --> lit3["LiteralNode<br/>30"]
```

Every node implements the `Node` interface, so you can type-switch on the
concrete types to walk or transform the tree:

```mermaid
classDiagram
class Node {
<<interface>>
+Type() NodeType
+String() string
+Pos() scanner.Position
}

Node <|.. IdentifierNode
Node <|.. LiteralNode
Node <|.. UnaryOperatorNode
Node <|.. BinaryOperatorNode
Node <|.. GroupNode
Node <|.. InNode
Node <|.. BetweenNode
Node <|.. IsNullNode
Node <|.. BooleanTestNode
Node <|.. DistinctNode
Node <|.. SimilarToNode
Node <|.. RegexMatchNode

class BinaryOperatorNode {
+Left Node
+Right Node
+Operator TokenType
}
class InNode {
+Field Node
+Values []Node
+IsNot bool
}
class BetweenNode {
+Field Node
+Lower Node
+Upper Node
+IsNot bool
+IsSymmetric bool
}
class RegexMatchNode {
+Field Node
+Pattern Node
+IsNot bool
+IsCaseInsensitive bool
}
```

Common concrete types:

| Node | Produced by |
| --- | --- |
Expand Down
Loading