diff --git a/README.md b/README.md index 1fc4461..7b4235d 100644 --- a/README.md +++ b/README.md @@ -66,6 +66,49 @@ func main() { - **Returns an AST** you can walk or render, with precise, aggregated errors. - **Concurrency-safe** parsers. +## How it works + +Every parser is gated by the same allow-list of fields. Untrusted query-string +parameters go in; validated, structured values come out — an AST for filters, +typed lists for fields and sorting — so nothing outside your allow-list ever +reaches your data layer. + +```mermaid +flowchart LR + Q["HTTP query string
?fields=…&filter=…&sort=…"] + + Q --> FP["FieldsParser"] + Q --> FI["FilterParser"] + Q --> SP["SortParser"] + + AL(["Allow-list of fields"]) -. gates .-> FP + AL -. gates .-> FI + AL -. gates .-> SP + + FP --> FN["FieldsNode
[]string"] + FI --> AST["Filter AST
(Node tree)"] + SP --> SN["SortNode
[]SortFieldNode"] + + FN --> APP["Your handler / SQL builder"] + AST --> APP + SN --> APP + + APP --> DB[("Database")] +``` + +The filter parser is a hand-written recursive-descent parser: a `Lexer` +tokenizes the input, then the parser builds the AST while honoring operator +precedence (`OR` < `AND` < comparison) and the allow-list. + +```mermaid +flowchart LR + IN["input string"] --> LEX["Lexer
tokenize"] + LEX --> TOK["[]Token"] + TOK --> PAR["FilterParser
recursive descent"] + PAR --> AST["AST root (Node)"] + PAR -. "collects every error" .-> ERR["errors.Join(…)"] +``` + ## Documentation Full documentation lives in [`docs/`](docs/README.md): diff --git a/docs/README.md b/docs/README.md index 51c5b9c..600c7c8 100644 --- a/docs/README.md +++ b/docs/README.md @@ -2,6 +2,38 @@ Reference documentation for the **Query Filters Validator** (`github.com/slashdevops/qfv`). +## The three parsers at a glance + +```mermaid +flowchart TB + subgraph inputs["Query-string parameters"] + fq["fields=first_name,email"] + flq["filter=age >= 18 AND status IN ('active')"] + sq["sort=created_at DESC"] + end + + fq --> FieldsParser + flq --> FilterParser + sq --> SortParser + + FieldsParser --> FN["FieldsNode
validated []string"] + FilterParser --> AST["AST
walkable Node tree"] + SortParser --> SN["SortNode
[]SortFieldNode"] + + allow(["Shared allow-list
(supports dotted paths)"]) + allow -. gates .-> FieldsParser + allow -. gates .-> FilterParser + allow -. gates .-> SortParser +``` + +| Parser | Input example | Output | Complexity | +| --- | --- | --- | --- | +| `FieldsParser` | `first_name, email` | `FieldsNode` | comma-split validation | +| `SortParser` | `first_name ASC, created_at DESC` | `SortNode` | comma-split + direction check | +| `FilterParser` | `age >= 18 AND status IN ('active')` | AST (`Node`) | full recursive-descent grammar | + +## Guides + | Guide | What it covers | | --- | --- | | [Getting Started](getting-started.md) | Install, update, and a complete runnable example of the three parsers | diff --git a/docs/configuration.md b/docs/configuration.md index b2e355a..d8601b0 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -24,6 +24,30 @@ p.Parse("name LIKE '%x%'") // error: operator "LIKE" is not The two options are **additive** — groups expand to their operators and merge with any individually listed operators. +```mermaid +flowchart LR + G["WithAllowedOperatorGroups(GroupComparison, GroupLogical)"] --> EXP["expand each group
to its operators"] + O["WithAllowedOperators(OpIn)"] --> SET + EXP --> SET(["allowed-operator set (union)"]) + SET --> CHK{"operator seen
while parsing?"} + CHK -->|"in set"| OK["accept"] + CHK -->|"not in set"| ERR["error:
operator is not allowed"] +``` + +### How gating decides + +Every operator the parser encounters is checked against the allowed set. A +**nil** set (neither option supplied) is the default and permits everything. + +```mermaid +flowchart TD + T["operator token in input"] --> C{"allow-list
configured?"} + C -->|"no (nil) — default"| OK["allowed
(every operator permitted)"] + C -->|"yes"| M{"operator in set?"} + M -->|"yes"| OK + M -->|"no"| ERR["QFVFilterError:
operator is not allowed"] +``` + ### Operators | Operator | Matches | @@ -59,6 +83,18 @@ with any individually listed operators. `OperatorGroup.Operators()` returns the operators a group expands to. +```mermaid +flowchart LR + GroupLogical --> OpAnd & OpOr & OpNot + GroupComparison --> OpEqual & OpNotEqual & OpLessThan & OpLessThanOrEqual & OpGreaterThan & OpGreaterThanOrEqual + GroupPattern --> OpLike & OpILike & OpSimilarTo & OpRegexMatch + GroupMembership --> OpIn + GroupRange --> OpBetween + GroupNull --> OpIsNull + GroupBoolean --> OpBooleanTest + GroupDistinct --> OpIsDistinctFrom +``` + ### Negation semantics A negated **predicate** is governed by its base operator, not by `OpNot`: @@ -68,6 +104,16 @@ A negated **predicate** is governed by its base operator, not by `OpNot`: `OpIsDistinctFrom` (respectively) are allowed. - `OpNot` governs only the **standalone** logical `NOT`, e.g. `NOT (age > 30)`. +```mermaid +flowchart TD + N["NOT appears in input"] --> K{"what follows?"} + K -->|"( expr ) or a comparison"| U["UnaryOperatorNode
governed by OpNot"] + K -->|"IN / LIKE / BETWEEN /
SIMILAR TO / DISTINCT FROM"| P["predicate node with IsNot=true
governed by the base operator"] + + U -. "allow-list check" .-> gnot{"OpNot allowed?"} + P -. "allow-list check" .-> gbase{"OpIn / OpLike / OpBetween /
OpSimilarTo / OpIsDistinctFrom allowed?"} +``` + So allowing `OpEqual` but not `OpNot` accepts `a = 1` but rejects `NOT (a = 1)`. ## Restricting sort directions diff --git a/docs/error-handling.md b/docs/error-handling.md index 261fee0..ab22c41 100644 --- a/docs/error-handling.md +++ b/docs/error-handling.md @@ -20,6 +20,51 @@ if err != nil { - `*QFVSortError` — sort parser failures - `*QFVFieldsError` — fields parser failures +```mermaid +classDiagram + class error { + <> + +Error() string + } + error <|.. QFVFilterError + error <|.. QFVSortError + error <|.. QFVFieldsError + + class QFVFilterError { + +Field string + +Message string + } + class QFVSortError { + +Field string + +Message string + } + class QFVFieldsError { + +Field string + +Message string + } +``` + +## How the filter parser reports errors + +Unlike the sort and fields parsers — which fail fast on the first problem — the +**filter** parser makes a single pass and keeps going after each error, so one +call can surface several problems at once. The collected errors are combined +with `errors.Join`. + +```mermaid +flowchart TD + IN["FilterParser.Parse(input)"] --> LEX["lex input, then scan
for ILLEGAL tokens"] + LEX --> EMPTY{"empty
expression?"} + EMPTY -->|"yes"| E0["return single QFVFilterError"] + EMPTY -->|"no"| P["single parse pass
(continues past each error)"] + P --> CONS{"entire input
consumed?"} + CONS -->|"no"| ADD["add 'unexpected token' error"] + CONS -->|"yes"| Q + ADD --> Q{"len(errors) > 0?"} + Q -->|"yes"| J["return errors.Join(…)
of *QFVFilterError"] + Q -->|"no"| OK["return AST, nil"] +``` + ## Aggregated filter errors `FilterParser.Parse` collects **every** problem in a single pass and returns diff --git a/docs/filtering.md b/docs/filtering.md index 97629d0..b8016d7 100644 --- a/docs/filtering.md +++ b/docs/filtering.md @@ -16,6 +16,48 @@ The parser guarantees that: - the syntax is valid and the **entire** input is consumed, - the result is a well-formed AST. +## How the filter parser works + +Parsing is a two-stage pipeline. The `Lexer` first turns the raw string into a +flat list of tokens; the recursive-descent parser then consumes those tokens to +build the AST, checking each field against the allow-list and each operator +against the (optional) operator allow-list as it goes. + +```mermaid +flowchart LR + IN["first_name = 'John' AND age >= 18"] --> LEX["Lexer.Parse()"] + + subgraph tokens["[]Token"] + direction LR + t1["IDENT
first_name"] --- t2["=
OPERATOR"] --- t3["STRING
'John'"] --- t4["AND"] --- t5["IDENT
age"] --- t6[">="] --- t7["INT
18"] + end + + LEX --> tokens + tokens --> PAR["parseExpression()"] + PAR --> AST["AST root (Node)"] + PAR -. "every problem collected" .-> ERR["errors.Join(…)"] +``` + +### Operator precedence + +The parser encodes precedence as a ladder of methods. `OR` binds loosest and is +tried first; each level descends to the next tighter-binding one, and a +parenthesized group re-enters the ladder at the top. + +```mermaid +flowchart TD + OR["parseLogicalOr()
OR — lowest precedence"] + AND["parseLogicalAnd()
AND"] + CMP["parseComparison()
NOT · ( ) · field OP value · IS/IN/BETWEEN/LIKE/…"] + PRI["parsePrimary()
string · int · float · bool — highest"] + + OR --> AND --> CMP --> PRI + CMP -->|"( expr )"| OR +``` + +So `a = 1 OR b = 2 AND c = 3` parses as `a = 1 OR (b = 2 AND c = 3)` — `AND` +binds tighter than `OR`. + ## Supported operators | Category | Operators | @@ -48,6 +90,25 @@ A dot is only valid **inside** an identifier, so numeric literals such as `3.14` are unaffected. Unknown dotted fields are rejected by the allow-list like any other field. +## The `IS` predicate family + +`IS` is a single entry point into several predicates. After consuming `IS` (and +an optional `NOT`), the parser branches on the next keyword to build the right +node: + +```mermaid +flowchart LR + IS(["field IS"]) --> NOT{"NOT?"} + NOT -->|"optional"| KW{"next keyword"} + + KW -->|"NULL"| N["IsNullNode"] + KW -->|"TRUE / FALSE"| B["BooleanTestNode"] + KW -->|"UNKNOWN"| B + KW -->|"DISTINCT FROM value"| D["DistinctNode"] + + N -. "shorthands" .- SH["field ISNULL / NOTNULL
→ IsNullNode"] +``` + ## Worked examples ```go @@ -103,7 +164,86 @@ other field. ## AST nodes -`Parse` returns a `qfv.Node`. Common concrete types: +`Parse` returns a `qfv.Node` — the root of a tree you can walk or render. +For example, the expression + +```text +(first_name = 'John' OR first_name = 'Jane') AND age > 30 +``` + +parses into this tree (precedence makes `AND` the root, with the parenthesized +`OR` on its left): + +```mermaid +graph TD + AND["BinaryOperatorNode
AND"] + AND --> GRP["GroupNode
( … )"] + AND --> GT["BinaryOperatorNode
>"] + + GRP --> OR["BinaryOperatorNode
OR"] + OR --> EQ1["BinaryOperatorNode
="] + OR --> EQ2["BinaryOperatorNode
="] + + EQ1 --> id1["IdentifierNode
first_name"] + EQ1 --> lit1["LiteralNode
'John'"] + EQ2 --> id2["IdentifierNode
first_name"] + EQ2 --> lit2["LiteralNode
'Jane'"] + + GT --> id3["IdentifierNode
age"] + GT --> lit3["LiteralNode
30"] +``` + +Every node implements the `Node` interface, so you can type-switch on the +concrete types to walk or transform the tree: + +```mermaid +classDiagram + class Node { + <> + +Type() NodeType + +String() string + +Pos() scanner.Position + } + + Node <|.. IdentifierNode + Node <|.. LiteralNode + Node <|.. UnaryOperatorNode + Node <|.. BinaryOperatorNode + Node <|.. GroupNode + Node <|.. InNode + Node <|.. BetweenNode + Node <|.. IsNullNode + Node <|.. BooleanTestNode + Node <|.. DistinctNode + Node <|.. SimilarToNode + Node <|.. RegexMatchNode + + class BinaryOperatorNode { + +Left Node + +Right Node + +Operator TokenType + } + class InNode { + +Field Node + +Values []Node + +IsNot bool + } + class BetweenNode { + +Field Node + +Lower Node + +Upper Node + +IsNot bool + +IsSymmetric bool + } + class RegexMatchNode { + +Field Node + +Pattern Node + +IsNot bool + +IsCaseInsensitive bool + } +``` + +Common concrete types: | Node | Produced by | | --- | --- | diff --git a/docs/getting-started.md b/docs/getting-started.md index f4fbbd1..d38eac5 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -35,6 +35,42 @@ invalid or malicious input reaching your database. All three parsers are **safe for concurrent use** — build them once and share them across goroutines. +## Where QFV fits in a request + +QFV sits between the untrusted query string and your data layer. Build the +parsers once at startup; run them per request to reject anything that isn't +explicitly allowed *before* you touch the database. + +```mermaid +sequenceDiagram + autonumber + participant C as Client + participant H as HTTP handler + participant P as QFV parsers + participant DB as Database + + C->>H: GET /users?fields=…&filter=…&sort=… + H->>P: Parse(fields), Parse(filter), Parse(sort) + alt any parameter invalid + P-->>H: error (unknown field / disallowed operator / bad syntax) + H-->>C: 400 Bad Request + else all valid + P-->>H: FieldsNode + AST + SortNode + H->>DB: build a safe query from the validated output + DB-->>H: rows + H-->>C: 200 OK + end +``` + +```mermaid +flowchart LR + boot["App start"] --> build["Build parsers once
NewFilterParser / NewSortParser / NewFieldsParser"] + build --> pool["Shared, concurrency-safe instances"] + pool -. reused per request .-> req1["Request 1"] + pool -. reused per request .-> req2["Request 2"] + pool -. reused per request .-> reqN["Request N"] +``` + ## Complete example ```go diff --git a/docs/migration.md b/docs/migration.md index 55cb8cc..061f570 100644 --- a/docs/migration.md +++ b/docs/migration.md @@ -32,6 +32,23 @@ produced a `UnaryOperatorNode{Operator: NOT}` wrapping the inner node (whose own `IsNot = true`, matching how `RegexMatchNode` already worked. `NOT LIKE` now produces a `BinaryOperatorNode` with `Operator = TokenOperatorNotLike`. +For example, `status NOT IN ('archived')` changed shape: + +```mermaid +flowchart LR + subgraph before["Before (v0.0.x)"] + direction TB + U["UnaryOperatorNode
Operator: NOT"] --> I1["InNode
IsNot: false"] + end + + subgraph after["v0.1.0"] + direction TB + I2["InNode
IsNot: true"] + end + + before ==>|"flattened"| after +``` + ```go // Before: type-switch had to unwrap the NOT node if u, ok := node.(*qfv.UnaryOperatorNode); ok && u.Operator == qfv.TokenOperatorNot { diff --git a/go.mod b/go.mod index a229809..b93ceb4 100644 --- a/go.mod +++ b/go.mod @@ -1,3 +1,3 @@ module github.com/slashdevops/qfv -go 1.25.0 +go 1.26