[2/2] chore(deps): upgrade EQL and eql-bindings to 3.0.3 - #431
Merged
freshtonic merged 44 commits intoJul 28, 2026
Conversation
Completes the v3 move. The workspace compiles again; cargo fmt and
clippy are clean.
The load-bearing change is in data_row.rs. EQL v2's eql_v2_encrypted was
a COMPOSITE type, so reads stripped a ('...') wrapper in text and a
12-byte rowtype header in binary. EQL v3's types are DOMAINS over jsonb
(95 CREATE DOMAIN, zero composites), and a domain is wire-identical to
its base type -- so both of those strips were wrong. Reads now take the
jsonb representation: bare JSON in text, a 1-byte version header plus
JSON in binary.
The four `to_ciphertext_*` decoder tests are regenerated from real EQL
v3 wire captures (a live encrypt round-trip against ZeroKMS + EQL
v3.0.2), replacing the stale v2 composite/rowtype fixtures this change
invalidates.
Also:
- Encrypt now carries EqlOutput, not EqlCiphertext, through the literal
and param paths: a query operand is not a ciphertext. Both v3 enums
are #[serde(untagged)], so the serialized wire form is unchanged.
- backend/bind/data_row imported cipherstash_client::eql::EqlCiphertext
directly, bypassing the crate alias and silently keeping the v2 type.
They now use crate::EqlCiphertext.
- EqlCiphertext::identifier is a method in 0.42.0, not a field.
- Map the new EqlError variants. UnsupportedSteVecOreInV3 gets its own
customer-facing error: v3 has no oc representation, so a ste_vec column
left in Standard (CLLW-ORE) mode cannot be written at all.
- cipherstash-config gained IndexType::SteVec { mode }. SteVecMode::Compat
(CLLW-OPE) is the default and the v3-compatible mode; Standard is
documented upstream as the legacy v2 protocol.
- CouldNotDecryptDataForKeyset gained a #[source]; Proxy's variant does
not carry it, so the chain stops at that boundary (noted in place).
Stable-Commit-Id: q-48c15nyszqvw09e8m6gydxqvfd
Captures the EQL v3 domain model, three impact maps over eql-mapper (type system, SQL surface, declarations/macros), and the open design questions, so the design session starts fresh with full context. Records one corrected finding: literals are inference sinks (value.rs:19 unifies them with a fresh tvar, not Native), so a token type on EqlValue buys no literal checking. Two subagents disagreed on this; resolved by reading the code. Stable-Commit-Id: q-2r73hgg9ex95aba238a8fbe4kp
Captures the shared understanding from the grill-with-docs session on extending the EQL Mapper type checker for EQL v3. CONTEXT.md: refresh the glossary to the v3 target vocabulary — EqlValue now carries domain identity; EqlTrait drops Contain and is documented as coarse; add term-extraction function, functional-index rewrite, query operand, token type, and domain identity. The old 'known model gap' note becomes 'design decisions in flight' pointing at the ADRs. ADRs (new packages/eql-mapper/docs/adr/): - 0001 functional-index rewrite (term functions, not native operators) — forced by Supabase's lack of CREATE OPERATOR; term-fn selection is the capability check. - 0002 token type as inert identity, not a checked dimension — it buys no safety (encrypted cols never unify; literals are inference sinks). Assessed against EQL/eql-bindings 3.0.2 (released 2026-07-20): additive query-twin bindings only; none of the design decisions change. Stable-Commit-Id: q-06b9k91e7pnntayq3rs11tqznz
…rands Two latent bugs in the trait-bound machinery, harmless only while bound checking is dead code (every encrypted column currently gets EqlTraits::all()). Fix them before the EQL v3 loader makes bounds reachable. - EqlTraits::difference was implemented as XOR (symmetric difference), so UnsatisfiedBounds would list traits the type *has* alongside the ones it is missing. It is now a true set difference (self AND NOT other). - Type::must_implement passed its operands reversed relative to Unifier::satisfy_bounds (implemented.difference(required) instead of required.difference(implemented)). Harmless only because XOR is commutative; wrong the moment difference is corrected. Now consistent. Adds unit tests pinning the asymmetric set-difference semantics and that must_implement reports exactly the missing bounds. Refs CIP-3595. Stable-Commit-Id: q-0e9kbc03hrqef307fg7qb5sqj1
Groundwork for the EQL v3 type checker (ADR-0002). Introduces the token type and its inert per-column domain identity, threading it through the type carriers without making it a checked dimension of unification. - New `TokenType` (integer/text/timestamp/…) and `DomainIdentity` (token type + v3 domain typname, e.g. eql_v3_text_ord_ore). - `EqlValue` widens to (TableColumn, Option<DomainIdentity>, EqlTraits); `ColumnKind::Eql`, `SchemaTableColumn` and the new `Column::eql_with_identity` carry the identity. `Column::eql` stays as a back-compat constructor that defaults the identity to None. - The identity is `None` everywhere for now: the branch still emits the v2 surface and no loader populates it yet (CIP-3598 does). It rides through unification and the associated-type machinery untouched, exactly because it is never inspected there. - ColumnKind loses `Copy` (DomainIdentity is not Copy); the handful of by-value uses now clone. - Also fixes the never-invoked, latently-broken no-bounds arm of the concrete_ty!/EQL() macro (wrong EqlValue arity, missing .into()). eql-mapper suite 79 passing; workspace checks clean; fmt + clippy clean. Refs CIP-3597. Stable-Commit-Id: q-5dxadzsggt5seabqhg3sdj8ht5
Implements the SchemaManager side of the v3 type checker (ADR-0002): per-column capability becomes *observed* from the Postgres domain name instead of a hardcoded EqlTraits::all(). - New schema/eql_domains.rs inverts the eql-bindings v3 catalog (v3::all()) into `typname -> (TokenType, EqlTraits)`. Because the catalog is generated from the same source as the installed SQL domains, the mapping cannot drift from them. Term -> capability: hm->Eq, op/ob->Ord, bf->TokenMatch, empty->storage-only; the JSON SteVec domains (term_json_keys == None) map to JsonLike (provisional, see note). - load_schema now resolves each column's v3 domain identity and traits and builds the column via Column::eql_with_identity. The legacy eql_v2_encrypted composite-type arm is retained for reading pre-v3 schemas; unrecognised domains load as Native. - select_table_schemas.sql now also selects information_schema domain_name: v3 encrypted columns are jsonb-backed DOMAINs, so udt_name reports the base type (jsonb) and the domain typname is only available via domain_name (cf. CIP-3441). - Adds eql-bindings 3.0.1 as a workspace dependency, used only by the proxy's schema loader; eql-mapper stays wire-format-agnostic. Re-exports DomainIdentity / TokenType from eql-mapper's crate root. Unit-tested offline against the catalog (12 tests asserting the SEM-term table, incl. the text hm+ord exception and query-twin exclusion). End-to-end validation needs a database with EQL v3 installed and is deferred; the encrypt-config cross-check is a follow-up within CIP-3598. Refs CIP-3598. Stable-Commit-Id: q-6p9ykbew5b6sjaf5p7y9qza4ft
…+ Contain
Applies three confirmed design decisions on the EQL v3 type checker.
1. Domain identity is non-optional (strict ADR-0002). EqlValue is now
(TableColumn, DomainIdentity, EqlTraits) and ColumnKind::Eql carries a
mandatory DomainIdentity — no more Option. The schema loader always
supplies the real identity; there is no honest v3 identity for a legacy
eql_v2_encrypted column, so the loader now drops the v2 arm and logs a
warning instead of fabricating one (v2 is already retired on this build).
- New helpers: TokenType::{as_domain_str, from_domain_name},
DomainIdentity::{from_domain_name, canonical}, and a test-only
EqlValue::with_canonical_identity.
- The schema!/concrete_ty!/test-helper macros synthesise a canonical
text-token identity by default; schema! also gains an
EQL("<domain>") form to pin an explicit v3 domain (token + OPE/ORE
variant) for tests that care.
2. JSON domains map to JsonLike + Contain, verified against the installed
v3 SQL (cipherstash-encrypt.sql): -> / ->> and @> / <@ are real
operators on eql_v3_json_search (not raise-stubs). Corrects the earlier
JsonLike-only guess.
3. The encrypt-config cross-check is dropped: the domain name is the sole
authority for a column's capability. (ADR-0002 doc update to follow.)
The Contain trait is therefore retained (scoped to JSON), NOT deleted as
the handoff/glossary/ticket assumed — @>/<@ raise only on scalar encrypted
columns. Docs/tickets updated separately.
eql-mapper 79 passing; proxy eql_domains 12 passing; workspace check,
fmt, clippy all clean.
Refs CIP-3597, CIP-3598.
Stable-Commit-Id: q-31sxsrtyrptajjpj0n383cb690
…h a real domain Review of the v3 domain-identity work (#424) noted that canonical() can synthesise a name that is not merely approximate but identical to an unrelated real catalog domain — canonical(Text, {json_like}) yields eql_v3_text_search, whose terms are hm/op/bf (Eq+Ord+TokenMatch), nothing to do with JSON. Strengthen the doc comment so no caller mistakes the synthesised name for the column's real domain; only from_domain_name is authoritative. Stable-Commit-Id: q-1ewbmpx74jnyxfd7dyftv08ka9
Review of #424 (finding 5) noted the v2->native fallback should be hard to miss in ops, because a v2 column served as a native passthrough takes no encryption on the write path: a plaintext value written to it is stored in plaintext (the eql_v2_encrypted AS ASSIGNMENT cast does no validation). Spell that out in the warning — the column is served as PLAINTEXT, Proxy does no encrypt/decrypt on it, migrate before writing — instead of the softer "ignoring unsupported column". Stable-Commit-Id: q-4dmrmqgewe5mzj0fzb2qdk5dja
Review of #424 (finding 1) flagged that `eql_v3_json_entry` — the element a `->` traversal returns, whose per-entry terms are `hm` XOR `op` — resolved to JsonLike+Contain, because `ste_vec_domain_type!` leaves `term_json_keys()` at None. A column mistakenly declared with that type would have type-checked as supporting `->`/`@>` but not `=`/`<`, which is inverted. Exclude the SteVec sub-structural domains (`NON_COLUMN_DOMAINS`) from the column catalog so such a column falls through to a native (plaintext) column instead of a wrong capability set. The every_public_column_domain assertion skips the same list. Makes the misclassification impossible rather than latent. Stable-Commit-Id: q-4dg749ehycbdazr4etn9rk1fvm
Two design corrections found during implementation:
- Containment is NOT removed in v3. Verified against the installed
cipherstash-encrypt.sql: @>/<@ raise on scalar encrypted columns but are
real, supported operators on encrypted JSON (eql_v3_json_search). The
Contain trait is retained as a JSON-only capability. Fixes the glossary
("removed in v3") and adds a consequence to ADR-0001.
- ADR-0002 amended: the domain name is the sole authority for a column's
capability (no encrypt-config cross-check), and the domain identity is
non-optional (legacy eql_v2_encrypted columns are dropped with a warning
rather than given a fabricated identity).
Refs CIP-3597, CIP-3598, CIP-3599.
Stable-Commit-Id: q-4rd3p778qhjfc5dessqeeg1b2e
…ation
Starts the v3 functional-index rewrite (ADR-0001) by pinning the pipeline
design and landing its hardest, most correctness-sensitive piece: choosing
the eql_v3 term-extraction function for an operator on a given column.
- ADR-0003 records the v3 rewrite pipeline: the stored-value vs query-operand
context split (column domain vs query twin cast targets), operator ->
term-function rewriting, JSON containment retained (not deleted), and the
rule inventory. All operand/term details verified against the installed
cipherstash-encrypt.sql, not the docs.
- DomainIdentity gains stores_hm/op/ob/bf and eq_term_fn/ord_term_fn/
match_term_fn/query_twin. These encode the verified selection rules:
* eq_term only where hm is stored; =/<> otherwise fall back to the ordering
term (an ord-only scalar like integer_ord compares via ord_term, exactly
as eql_v3.eq does).
* ord_term (op) vs ord_term_ore (ob) chosen from the domain typname.
* text is the hm exception (text_ord* stores hm alongside its order term).
* query_twin names the term-only operand domain (eql_v3.query_<bare>).
Unit-tested against the SEM-term table. The rewrite rule that consumes these
lands next.
Refs CIP-3599.
Stable-Commit-Id: q-4fed4c4vg8sehf76wwrn5yaysf
…ex form
First working slice of the v3 rewrite pipeline (ADR-0003). Scalar
comparisons on encrypted columns now emit the functional-index form, and
all cast targets move from the opaque eql_v2_encrypted to real v3 domains.
- New RewriteEqlComparisonOps: `col <op> x` -> `eql_v3.<term>(col) <op>
eql_v3.<term>(x)`, the term function chosen from the column's domain
identity (eq_term / ord_term / ord_term_ore). A domain with no term for
the operator is a capability error.
- Cast rules retargeted (helpers::cast_to_v3_domain): a query operand casts
to the eql_v3.query_* twin, a stored value (INSERT/UPDATE) to the
public.eql_v3_* column domain. Role is detected from the AST context
(post-order traversal + full NodePath ancestry): a value under a
comparison predicate is an operand, otherwise stored.
- CastLiteralsAsEncrypted gains node_types so it can name the target domain;
EqlTerm gains eql_value(); helpers gain eql_v3_term_call / is_comparison_op.
Example:
WHERE salary > $1
-> WHERE eql_v3.ord_term(salary) > eql_v3.ord_term($1::JSONB::eql_v3.query_text_ord)
Transitional: JSON (->, @>, jsonb_*) and aggregate (min/max) rewriting still
emit the eql_v2.* function wrappers over v3-cast operands; those wrappers are
retargeted in the next slices. Tests updated to the current output; the
params fixture gains `EQL: Eq` since a bare (capability-less) v3 column is
storage-only and rightly rejects `=`.
eql-mapper 85 passing; workspace check, clippy, fmt clean.
Refs CIP-3599.
Stable-Commit-Id: q-62kqdp86gcbb66tvwhy69tgn5s
Continues the v3 rewrite (ADR-0003). min/max and the jsonb_* passthrough functions on encrypted columns now rewrite to their eql_v3 counterparts. - RewriteStandardSqlFnsOnEqlTypes resolves a pg_catalog function to its eql_v3 counterpart via the new get_eql_v3_function_name, and rewrites only when one is declared. This fixes the long-standing count bug: count has no eql_v3 counterpart (it runs natively on the encrypted value), so it is no longer rewritten to a non-existent eql_v3.count. - sql_decls: the min/max/jsonb_path_*/jsonb_array_* target declarations move from eql_v2 to eql_v3. The containment declarations (jsonb_array, jsonb_contains, jsonb_contained_by) stay under eql_v2 until the containment slice. min(salary) -> eql_v3.min(salary) jsonb_path_query(col, '$.x') -> eql_v3.jsonb_path_query(col, '$.x'::...) count(col) -> count(col) (unchanged) eql-mapper 85 passing; workspace check, clippy, fmt clean. Refs CIP-3599. Stable-Commit-Id: q-092hd8jb1d31ck4va4n7r2gzet
Continues the v3 rewrite (ADR-0003). `@>`/`<@` on encrypted JSON columns now rewrite to the eql_v3 containment functions. Containment is retained in v3, scoped to JSON (ADR-0002 amendment) — `@>`/`<@` on scalar encrypted columns still raise. - RewriteContainmentOps emits eql_v3.jsonb_contains / jsonb_contained_by. - sql_decls: the jsonb_array / jsonb_contains / jsonb_contained_by target declarations move from eql_v2 to eql_v3. notes @> needle -> eql_v3.jsonb_contains(notes, needle) needle <@ notes -> eql_v3.jsonb_contained_by(needle, notes) No eql_v2.* function names remain in the mapper's rewrite output. eql-mapper 85 passing; clippy, fmt clean. Refs CIP-3599. Stable-Commit-Id: q-6dne3hbwwfsj7q42yqf4da8rqv
Continues the v3 rewrite (ADR-0003). `->`/`->>` on encrypted JSON columns are functionalised, since managed Postgres (Supabase) forbids the CREATE OPERATOR DDL the native operators would need (ADR-0001). - RewriteContainmentOps (now the general encrypted-JSON binary-op rule) also rewrites `col -> sel` -> `eql_v3."->"(col, sel)` and `col ->> sel` -> `eql_v3."->>"(col, sel)`. Operator-symbol function names are quoted; ordinary names are not. - CastLiteralsAsEncrypted passes a JSON field selector (an EqlTerm::JsonAccessor, the RHS of ->/->>) as encrypted *text* rather than a jsonb-domain cast, to match eql_v3."->"(json_search, text). notes -> 'medications' -> eql_v3."->"(notes, '<encrypted-selector>') ASSUMPTION TO CONFIRM (flagged in code): the encrypted selector replacement is the selector string itself. If the encrypt pipeline instead produces a jsonb payload for JSON selectors, the selector must be extracted rather than emitted verbatim — a one-line change once confirmed. eql-mapper 88 passing; clippy, fmt clean. Refs CIP-3599. Stable-Commit-Id: q-03j8aeq7we1c74jhkrgrny3rt4
LIKE/ILIKE on an encrypted column previously only unified the result with Native, so it bypassed the trait system entirely — a column with no match capability silently accepted LIKE. They now apply the `~~`/`~~*` (and negated) operator rules, so the encrypted LHS must implement TokenMatch and the pattern takes its Tokenized associated type. Adds tests: LIKE on a TokenMatch column type-checks; LIKE on an Eq-only encrypted column is a capability error. (The v3 rewrite of LIKE/@@ to `eql_v3.match_term(a) @> eql_v3.match_term(b)` is a separate follow-up; this fixes the type-checking gap.) eql-mapper 87 passing; clippy, fmt clean. Refs CIP-3599. Stable-Commit-Id: q-429ag1vxhrefpb1vy71sbkzg34
Continues the v3 rewrite (ADR-0003). LIKE/ILIKE on encrypted columns now
rewrite to the fuzzy-match functional form.
- New RewriteEqlMatchOps: `col LIKE pat` -> `eql_v3.match_term(col) @>
eql_v3.match_term(pat)`; `NOT LIKE` wraps it in `NOT (...)`. Fuzzy match
compares bloom-filter terms with @> (containment), so the operator becomes
@> between the two match_term calls, mirroring eql_v3.matches. A column
whose domain stores no bloom filter has no match_term -> capability error.
- Operand role detection (helpers::is_query_operand) now also treats a value
under a LIKE/ILIKE predicate as a query operand, so the pattern casts to the
query twin.
email LIKE 'a%'
-> eql_v3.match_term(email) @> eql_v3.match_term('<ct>'::JSONB::eql_v3.query_text_match)
`@@` (BinaryOperator::AtAt) is the remaining fuzzy-match surface; it needs the
operator declaration to parse first (separate slice).
eql-mapper 88 passing; clippy, fmt clean.
Refs CIP-3599.
Stable-Commit-Id: q-0gdjjy9gdt34jne9yhvs1kjw9m
… double domain Fills the rewrite-output test gaps called out in review of the v3 rewrite pipeline (#428): - ILIKE and NOT LIKE / NOT ILIKE — the negated match arm in RewriteEqlMatchOps had no rewrite-output coverage (only positive LIKE/@@ did). - UPDATE SET stored-value cast — of ADR-0003's two stored-value contexts only INSERT had cast-target coverage; UPDATE now pins `SET salary = 'ENCRYPTED'::JSONB::public.eql_v3_text`. - native LIKE still type-checks — regression lock for routing LIKE/ILIKE through the TokenMatch-bounded rule (Native satisfies all bounds). - the `double` token does not swallow its capability suffix — pins the eql_v3_double_ord invariant that suffix()/stores_* only documents in a comment. Stable-Commit-Id: q-47bk2x9aqbw7j8v04j81ecbec4
Completes the fuzzy-match surface (ADR-0003). `@@` now parses in the
operator DSL and rewrites like LIKE/ILIKE.
- eql-mapper-macros: the binary-operator parser accepts `@@` (it previously
only accepted `@>` after `@`), so `@@` can be declared.
- sql_decls: declare `<T>(T @@ <T as TokenMatch>::Tokenized) -> Native where
T: TokenMatch` — `@@` requires the match capability.
- RewriteEqlMatchOps now also handles `BinaryOp::AtAt`:
col @@ pat -> eql_v3.match_term(col) @> eql_v3.match_term(pat)
and `is_query_operand` treats an `@@` operand as a query operand so the
pattern casts to the query twin.
eql-mapper 89 passing; eql-mapper-macros 3 passing; clippy, fmt clean.
Refs CIP-3599.
Stable-Commit-Id: q-7s39kd5jdyhfjh1qyhkg2s858f
Validates the schema! EQL("<domain>") explicit-domain form end to end: a
block-ORE ordering column (eql_v3_integer_ord_ore) rewrites through
ord_term_ore (not ord_term) with the query_integer_ord_ore twin. This
exercises the ord-vs-ord_ore distinction that the inert domain identity
(ADR-0002) exists to carry.
seq > 5 -> eql_v3.ord_term_ore(seq) > eql_v3.ord_term_ore(...::eql_v3.query_integer_ord_ore)
eql-mapper 90 passing.
Refs CIP-3599, CIP-3600.
Stable-Commit-Id: q-0pjttkst7g5xwva2ag41v1nk6j
The queue changes user-visible behaviour (v3 typed jsonb domains replace eql_v2_encrypted, cipherstash-client 0.42.0 / EQL 3.0.2, LIKE/ILIKE now capability-checked, `@@` newly supported) but touched no CHANGELOG entry. Add one [Unreleased] section covering the migration, per CLAUDE.md and the review notes on #424/#428. Stable-Commit-Id: q-0s2fx8fb1q2563t7b45x9crjmf
query_twin() built the query-operand type as query_<bare-domain>, so a JSON column (eql_v3_json / eql_v3_json_search) cast its query operand to a non-existent eql_v3.query_json_search. The catalog defines a single jsonb query operand type, eql_v3.query_json, for every JSON domain — the shape of a jsonb needle does not vary by searchable capability. Special-case the JSON token so it maps to eql_v3.query_json; scalars keep query_<bare>. Found by running the integration suite end-to-end against a live EQL v3 DB. NOTE: this unblocks the operand *type*, but the jsonb operator surface still has a deeper gap — the mapper rewrites `col -> sel` to `eql_v3."->"(col, sel::...::eql_v3.query_json)` while the catalog's `eql_v3."->"` takes a plain text selector, so jsonb queries remain broken (the reviewer's JsonAccessor-selector concern; tracked separately). Stable-Commit-Id: q-7sq54hzge8yqq1xz0wmdmv30xe
The RHS of `->`/`->>` and the path arg of `jsonb_path_query` are SELECTORS, passed to the EQL v3 functions as encrypted-selector *text* (`eql_v3."->"(json, text)`, `jsonb_path_query(json, text)`), not a jsonb query-domain payload. The literal path already special-cased JsonAccessor, but the param path (cast_params_as_encrypted) cast every operand to the query twin — so a `->` selector param became `$1::JSONB::eql_v3.query_json` and matched no function. Leave JsonAccessor/JsonPath param placeholders bare (type inferred from the function signature; the proxy encrypts them as SteVec selectors via QueryOp::SteVecSelector), and extend the literal path to JsonPath. Resolves the reviewer's 'JsonAccessor selector assumption' NOTE — confirmed against the live encrypt path (the client emits the tokenized selector string). Found by running the integration suite against a live EQL v3 DB: this makes the `->`/jsonb_path_query SQL valid. Deeper jsonb gaps remain (field-access results, `ord_term` comparison) but those live in the EQL functions / client term encoding, not the mapper. Stable-Commit-Id: q-1216j97nm8w014hketysnfns6n
A jsonb selector (RHS of `->`/`->>`, or the `jsonb_path_query` path) is emitted by cipherstash-client as `EqlQueryPayloadV3::Selector(String)` — a bare tokenized-selector hash used directly as the `text` argument of `eql_v3."->"(json, text)` / `eql_v3.jsonb_path_query(json, text)`. Both bind paths serialized the whole EqlOutput as JSON — the literal path via `to_json_literal_value` (`serde_json::to_string`) and the param path via `Bind::rewrite` (`serde_json::to_value(..).to_string()`). For the bare-string Selector variant that re-adds JSON quotes, so the proxy bound `"<hash>"` (quoted). The function matches `@.s == $sel` against the stored per-entry `s` (unquoted hash), so it never matched and `->` returned NULL. Special-case the Selector variant in both paths to bind the raw token. Verified live: `encrypted_jsonb -> 'string'` now matches the stored entry (was NULL). NOTE: the matched entry still comes back encrypted — return-path decryption of the eql_v3_json_entry is a separate, remaining gap (CIP-3631). Stable-Commit-Id: q-642wqbdcnchf6v1y21111edetb
`encrypted_jsonb -> 'field'` and `jsonb_path_query{,_first}(...)` rewrite to
`eql_v3."->"(...)` / `eql_v3.jsonb_path_query(...)`, which return a single
`eql_v3_json_entry`. Two proxy-side gaps left those queries returning NULL or
an undecrypted entry:
- Return path: a json_entry `{v,i,h,s,c,op}` carries a `c`, so it deserialised
as a scalar `Encrypted` payload whose ciphertext is not self-decryptable
(an entry ciphertext only opens with its selector-derived nonce). Detect the
bare entry by its root-level selector `s` and reshape it into a one-entry
SteVec document `{v,k:"sv",i,h,sv:[{s,c,op?}]}` so the existing SteVec
decrypt path recovers the field value.
- Param path: the tokenized selector is bound as `text`, but a binary Bind
unconditionally prepended the jsonb version byte (0x01), corrupting the
selector so `->` matched no stored entry. Bind it as bare text with no
header (the binary wire form of `text` is just its raw bytes).
Together these make encrypted JSON field access and path queries work over
both the simple and extended protocols (17 integration tests). The remaining
jsonb ordering, containment, array and term-filter gaps are tracked in
CIP-3630 / CIP-3631.
Stable-Commit-Id: q-2rq44tjqgd12sceeq283d6gk1s
Implements `WHERE col -> sel <op> value` (`<`, `<=`, `>`, `>=`, and the
`jsonb_path_query_first(col, sel) <op> value` form) over encrypted JSON, in
both the simple and extended protocols. Rewrites to the EQL v3 functional-index
form (matching the stack adapters' `eql_v3.gt(col -> $1, $2::query_*_ord)`):
eql_v3.ord_term(eql_v3."->"(col, <selector>))
> eql_v3.ord_term($2::jsonb::eql_v3.query_integer_ord)
- New `EqlTerm::JsonOrd` / `EqlTermVariant::JsonOrd` — the scalar ordering
operand of a JSON field comparison. Inferred (infer_type_impls/expr.rs) for
the non-`->` side of a `<`/`<=`/`>`/`>=` comparison whose other side is a
JsonLike field access. Equality is intentionally excluded (exact JSON equality
is value-selector containment, a separate feature).
- The operand casts to `eql_v3.query_integer_ord` — a shape-only, type-agnostic
ord twin. `eql_v3.ord_term` extracts the `op` bytes regardless of scalar type
and the twin's CHECK is shape-only (`{v,i,op}`), so ONE twin serves any JSON
leaf type. This is what makes it work in the extended protocol, where the
operand's scalar type is unknown at rewrite time.
- The proxy encrypts a JsonOrd operand via `QueryOp::SteVecTerm` (zerokms.rs)
and encodes the scalar to match the stored leaf's `for_json_value` SteVec `op`
(from_sql.rs): a JSON number → float (the 8-block CLLW-OPE encoding, NOT a
small int), a JSON string → its unquoted text. Both the bare-literal
(`> '4'` / `> 'C'`) and text-format jsonb-param renderings are parsed as JSON
to recover the scalar type/content.
Fixes the 8 `select_where_jsonb_{gt,gte,lt,lte}` integration tests (numeric and
string); no regressions (mapper 97 unit tests green, jsonb suite 54→62 passing).
Stable-Commit-Id: q-3ecs5hbqrk8vw4827atp3zp17n
Cleanup pass (CIP-3601), now that the v3 rewrite is in place. - eql-mapper-macros: the EqlTrait / operator parse-error messages listed only Eq, Ord, TokenMatch, JsonLike but Contain is a valid keyword (retained in v3 for JSON containment). Both messages now include Contain. - eql-mapper CONTEXT.md: the "migration in flight" note claimed the code still emits the v2 surface; it now emits v3 (no eql_v2.* names remain in output), with end-to-end validation still pending. Note on the default()/all() reconciliation (the other CIP-3601 item): no production change is needed. schema_delta's collect_ddl never fabricates EQL traits (new/added columns are Native), so its EqlTraits::default() only appears in test assertions, consistent with the schema! macro's bare `EQL` (a capability-less, storage-only v3 column). The former mismatch was the loader's hardcoded all(), which CIP-3598 replaced with domain-derived capability. eql-mapper 90 passing; eql-mapper-macros 3 passing; fmt clean. Refs CIP-3601. Stable-Commit-Id: q-56sd3v5nxbdh5b4kpepyc1mm31
The showcase's encrypted columns move from the EQL v2 opaque type +
per-column search config to EQL v3 self-configuring domain types.
- schema.sql: `pii/medication/procedure eql_v2_encrypted` +
`eql_v2.add_search_config('ste_vec', ...)` -> `eql_v3_json_search` columns
(the searchable encrypted-JSON / SteVec domain). The domain type alone
declares the encryption and searchability, so the add_search_config calls
are removed.
- data.rs::clear(): drop the `eql_v2_configuration` DELETE — v3 has no such
config table (self-configuring domains); clearing now just truncates.
- Cargo.toml / README.md / CLAUDE.md / model.rs / main.rs diagram: reworded
from EQL v2 to v3, incl. a capability -> v3-domain-suffix mapping table in
the showcase CLAUDE.md.
The demo queries are unchanged: they are plain SQL (->, ->>, @>, <@,
jsonb_path_*, comparisons, ORDER BY, aggregates) that Proxy's mapper now
rewrites to the v3 functional-index form. End-to-end validation still needs
a live Proxy + database with EQL v3 installed.
Compiles clean.
Refs CIP-3595.
Stable-Commit-Id: q-2sptjp0nj4f9wer95m8yfk4nxg
The proxy's encrypt config is now derived from the database schema instead
of the `eql_v2_configuration` table. EQL v3 columns are self-configuring
domain types, so `eql_v2.add_search_config` and the config table are
redundant — the schema is the single source of truth.
- New encrypt_config/from_domain.rs: `column_config_from_domain` builds a
ColumnConfig from a column's v3 domain typname — cast type from the token,
indexes from the stored SEM terms (verified against cipherstash-client's
indexers + eql-bindings v3::terms): hm→Unique, op→Ope, ob→Ore, bf→Match,
JSON SteVec→SteVec{Compat}. Scalar non-text `_ord` domains store only `op`
(a single Ope index, no HMAC); text carries `hm` alongside its ordering
term. Storage-only and non-EQL domains yield no indexes / None.
- load_encrypt_config now runs the shared SCHEMA_QUERY (which already returns
information_schema.domain_name) and maps each encrypted column via
column_config_from_domain. Removes the eql_v2_configuration query, the
canonical-JSON path, ENCRYPT_CONFIG_QUERY + select_config.sql, and the
MissingEncryptConfigTable startup handling (a load error is now a genuine
database failure; an empty encrypted schema is a successful empty config).
- The obsolete canonical-config parsing tests are dropped; from_domain.rs
carries the equivalent domain→config coverage (9 tests).
This unblocks running against a v3-only EQL install (which has no
eql_v2_configuration). proxy builds; from_domain 9 tests pass; workspace
check, clippy, fmt clean (pre-existing data_row::to_ciphertext_* failures
are unrelated).
Refs CIP-3595, CIP-3579.
Stable-Commit-Id: q-6xh7ejw90r1kxzdd0jmmmkzd79
Replaces the EQL v2 `eql_v2_encrypted` columns + `add_search_config` / `add_encrypted_constraint` / `eql_v2_configuration` truncate with self-configuring v3 domain types (the proxy now infers encrypt config from the schema). - Main `encrypted` / `encrypted_elixir`: scalars use the default CLLW-OPE ordering domain `_ord` (op; also supports equality); `encrypted_text` is `text_search` (eq+ord+match); jsonb -> `json_search`. - `encrypted_bool` is `eql_v3_boolean` (storage-only) — boolean has no searchable capability in v3; the bool search fixtures/columns are dropped. - ORE/OPE fixture tables select their ordering family by `kind`: `ore` -> block-ORE (`_ord_ore`, `text_search_ore`), `ope` -> CLLW-OPE (`_ord_ope`, `text_ord_ope`). The `encrypted_bool` column and the `*_where_bool` fixtures are removed. - `unconfigured` columns become storage-only `eql_v3_text`. - schema-uninstall drops the tables; there is no v2 config table to remove. Refs CIP-3595. Stable-Commit-Id: q-5mf9na2j8evysprjfqpn7a8587
Follows the v3 schema conversion. The test crate now compiles and is consistent with EQL v3 semantics (runtime validation still needs a live Proxy + database with EQL v3 installed). - disable_mapping: the EqlEncrypted struct maps to `eql_v3_text_search` (the encrypted_text column's v3 domain) instead of `eql_v2_encrypted`. - jsonb_containment_index: the explicit `eql_v2.jsonb_contains(...)` call and comments become `eql_v3.jsonb_contains(...)`. - indexing: the encrypted index is now the v3 functional form `CREATE INDEX ON encrypted (eql_v3.ord_term(encrypted_text))`. - Bool search tests dropped (v3 boolean is storage-only): map_unique_index_bool, map_ore_where_generic_bool, map_ope_where_generic_bool. Bool roundtrip tests (encrypt/decrypt of a storage-only bool) are kept. - eql_regression (v2->v3 backwards-compat, which is out of scope — prior releases are not in use) is #[ignore]d with a note; regenerate fixtures from a v3 baseline to re-enable. Compiles clean; fmt clean. Refs CIP-3595. Stable-Commit-Id: q-17gkjm4bew1pevb1j47f7z0jvb
Convert the last docs, comments, and example SQL that still described the EQL v2 model (opaque `eql_v2_encrypted` type + `eql_v2.add_search_config` + `eql_v2_configuration` table) to the v3 self-configuring domain-type model. - ARCHITECTURE.md: rewrite the transformation-rules table (v3 rule set incl. RewriteEqlComparisonOps / RewriteEqlMatchOps) and the schema-loading paragraph (config inferred from domain types, no config table). - CONTEXT-MAP.md: update context/glossary entries; rewrite the "capability across the seam" note — the v3 schema loader now derives real per-column traits from the domain type, so the old "currently broken" bug is resolved. - docs/how-to/index.md: teach the `eql_v3_<token>_<cap>` domain-type model in place of add_search_config index setup; eql_v3.version(). - docs/reference/index.md: domain type enforces the encrypted-payload constraint; no add_encrypted_constraint call. - docs/reference/searchable-json.md: eql_v3_json_search schema + self-config note; caveat the v2 add_search_config option examples. - docs/errors.md, docs/sql/schema-example.sql, benchmark-schema.sql, parse.rs, psql-passthrough.sh, CLAUDE.md: v3 domain-type phrasing. Intentional v2 mentions retained: contrastive prose, the legacy-warn arm in schema/manager.rs, and the #[ignore]'d backwards-compat regression tests. Stable-Commit-Id: q-76caz2g4z3fptnpp8dn70s8v6g
…ALL) `eql_v3_json_search` columns were configured with `ArrayIndexMode::NONE`, which stores an array as a single opaque entry and never indexes its elements — so `@>`/`<@` containment against an array, `jsonb_array_elements`, and `[@]`/`[*]` path queries over encrypted arrays all failed (array element selectors did not exist to match). Index JSON arrays with `ArrayIndexMode::ALL` (item `[@]`, position `[n]`, and wildcard `[*]` element entries) so encrypted arrays are searchable the same way scalars are. This is the storage/searchability tradeoff a searchable JSON column opts into; the mode is a per-column config knob if a lighter footprint is wanted. Fixes 8 array integration tests: `jsonb_contains`/`jsonb_contained_by` with array needles (numeric + string), `jsonb_array_elements`, and `jsonb_path_query_first` array-wildcard (numeric + string). No regressions (jsonb suite 62->70 passing). `jsonb_array_length` over `[@]` remains open — it needs the flattened array-element rows counted, which is a separate rewrite. Stable-Commit-Id: q-3hn5yd17zqm6v6k7xex60x4wwa
Encrypted JSON equality (`col -> sel = value`) is not a term comparison. In EQL v3 exact equality is *containment of a value selector*: a single keyed MAC over the path and the canonicalised value together (`QueryOp::SteVecValueSelector`). One needle, built from TWO SQL operands. This is the mapper half. It breaks the implicit 1:1 correspondence between input and output operands for the first time: - New `EqlTerm::JsonValueSelector` / `EqlTermVariant::JsonValueSelector` — the value operand of a JSON field equality. Inferred for the non-JSON side of `=`/`<>` whose other side is a genuine field ACCESS (`->`, `->>`, `jsonb_path_query_first`). A bare `col = $1` on a whole encrypted JSON column is document equality and keeps its ordinary typing. - New `JsonValueSelectors` on `TypeCheckedStatement` records the one relationship node types cannot express: which operand supplies the path for which value. The mapper holds no encryption key, so it emits the composition *input*, not the needle; the proxy fuses and encrypts. - New `RewriteJsonValueSelectorEq` rewrites the comparison to `eql_v3.jsonb_contains(col, <value>::eql_v3.query_json)` (`<>` negates), discarding the field access. `RewriteEqlComparisonOps` now skips these — `eql_v3.eq_term` has no unique overload for a JSON query operand. - The value operand casts to `eql_v3.query_json`, alongside the existing `JsonOrd` -> `query_integer_ord` rule (helper renamed to `json_query_operand_cast_target` to cover both). A discarded selector *placeholder* stays declared in Parse but unreferenced in the SQL. PostgreSQL permits that as long as its type is known, which is what lets input and output param numbering stay identical — so Bind, ParameterDescription and the encrypt pipeline all stay positional. Verified against PostgreSQL 17: `PREPARE p (text) AS SELECT $2::jsonb` prepares and executes; the same statement with no declared types fails with "could not determine data type of parameter $1". Mapper unit tests 97 -> 102, no regressions. Proxy-side composition (the literal and Bind paths) follows in subsequent commits. Stable-Commit-Id: q-7qfnv43maha2chq6shq55sc3m0
The proxy half of encrypted JSON field equality. The mapper types the
value operand `JsonValueSelector` and records where its path comes from;
this fuses the two SQL operands into one `{"path", "value"}` plaintext and
encrypts it with `QueryOp::SteVecValueSelector`, producing the one-entry
containment needle `eql_v3.jsonb_contains` matches.
Both protocols:
- Simple: `literals_to_plaintext` composes the needle for a value-selector
literal from its own JSON scalar and the selector literal's text. The
discarded selector literal is still encrypted but never placed, since the
rewrite removed its node.
- Extended: `Bind::to_plaintext` gains a second pass. A fused param is
skipped by the positional pass — its own bytes are only half a needle and
would not decode as a standalone operand for the column — then composed
from the path param (or literal) and its own value.
- `Parse::declare_unreferenced_param_types` declares TEXT for the selector
placeholder the rewrite dropped. Clients that let the server infer types
send no types at all, and PostgreSQL will not prepare a statement with an
unreferenced param of unknown type. Still-referenced slots are left at
OID 0 ("infer"), so this never overrides a type the query determines.
Param counts are unchanged end to end, so Bind, ParameterDescription and
the encrypt pipeline all stay positional.
Also folds the four copies of the `$.`-rooting of a JSON selector into one
`json_selector_path` helper — the fused path needs the same normalisation
the accessor and path-query operands already did.
Fixes `select_jsonb_where_{string,numeric}_eq` (both protocols, both the
`->` and `jsonb_path_query_first` spellings). jsonb suite 70 -> 72
passing. Verified zero regressions by running the full integration suite
against the base binary and diffing failure sets: the only differences are
these two tests plus known-flaky ORE ordering tests that fail identically
on base.
The 7 `jsonb_term_filter` tests remain red for an unrelated reason:
`encrypted_jsonb_filtered` is declared `eql_v3_json_search`, the same
domain as the unfiltered column, and `from_domain.rs` hardcodes
`term_filters: Vec::new()`. EQL v3 has no domain or config that encodes a
term filter, so the proxy cannot infer one. Equality itself works on that
column — it matches case-sensitively. Case-insensitive JSON needs an
EQL-side way to declare filters; that is a separate feature.
Stable-Commit-Id: q-5afqdbdnjhd299x8777mwdm4fj
Stable-Commit-Id: q-6658b8n0t0qbfkcksk6720x0b5
Replaces the dangling-param trick with the real shape of the problem: an output param may be derived from more than one input param, so there is no 1:1 correspondence to rely on. Previously the JSON-equality rewrite left the fused path operand in the SQL as an unreferenced `$n`, purely so that param numbering stayed positional. That kept the plumbing simple at the cost of lying about the statement: a param that means nothing, and a neighbouring param whose meaning has quietly become "composite". Both are invisible at the call sites that bind them. The correspondence is now explicit: - `TypeCheckedStatement::transform` returns a `TransformedStatement` — the rewritten SQL plus a `ParamPlan`. Each `OutputParam` names the input param(s) its value is built from (`Input`, or `JsonValueSelector` carrying both operands). - A `RenumberParams` pass runs after the rewrite rules and assigns `$1..$m` in SQL order, so rules may freely drop or duplicate placeholders without maintaining numbering themselves. Running it as a separate pass keeps `FailOnPlaceholderChange` governing the rules. - `ParamPlan::check_covers` enforces the invariant that replaces 1:1 — **coverage**: every input param must be consumed by some output. An input that feeds nothing has been silently dropped, which is a bug in a rewrite rule. The proxy binds against the plan rather than by position. `Statement` carries both shapes: `param_columns` (what the client binds, used to decode and to answer Describe) and `output_params` (what PostgreSQL receives). Bind builds each output from the inputs its source names; ParameterDescription is rebuilt from the input params, since the server describes the rewritten statement and would otherwise tell the client too few params to bind. Parse carries each client-declared type across to the output param that consumes it. Every output param is now referenced by the rewritten SQL, so `declare_unreferenced_param_types` and its unreferenced-placeholder workaround are gone. When the plan is positional — every statement that is not a JSON equality — Bind still patches values in place, leaving the client's wire framing byte-for-byte as sent. Mapper units 102 -> 104 (renumbering across a fusion, and identity plans); proxy units 116 -> 117. Integration unchanged: the 2 JSON equality tests pass, and the four ORE-ordering tests that differ from the previous run fail identically on the base binary when run in isolation. Stable-Commit-Id: q-67avvp278knhdy7xe192mq667s
Removes `CastLiteralsAsEncrypted` and `CastParamsAsEncrypted`. Both were context-blind: they fired on any encrypted literal or placeholder and then guessed which domain to cast it to by walking up the ancestor chain (`is_query_operand`) looking for a comparison or a LIKE. The rule that actually knew the context — the one rewriting the predicate — was a different rule entirely, running later. Now the rule that owns a construct casts the operands of that construct, where the context is a fact rather than an inference: - `RewriteEqlComparisonOps`, `RewriteEqlMatchOps` — query operands, cast to the term-only `eql_v3.query_*` twin. - `RewriteJsonValueSelectorEq` — the fused needle, `eql_v3.query_json`. - `RewriteContainmentOps` — a `@>`/`<@` needle is a whole document, so it casts to the column domain; a `->`/`->>` selector takes no cast. - `CastFullPayloadOperands` (new) — the contexts with no rewrite of their own: INSERT values, UPDATE assignments, and directly-written `eql_v3.jsonb_contains(col, $1)` calls, which clients on platforms without operator support write themselves and which need the column-domain cast for the GIN index over `eql_v3.jsonb_array(col)`. It fires on the enclosing `Values`/`Assignment`/`Function` node, not on the literal, so the context is again structural. Substitution is separated from casting: `SubstituteEncryptedLiterals` puts each ciphertext in place wherever it appears and casts nothing, since where a literal sits decides its domain but not whether it needs replacing. It keeps the postcondition that every encrypted literal was consumed. The two domain choices are now named for the distinction they encode — `query_operand_domain` (just the terms a predicate needs) and `full_payload_domain` (ciphertext plus every indexed term) — and `is_query_operand`, the ancestor walk, is gone. Every rule matches against the ORIGINAL node via `node_path`, so a construct already rewritten by an earlier rule cannot be cast twice. Mapper 104 unit tests and proxy 117 unchanged. Integration: no regressions — the only tests that differ from the previous run are in the flaky ORE/OPE ordering family (net +1 passing), and the one that regressed both appears in the base binary's failure list and flips between ok and failed across identical isolated runs. Stable-Commit-Id: q-3y30vhx105f1ms7h36tanr8k7m
…term
`ORDER BY col` on an encrypted column was passed through untouched. The
column is a domain over `jsonb`, so PostgreSQL compared whole payloads by
jsonb rules — and jsonb compares objects field by field starting at `c`,
the ciphertext, which is randomised per encryption. Results came back in
an order that was not merely wrong but different on every insert.
Rewrites to the ordering term, which is what makes the comparison
meaningful:
ORDER BY col -> ORDER BY eql_v3.ord_term(col)
ORDER BY t.col DESC -> ORDER BY eql_v3.ord_term(t.col) DESC
ORDER BY col NULLS FIRST-> ORDER BY eql_v3.ord_term(col) NULLS FIRST
(`ord_term_ore` for block-ORE domains.) Sort options are untouched, so
ASC/DESC and NULLS FIRST/LAST keep working — the term is a plain orderable
value and NULL stays NULL. `ord_term` yields `ope_cllw`, a `bytea` domain
ordered bytewise; `ord_term_ore` yields `ore_block_256`, ordered by its own
btree operator class. Verified directly against PostgreSQL: over four
values re-encrypted five times, the bare `ORDER BY` gave a different wrong
order every run while the term ordering was correct every run.
Ordering by a column whose domain carries no ordering term is now a
capability error rather than an arbitrary sort, matching how
`RewriteEqlComparisonOps` treats an unsupported operator.
This is the cause of the "flaky" ORE/OPE ordering tests. They were not
flaky: the ordering was random, so a test comparing two values passed
about half the time while one comparing five values essentially never did.
Integration: 236 -> 298 passing, 168 -> 106 failing, nothing newly
failing. Fixes all of map_ope_index_order (6), map_ore_index_order (18)
and select::order_by (42), and those families are now deterministic across
repeated runs. Mapper units 104 -> 107.
Stable-Commit-Id: q-7znx6wzcc63yyk8hnganvrbdce
Every scalar predicate on an encrypted column failed:
ERROR: value for domain eql_v3.query_integer_ord violates check
constraint "query_integer_ord_check"
A query operand carries the column's search terms but never a decryptable
ciphertext — the `eql_v3.query_*` domains enforce that with `NOT (VALUE ?
'c')`. The proxy encrypted every scalar operand with `EqlOperation::Store`
and sent the storage payload, `c` and all, into a query position.
It cannot simply encrypt differently: a single `EqlOperation::Query`
yields only ONE term, while an operand may need several (`{v,i,hm,ob}` for
`query_text_ord`). The client's answer is to encrypt in Store mode and
project — `EqlCiphertextV3::into_query_operand`, whose own test is named
`into_query_operand_scalar_keeps_all_terms_and_drops_c`. The proxy never
called it.
Projecting needs to know which operands are query operands, and nothing
carried that: the operand of `WHERE col = $1` and the value of
`INSERT ... VALUES ($1)` are both `EqlTerm::Full`. So the mapper now
records it, in the new `QueryOperands`, alongside the existing
`JsonValueSelectors`. Membership is decided syntactically by the predicate
an operand belongs to — comparisons, `LIKE`/`ILIKE`, `@@` — which is the
same set of contexts whose rewrite rules cast to a `eql_v3.query_*` twin.
Containment (`@>`/`<@`) is deliberately excluded: its needle is a whole
document and keeps its full payload, as do INSERT values and UPDATE
assignments.
The flag rides through `ParamPlan` to the proxy's `OutputParam`, so Bind
projects the params it sends, and the literal path projects at Parse.
Payloads the encryptor already produced query-shaped (the JSON selectors
and SteVec terms, which do go through `EqlOperation::Query`) are left
alone.
Fixes map_ope_index_where (6), map_ore_index_where (6), map_match_index
(1) and 7 of 8 map_unique_index — scalar encrypted search worked in
neither protocol before this. Integration 298 -> 317 passing, 106 -> 87
failing, nothing newly failing.
The remaining map_unique_index failure (`..._all_with_wildcard`, a
`SELECT *` with six encrypted params) fails with a protocol-level
UnexpectedMessage and is a separate, pre-existing bug.
Stable-Commit-Id: q-4jyyztp3wxv6ve8ndnwpp59r2y
…term
The third instance of the same trap. An encrypted column is a domain over
`jsonb`, so a bare `GROUP BY col` grouped on the whole payload — including
`c`, the ciphertext, which is randomised per encryption. Rows holding the
same plaintext landed in different groups, degrading `GROUP BY` into
`GROUP BY <every row>`: `select::group_by` saw 20 groups for 10 distinct
values.
Grouping is equality, so the key is the same term `=` uses —
`eql_v3.eq_term(col)`, or `ord_term` for a domain that stores no `hm`
(`eql_v3_integer_ord` groups through its ordering term, exactly as it
compares).
Once the key is `eq_term(col)`, PostgreSQL no longer sees the bare column
as functionally dependent on it and rejects `SELECT col`. So a projection
of a grouped column is lifted through an aggregate, as James specified:
SELECT col, COUNT(*) FROM t GROUP BY col
->
SELECT any_value(col) AS col, COUNT(*) FROM t GROUP BY eql_v3.eq_term(col)
Every row in a group holds the same plaintext, so any of them decrypts to
the right answer. `FIRST` does not exist in PostgreSQL and `min`/`max` are
not defined over `jsonb`; `any_value` is polymorphic and works on the
domain directly. It requires PostgreSQL 16+, but only for the projection
case — grouping without selecting the column works on any version.
The projection is matched on the resolved column rather than on syntax, so
`SELECT t.col ... GROUP BY col` — which PostgreSQL accepts today — keeps
working. The original projection name is preserved, so clients selecting
by name are unaffected: this is what keeps
`select::regression::select_with_order_by_in_subquery` (a
`SELECT encrypted_text ... GROUP BY encrypted_text`) passing, which a
group-key-only rewrite would have broken.
Grouping by a column whose domain carries no equality term is now a
capability error rather than a grouping on ciphertext.
Integration 317 -> 323 passing, 87 -> 81 failing, nothing newly failing.
Mapper units 107 -> 109.
Stable-Commit-Id: q-4080dw66nm5aqm7g0q717mqvmw
Replaces the `any_value` stand-in with the aggregate EQL provides for
exactly this case (CIP-3657, EQL PR 423):
SELECT col, COUNT(*) FROM t GROUP BY col
->
SELECT eql_v3.grouped_value(col) AS col, COUNT(*)
FROM t GROUP BY eql_v3.eq_term(col)
`grouped_value` returns one representative value per group without
decrypting or comparing anything, which is all that is needed: every row
in a group is an encryption of the same plaintext. It re-creates the
aggregate the eql_v2 surface provided and that 3.0.0 dropped, and it
accepts any encrypted-domain value since all of them are jsonb-backed.
Being EQL's own answer to this problem it is also the form EQL's docs
describe, so a hand-written query and a proxy-rewritten one now agree.
`any_value` worked but was the wrong dependency to take: it is a
PostgreSQL 16+ builtin, so it silently raised the proxy's minimum server
version for a case EQL already covers.
**Requires an EQL release later than the 3.0.2 pinned in `mise.toml`** —
`eql_v3.grouped_value` does not exist in 3.0.2, so `select::group_by` and
`select::regression::select_with_order_by_in_subquery` will fail on a
clean install until `CS_EQL_VERSION` is bumped. Verified here by
installing the aggregate from PR 423 into the dev database by hand: all 6
group_by tests and the regression test pass, and the full integration
suite is unchanged at 323 passing / 81 failing with nothing newly failing.
Stable-Commit-Id: q-4h7rjf4sncgcvk7v1cpy3ndxjd
`CS_EQL_VERSION` eql-3.0.2 -> eql-3.0.3, and the `eql-bindings` crate =3.0.1 -> =3.0.3. EQL 3.0.3 carries `eql_v3.grouped_value` (CIP-3657, EQL PR 423), which the GROUP BY rewrite emits. Until now that commit was red on a clean install — the aggregate did not exist in 3.0.2 and had to be applied to the dev database by hand to test. It is now installed by the release, so `select::group_by` and `select::regression` pass with no local patching. `eql-bindings` was pinned two releases behind the SQL at 3.0.1; 3.0.2 and 3.0.3 are both the same ts-rs serde-attribute warning fix, with the emitted TS/JSON byte-identical, so the jump carries no behaviour change. The proxy uses the crate only in its schema loader, to map v3 domains to capabilities. Integration suite unchanged at 323 passing / 81 failing, with nothing newly failing or passing. Mapper 109 and proxy 117 unit tests green. Stable-Commit-Id: q-67fwe28n0ab44t0zefphhnqd4p
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
Contributor
Author
|
Superseded: this branch has been collapsed into #423, which is now the complete EQL v2→v3 migration as a single PR. Every commit here is present in #423 unchanged (same tree, same SHAs) — nothing is lost by closing this. Rationale: the queue boundary cut through an atomic migration. #423 installs EQL 3.0.2, but the v3 test fixtures, the schema-derived encrypt config, and the type checker all lived here — so #423's integration CI could never go green on its own, and |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
📚 eql-v3 PR · 2 of 2
Part of a queue. The PRs merge in FIFO order — the numbered order below, #1 first. Merging one supersedes the PRs after it until the author runs
git queue sync(rebases the rest onto the merged base) andgit queue submit(retargets their PRs).✅🟢 #423
queue/eql-v3/upgrade-deps→main⏳🟢 #431
queue/eql-v3/typecheck-and-transform→queue/eql-v3/upgrade-deps👈 this PR✅ approved · ♻️ changes requested · ⏳ review pending | 🟣 merged · 🟢 open · ⚫ closed — status as of the last
git queue submit.🥞 Managed by git-queue — do not edit this list by hand.
About this queue
Migrates CipherStash Proxy from EQL v2 to EQL v3 (cipherstash-client 0.34.1-alpha.4 → 0.42.0, EQL 2.3.0-pre.3 → 3.0.2), replacing the opaque eql_v2_encrypted composite type with 53 typed jsonb domains that encode both scalar type and searchable capability in the column type itself.