From 9801ae6c24f36aa65b6ca3bb5732a08188511eb9 Mon Sep 17 00:00:00 2001 From: David de Boer Date: Sat, 22 Aug 2026 15:18:56 +0200 Subject: [PATCH 1/2] =?UTF-8?q?feat(search):=20narrow=20a=20reference?= =?UTF-8?q?=E2=80=99s=20facet=20to=20the=20keys=20its=20target=20admits?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A Root Type may declare `facetKeys: { only }`, a predicate over the document key that says which of its documents get a facet bucket. It is declared once, on the target, and inherited by every facetable reference that names the type (a `lookup`’s `target`, an `idOnly`’s `labelSource`) – the same boundary along which a reference is re-keyed and a join is drawn. Only the facet narrows: the field keeps every value, so an excluded referent still displays and still filters exactly. `physicalFields(field, schema?)` gains a `facet` member – the field itself, or the `${name}_facet` companion of a reference inheriting a policy – so the projection, the collection definition and the query compiler keep reading one source for which engine field a facet reads. The projection writes the companion from the values the field stores, after its own `transform`, so the policy applies to keys and never to node IRIs. The `schema` parameter is optional, as it already is on `projectDocument` and `buildCollectionDefinition`: without one a field facets on its own name, the same degraded reading the projection makes when it cannot re-key a reference. The engine conformance suite gains one case: a policy facet buckets admitted keys only and carries labels, and a filter on an excluded value stays exact. --- ...022-key-a-root-type-on-a-declared-field.md | 65 ++++- docs/reference/search.md | 66 ++++- packages/search/src/index.ts | 1 + packages/search/src/project.ts | 42 +++- packages/search/src/schema.ts | 138 +++++++++- packages/search/src/testing.ts | 62 +++++ packages/search/test/project.test.ts | 237 ++++++++++++++++++ packages/search/test/schema.test.ts | 98 ++++++++ packages/search/vite.config.ts | 2 +- 9 files changed, 683 insertions(+), 28 deletions(-) diff --git a/docs/decisions/0022-key-a-root-type-on-a-declared-field.md b/docs/decisions/0022-key-a-root-type-on-a-declared-field.md index 4aeb6c39..36e46475 100644 --- a/docs/decisions/0022-key-a-root-type-on-a-declared-field.md +++ b/docs/decisions/0022-key-a-root-type-on-a-declared-field.md @@ -6,6 +6,11 @@ Date: 2026-08-21 Accepted +Amended 2026-08-22: gains the **facet policy** – `facetKeys` on a Root Type, +inherited by every facetable reference that names it, along the same boundary +as re-keying – as the index-side half of the same design. See “The facet +policy” under Decision and the two consequences it adds. + Extends [ADR 20](./0020-resolve-a-references-fields-from-the-targets-own-collection.md), whose contract – a reference holds ids of documents in the target’s collection – is what makes reference rewriting a consequence rather than a new rule. Relates @@ -71,14 +76,30 @@ reference dangling. needs a node’s key before the projection runs reads the same answer the projection will. -### The boundary, for keys and for joins alike +### The boundary, for keys, facets and joins alike Only a reference that **names** its target – a `lookup`’s `target`, an -`idOnly`’s `labelSource` – is re-keyed. That is the same line a join draws, and -for the same reason: naming the target is what asserts that the field holds ids -of that collection’s documents. An `idOnly` reference with no label source, and -a `derive`d reference over a raw internal path, never claimed as much, so -nothing rewrites them. +`idOnly`’s `labelSource` – is re-keyed, and only such a reference inherits the +target’s facet policy. That is the same line a join draws, and for the same +reason: naming the target is what asserts that the field holds ids of that +collection’s documents. An `idOnly` reference with no label source, and a +`derive`d reference over a raw internal path, never claimed as much, so nothing +rewrites them and no policy narrows them. + +### The facet policy + +A Root Type may declare which of its documents get a facet bucket – +`facetKeys: { only: isCovered }`, a predicate over the document key – and every +facetable reference naming the type inherits it. _Which ids deserve a bucket_ +is a fact about the target, not about each field pointing at it, so it is +declared once; the alternative, a query-time facet variant, adds a facets entry +type, a validation rule and a surface enum for a choice the deployment already +made when it built a cross-dataset index. Only the facet narrows: the field +keeps every value, so a local place still displays and still filters exactly. +The mechanism is a second physical field per inheriting facet – `${name}_facet`, +the admitted subset, written by the projection from the already-keyed values – +which the engine facets instead of the field, so the facet is exact under any +bucket cap. ## Consequences @@ -113,16 +134,40 @@ nothing rewrites them. - **The key is assigned before any `derive` runs**, so a derive sees the key and never the node IRI. A deployment that wants the node IRI declares a plain `idOnly` reference over the same path. +- **A facet policy is a second physical field in the engine** – a subset of the + same ids stored twice – and a collection-definition change for every type + referencing the policy’s type. Under a blue-green rebuild that is the next + run; under an in-place rebuild it is a loud failure at run open, after the + lock and before any write, asking for the collection to be dropped once: + rotating a pipeline version reprocesses datasets, it does not recreate + collections. The Typesense adapter keys facet results off the engine’s + physical field name, so the companion’s name is mapped back to the declared + field once, at the response boundary, before labels are resolved and counts + filed; and a membership filter follows the field’s _engine_ facet status + rather than its declaration, or a filter on an excluded value – the very case + the policy promises stays whole – would compile to the tokenised operator and + partial-match on a shared path prefix. +- **An over-strict predicate empties a facet instead of erroring.** Applied to + a type whose keys are not what it tests for – or to one keyed after its + references were indexed, so the stored values are still node IRIs – it admits + nothing and every facet referencing the type comes back empty, silently. A + documented footgun on `facetKeys` for now; a writer noticing a companion that + stayed empty across a run while its field did not is a cheap tell to add + later. - **A cross-dataset node reference does not resolve.** A work in dataset A pointing at a local node in dataset B gets no candidates – the hop runs against A’s distribution – so it stores the node IRI and dangles against B’s keyed document. Publishers reference other publishers through `sameAs` rather than directly, and such a reference is already unresolvable today for every purpose but labels. -- `@lde/pipeline`, `@lde/search-indexer` and the API packages are untouched, and - `@lde/search-typesense` only adopts the shared `rootTypeNamed` in place of a - by-name map of its own: the change is a schema member, the projection, and one - hop in the extraction generator. A schema declaring no `key` extracts, +- `@lde/pipeline`, `@lde/search-indexer` and the API packages are untouched. + The key is a schema member, the projection, and one hop in the extraction + generator, with `@lde/search-typesense` only adopting the shared + `rootTypeNamed` in place of a by-name map of its own; the facet policy is a + second schema member, the companion in the projection, and – in the adapter – + the companion in the collection definition, the facet and membership clauses + in the query compiler, the name mapping at the response boundary and the + in-place check. A schema declaring neither `key` nor `facetKeys` extracts, projects, indexes and queries exactly as before. ## Rejected diff --git a/docs/reference/search.md b/docs/reference/search.md index aa6d9951..e1435672 100644 --- a/docs/reference/search.md +++ b/docs/reference/search.md @@ -72,7 +72,9 @@ Exports are stratified by audience: - `assertTypeInSchema` – the port membership guard (the exact declaration object must be in the schema); - `physicalFields` / `PhysicalFields` – the indexed physical fanout a field - produces (per-locale search/sort keys); + produces (per-locale search/sort keys, and the field a facet reads – the + field itself, or the companion of a reference inheriting a + [facet policy](#facet-policy), which is why it takes the schema); - `physicalNameTokens` – the neutral name tokens an engine formats its own collection/index names from; - `irAlias` – the minted extraction predicate @@ -170,9 +172,9 @@ graph. A field that declares **no** role is an **internal field**: projected so a later `derive` can read it, then pruned before the writer and absent from the collection definition – not stored, not indexed, no RAM. The physical field names a declaration fans -out to (per-locale search/sort keys) come from `physicalFields`, the single -convention projection, the collection definition and the query compiler all -share. +out to (per-locale search/sort keys, the field a facet reads) come from +`physicalFields`, the single convention projection, the collection definition +and the query compiler all share. ```ts import { defineSearchType, projectRoots, searchSchema } from '@lde/search'; @@ -614,6 +616,62 @@ later: straight into the `is` filter that selects it; - a range-facet bin carries `min`/`max` instead – see below. +### Facet policy + +A Root Type may declare **`facetKeys`**: which of its documents get a facet +bucket, as a predicate over the [document key](#document-key). It is declared +once, on the type, and inherited by every facetable reference that _names_ it – +a `lookup`’s `target`, an `idOnly`’s `labelSource` – the same boundary along +which a reference is re-keyed and a join is drawn (so a `derive`d reference, +which reads no referent, is narrowed by nothing either): + +```ts +const place = defineSearchType({ + name: 'Place', + class: `${SCHEMA}Place`, + key: { field: '_sameAs', pick: (candidates) => candidates.find(isCovered) }, + // Cross-dataset index: only a key in a covered source gets a bucket, on every + // facet that references Place – a publisher’s local place is never one. + facetKeys: { only: isCovered }, + fields: [/* … */], +}); +``` + +_Which `Place` ids deserve a bucket_ is a fact about `Place`, not about each of +the fields that point at it (`CreativeWork.locationCreated`, `Person.birthPlace`, +`Organization.location`, …), so the policy lives on the target: a per-field +declaration would state one rule several times, and forgetting one would +silently reintroduce the buckets on that facet alone. The typical reason is a +**cross-dataset index**: two publishers who both left the same-named place +unaligned yield two buckets with one label, which a consumer cannot tell apart – +while a single-dataset app on its own index wants every bucket. The deployment +knows which it is building, so the choice is an indexing policy, not a query +option; a consumer sees one facet that simply has no local buckets. + +**Only the facet narrows.** The referring field keeps every value: a document +still displays the excluded place, and `where: { locationCreated: { in: +[kessel] } }` still matches it exactly. _Facets are discovery, filters are +exact_ – the one place the two deliberately disagree, so state it where your +consumers read. + +How it works: for a reference inheriting a policy, the projection writes a +`${name}_facet` companion holding the admitted subset of the field’s values – +taken after the field’s own `transform`, so the policy sees what the field +stores, which for a keyed target is the key – and the engine facets the +companion instead of the field (`physicalFields(field, schema).facet`). The +facet is therefore exact under any bucket cap: the engine never sees an +excluded value. Declaring a policy changes the collection definition of every +type that references the policy’s type; see the +[Typesense adapter](./search-typesense#collection-schema-and-engine) for what +that means for a live collection. + +**The failure mode is silence.** A predicate that admits none of a type’s keys +empties every facet referencing that type, with no error: a `Term` aligned to +AAT given the places’ `isCovered`, or – the likelier trap – a type that was +`key`ed _after_ its references were indexed, so the stored values are still +node IRIs. Key a type before declaring a policy over its keys, and give each +type its own predicate rather than reuse another’s. + ### Range facets A facetable numeric field (`integer`/`number`/`date`) may declare diff --git a/packages/search/src/index.ts b/packages/search/src/index.ts index 8debc101..d2f3967b 100644 --- a/packages/search/src/index.ts +++ b/packages/search/src/index.ts @@ -50,6 +50,7 @@ export type { RootTypeOf, SearchTypeIssue, KeyField, + FacetKeys, SearchSchema, FacetRange, ProjectionValue, diff --git a/packages/search/src/project.ts b/packages/search/src/project.ts index a68cea09..cbf8afb9 100644 --- a/packages/search/src/project.ts +++ b/packages/search/src/project.ts @@ -10,6 +10,7 @@ import { displayFieldName, documentKeyOf, fieldNamed, + inheritedFacetKeys, inlineFramingDepth, irAlias, isAbsoluteIri, @@ -262,7 +263,7 @@ function applyField( (field.kind === 'keyword' || field.kind === 'reference') && field.from !== undefined ) { - return applyProjectionValue(document, field, field.from, context); + return applyProjectionValue(document, field, field.from, context, schema); } if (field.derive !== undefined) { const derived = field.derive(document, context); @@ -322,12 +323,13 @@ function applyField( case 'text': return applyText(document, langValuesOf(node, alias), field); case 'keyword': - return applyFacet(document, literalsOf(node, alias), field); + return applyFacet(document, literalsOf(node, alias), field, schema); case 'reference': return applyFacet( document, referenceValues(node, alias, field, schema), field, + schema, ); case 'integer': return setNumber( @@ -479,6 +481,16 @@ function foldedSearchValue(values: readonly string[]): string { * `keyword` reads literals; `reference` reads IRIs (the caller passes the * already-read raw values). * + * A reference inheriting a {@link FacetKeys facet policy} from the type it + * names ({@link inheritedFacetKeys}) also writes the `${name}_facet` companion + * the engine facets instead of the field: the subset of **what the field + * stores** that the policy admits. So it is written after the field’s own + * `transform` and the IRI filter, from the same `values` – for a keyed target + * those are already keys ({@link referenceValues}), which is what makes a + * predicate over document keys the right shape. The field itself keeps every + * value; only the facet narrows. An empty subset writes nothing, exactly as a + * path that matched nothing does. + * * **`array` decides the shape**, as it does for every other kind: a declared * `array` field stores a list, and a single-valued one stores the first value – * the graph may still carry several, exactly as it may carry several literals @@ -493,6 +505,7 @@ function applyFacet( document: ProjectedNode, raw: readonly string[], field: KeywordField | ReferenceField, + schema: SearchSchema | undefined, ): void { // A `reference` stores identity, so what it stores must be an IRI whatever // route the value arrived by. {@link iriString} guards the graph path, but a @@ -505,18 +518,34 @@ function applyFacet( referenced ? transformed.filter(isAbsoluteIri) : transformed, ); const folded = dedupe(values.map((value) => fold(value))); - const searchField = physicalFields(field).search[0]; + const names = physicalFields(field, schema); + const searchField = names.search[0]; + const policy = inheritedFacetKeys(field, schema); + // `names.facet` is the companion exactly when a policy is inherited – + // physicalFields reads the same `inheritedFacetKeys`. + const facetField = names.facet as string; + // The companion is a subset of what the field STORES – for a single-valued + // field its first value, not the first admitted of all of them, or the facet + // would count a value no filter on the field can reproduce. + const stored = field.array === true ? values : values.slice(0, 1); + const admitted = policy === undefined ? [] : stored.filter(policy.only); if (field.array === true) { - setArray(document, field.name, values); + setArray(document, field.name, stored); if (field.searchable) { setArray(document, searchField, folded); } + if (policy !== undefined) { + setArray(document, facetField, admitted); + } return; } - setString(document, field.name, values[0]); + setString(document, field.name, stored[0]); if (field.searchable) { setString(document, searchField, folded[0]); } + if (policy !== undefined) { + setString(document, facetField, admitted[0]); + } } /** @@ -535,13 +564,14 @@ function applyProjectionValue( field: KeywordField | ReferenceField, from: ProjectionValue, context: ProjectionContext, + schema: SearchSchema | undefined, ): void { // `ProjectionContext` is keyed by `ProjectionValue`, so the declaration reads // the context directly: a projection value that has no context key, or a // context key no declaration can name, is a compile error rather than a // silently unpopulated field. const value = context[from]; - applyFacet(document, value === undefined ? [] : [value], field); + applyFacet(document, value === undefined ? [] : [value], field, schema); } /** diff --git a/packages/search/src/schema.ts b/packages/search/src/schema.ts index a5c3e243..36178f37 100644 --- a/packages/search/src/schema.ts +++ b/packages/search/src/schema.ts @@ -404,6 +404,41 @@ export interface KeyField { readonly pick?: (candidates: readonly string[]) => string | undefined; } +/** + * A {@link RootType}’s **facet policy**: which of its documents get a facet + * bucket, as a predicate over the document key. Declared once, on the type, + * and inherited by every facetable reference that *names* the type – a + * `lookup`’s `target`, an `idOnly`’s `labelSource` ({@link labelSourceNameOf}) + * – because *which of a type’s ids deserve a bucket* is a fact about the type, + * not about each field that points at it. A per-field policy would be one rule + * declared N times, and forgetting one would silently reintroduce the buckets + * on that facet alone. + * + * Only the facet narrows. The referring field keeps **every** value: a + * document still displays the excluded referent, and a `where` filter on its + * IRI still matches exactly. *Facets are discovery, filters are exact.* + * + * The mechanism is a second physical field per inheriting facet – + * `${name}_facet`, holding the admitted subset ({@link physicalFields}) – which + * the engine facets instead of the field itself, so the facet is exact under + * any bucket cap: the engine never sees an excluded value. Declaring a policy + * therefore changes the collection definition of every type that references + * this one; an engine adapter that reuses a live collection must notice. + * + * **The failure mode is silence.** A predicate that admits none of a type’s + * keys – a type whose alignments are not what it tests for, or stored values + * that are still node IRIs because the type was keyed after its references + * were indexed – empties every facet referencing the type without an error. + * Give each type its own predicate rather than reuse another’s, and key a type + * ({@link RootType.key}) before declaring a policy over its keys. + */ +export interface FacetKeys { + /** Whether a document key gets a bucket. Must be **pure**: it is applied per + * document at projection time, so an impure one would bucket one document + * and not its twin. */ + readonly only: (id: string) => boolean; +} + /** * A **Root Type**: a {@link SearchType} that is indexed. It declares a `class`, * roots are selected for it, a Writer owns a collection for it, and the @@ -419,6 +454,10 @@ export interface RootType extends SearchTypeBase { /** Which declared field holds this type’s document key, when it is not the * node’s own IRI ({@link KeyField}, {@link documentKeyOf}). */ readonly key?: KeyField; + /** Which of this type’s documents get a facet bucket, on every facetable + * reference that names this type ({@link FacetKeys}). Unset, every + * document does. */ + readonly facetKeys?: FacetKeys; } /** @@ -441,6 +480,9 @@ export interface ReferenceType extends SearchTypeBase { * referrer rather than keyed in a collection of its own. `never` for the * same reason `class` is. */ readonly key?: never; + /** Nothing references a Reference Type by id, so there is no facet for a + * policy to narrow. `never` for the same reason `key` is. */ + readonly facetKeys?: never; } /** @@ -997,7 +1039,9 @@ export interface SearchTypeIssue { | 'key-field-not-array' | 'key-field-inline' | 'key-pick-not-a-function' - | 'key-not-allowed'; + | 'key-not-allowed' + | 'facet-keys-only-not-a-function' + | 'facet-keys-not-allowed'; } /** @@ -1112,7 +1156,10 @@ const LOCALE_PATTERN = /^[A-Za-z0-9]+(-[A-Za-z0-9]+)*$/; * the dataset off a declaration no rule picks between; * - a {@link RootType.key} names a declared, `path`-bearing, `array`, * non-`inline` `reference` field of this type, and its `pick` is a function - * ({@link keyIssues}); a {@link ReferenceType} declares no key at all. + * ({@link keyIssues}); a {@link ReferenceType} declares no key at all; + * - a {@link RootType.facetKeys} policy’s `only` is a function + * ({@link facetKeysIssues}); a {@link ReferenceType} declares no policy at + * all. * * Pure and total: returns every issue rather than throwing; * {@link assertValidSearchType} is the throwing entry point. @@ -1275,7 +1322,7 @@ export function validateSearchType( } } } - issues.push(...keyIssues(searchType)); + issues.push(...keyIssues(searchType), ...facetKeysIssues(searchType)); return issues; } @@ -1331,6 +1378,31 @@ function keyIssues(searchType: SearchType): readonly SearchTypeIssue[] { return issues; } +/** + * The issues a type’s {@link RootType.facetKeys} declaration carries, filed + * under the member’s name: the policy names no field – it is a rule about the + * type’s own keys, applied through other types’ fields – so there is no field + * name to file it under. + */ +function facetKeysIssues(searchType: SearchType): readonly SearchTypeIssue[] { + const facetKeys = (searchType as RootType).facetKeys; + if (facetKeys === undefined) { + return []; + } + const issue = (reason: SearchTypeIssue['reason']) => ({ + field: 'facetKeys', + reason, + }); + // Nothing references a Reference Type by id, so a policy on one narrows no + // facet – it states a rule nothing could apply. + if (searchType.class === undefined) { + return [issue('facet-keys-not-allowed')]; + } + return typeof facetKeys.only === 'function' + ? [] + : [issue('facet-keys-only-not-a-function')]; +} + /** The union flattened to every possible member – the uniform shape runtime * validation and generic iteration read; never a declaration type. */ interface FlatField extends SearchFieldBase, Searchable, RangeFacetable { @@ -1389,6 +1461,11 @@ export interface PhysicalFields { /** Per-locale folded sort keys `${name}_sort_${locale}` (localized text, * `sortable`); a non-localized field sorts on its own `name` field. */ readonly sort: readonly string[]; + /** The field a facet on this field reads, when `facetable`: the field’s own + * `name`, or the `${name}_facet` companion holding the subset its target’s + * {@link FacetKeys facet policy} admits. `undefined` for a field that is + * not facetable. */ + readonly facet: string | undefined; } /** @@ -1736,11 +1813,21 @@ export function unixSecondsToIso(seconds: number): string { /** * Derive the **indexed** physical engine field names a declaration produces: - * the per-locale `search`/`sort` fanout. A localized `text` field’s display - * fields are pattern-based and not enumerated here – see {@link displayFieldName} - * and its siblings. + * the per-locale `search`/`sort` fanout, and the field a `facet` reads. A + * localized `text` field’s display fields are pattern-based and not enumerated + * here – see {@link displayFieldName} and its siblings. + * + * The facet is the one member that depends on more than the declaration: a + * facetable reference whose target declares a {@link FacetKeys facet policy} + * facets a `${name}_facet` companion rather than itself, and only the `schema` + * can resolve that target ({@link inheritedFacetKeys}). Without one the field + * facets on its own name – the same reading the projection makes without a + * schema, where it cannot re-key a reference either. */ -export function physicalFields(field: SearchField): PhysicalFields { +export function physicalFields( + field: SearchField, + schema?: SearchSchema, +): PhysicalFields { if (field.kind === 'text') { const locales = field.locales; return { @@ -1750,10 +1837,47 @@ export function physicalFields(field: SearchField): PhysicalFields { sort: field.sortable ? locales.map((locale) => `${field.name}_sort_${locale}`) : [], + // Text is never facetable (`validateSearchType`). + facet: undefined, }; } return { search: field.searchable !== undefined ? [`${field.name}_search`] : [], sort: [], + facet: + field.facetable !== true + ? undefined + : inheritedFacetKeys(field, schema) === undefined + ? field.name + : `${field.name}_facet`, }; } + +/** + * The {@link FacetKeys facet policy} a field inherits: the `facetKeys` of the + * Root Type it names ({@link labelSourceNameOf}), when the field is a facetable + * reference and the schema resolves that type. The boundary is *naming the + * target* – the same line along which a reference is re-keyed and a join is + * drawn – so a reference that names no type inherits nothing, whatever type + * its values happen to point at; and a `derive`d reference, which produces its + * own values rather than reading a referent, is re-keyed by nothing and so + * narrowed by nothing either. The one reading the projection (which writes the + * companion) and {@link physicalFields} (which names it) share. + */ +export function inheritedFacetKeys( + field: SearchField, + schema: SearchSchema | undefined, +): FacetKeys | undefined { + if ( + schema === undefined || + field.kind !== 'reference' || + field.facetable !== true || + field.derive !== undefined + ) { + return undefined; + } + const targetName = labelSourceNameOf(field); + return targetName === undefined + ? undefined + : rootTypeNamed(schema, targetName)?.facetKeys; +} diff --git a/packages/search/src/testing.ts b/packages/search/src/testing.ts index 10360ea0..723a933f 100644 --- a/packages/search/src/testing.ts +++ b/packages/search/src/testing.ts @@ -11,6 +11,7 @@ import { facetableFields, filterableFields, ID_FIELD, + inheritedFacetKeys, nestedReferenceType, outputFields, type RootType, @@ -256,6 +257,54 @@ export function describeSearchEngineContract( } }); + it('buckets a facet inheriting a policy on admitted keys only, labelled, while filtering stays exact', async () => { + // A facet policy (`RootType.facetKeys`) narrows the FACET and nothing + // else. Two things an adapter can get wrong, each silently: facet the + // companion field but report its buckets under the engine’s name, which + // loses every label and files the counts under a key no consumer asked + // for; and pick the filter operator off the declaration, so a `where` + // on an excluded value – still `facetable` in the schema, no longer a + // facet in the engine – partial-matches on a shared path prefix. The + // adapter’s fixture should carry two IRIs sharing a prefix so the + // second assertion bites. + for (const searchType of types()) { + for (const field of facetableFields(searchType)) { + const policy = inheritedFacetKeys(field, engine().schema); + if (policy === undefined) { + continue; + } + const result = await engine().search(searchType, { + ...browse(searchType), + limit: 20, + facets: [field.name], + }); + const buckets = result.facets[field.name] ?? []; + for (const bucket of buckets) { + expect(policy.only(bucket.value)).toBe(true); + expect(bucket.label).toBeDefined(); + } + if (field.output !== true || field.filterable !== true) { + continue; // Nothing to read the stored values back from. + } + const excluded = result.hits + .flatMap((hit) => referenceIds(hit.document[field.name])) + .find((id) => !policy.only(id)); + if (excluded === undefined) { + continue; + } + const filtered = await engine().search(searchType, { + ...browse(searchType), + limit: 20, + where: [filterOn({ field: field.name, in: [excluded] })], + }); + expect(filtered.total).toBeGreaterThan(0); + for (const hit of filtered.hits) { + expect(referenceIds(hit.document[field.name])).toContain(excluded); + } + } + } + }); + it('answers a browse query with hits, a total and facets', async () => { for (const searchType of types()) { const result = await engine().search(searchType, browse(searchType)); @@ -281,6 +330,19 @@ function mismatchedFilter( : filterOn({ field, in: ['x'] }); } +/** The ids a reconstructed reference value carries – one labelled reference + * or a list of them – read the way a consumer reads them. */ +function referenceIds(value: unknown): readonly string[] { + const references = Array.isArray(value) ? value : [value]; + return references + .map((reference: unknown) => + typeof reference === 'object' && reference !== null && 'id' in reference + ? String(reference.id) + : undefined, + ) + .filter((id): id is string => id !== undefined); +} + /** The locale a query against this type may select (any is contract-valid). */ function firstLocale(searchType: RootType): string { for (const field of searchType.fields) { diff --git a/packages/search/test/project.test.ts b/packages/search/test/project.test.ts index 387c199d..d2aa805d 100644 --- a/packages/search/test/project.test.ts +++ b/packages/search/test/project.test.ts @@ -2035,3 +2035,240 @@ describe('document keys (a type keyed on a declared field)', () => { ]); }); }); + +describe('facet policy (a type narrowing the facets that reference it)', () => { + const SCHEMA_ORG = 'https://schema.org/'; + const GEONAMES = 'https://sws.geonames.org/'; + /** The deployment’s one predicate: is this key in a source an authority can + * resolve? Built into `pick` and the policy alike, so they cannot disagree. */ + const isCovered = (iri: string) => iri.startsWith(GEONAMES); + const normaliseIri = (iri: string) => + iri.replace(/^http:/, 'https:').replace(/\/$/, ''); + + const place = defineSearchType({ + name: 'Place', + class: `${SCHEMA_ORG}Place`, + labelField: 'name', + key: { field: '_sameAs', pick: (candidates) => candidates.find(isCovered) }, + // Cross-dataset index: only a key in a covered source gets a bucket, on + // every facet that references Place. + facetKeys: { only: isCovered }, + fields: [ + { + name: 'name', + kind: 'text', + path: `<${SCHEMA_ORG}name>`, + locales: ['und'], + output: true, + searchable: { weight: 1 }, + }, + { + name: '_sameAs', + kind: 'reference', + array: true, + path: `<${SCHEMA_ORG}sameAs>`, + transform: normaliseIri, + }, + ], + }); + + const work = defineSearchType({ + name: 'CreativeWork', + class: `${SCHEMA_ORG}CreativeWork`, + fields: [ + { + name: 'locationCreated', + kind: 'reference', + path: `<${SCHEMA_ORG}locationCreated>`, + facetable: true, + filterable: true, + output: true, + ref: { strategy: 'lookup', target: 'Place' }, + }, + { + name: 'about', + kind: 'reference', + array: true, + path: `<${SCHEMA_ORG}about>`, + facetable: true, + labelSource: 'Place', + transform: normaliseIri, + }, + { + // Names no type, so it inherits no policy – whatever it points at. + name: 'unnamed', + kind: 'reference', + array: true, + path: `<${SCHEMA_ORG}mentions>`, + facetable: true, + }, + { + name: 'genre', + kind: 'keyword', + array: true, + path: `<${SCHEMA_ORG}genre>`, + facetable: true, + }, + ], + }); + + const schema = searchSchema(place, work); + const workKey = (field: string) => alias('CreativeWork', field); + const placeKey = (field: string) => alias('Place', field); + + /** A referenced place as the frame embeds it: its node IRI, with the key + * candidates the extraction hop read for it. */ + const placeRef = (id: string, ...sameAs: string[]) => ({ + '@id': id, + ...(sameAs.length === 0 + ? {} + : { [placeKey('_sameAs')]: sameAs.map((iri) => ({ '@id': iri })) }), + }); + const aligned = placeRef('https://a/place/1', `${GEONAMES}2751283`); + const kessel = placeRef('https://a/place/2'); + const otherKessel = placeRef('https://b/place/7'); + + it('writes the admitted keys to the `_facet` companion and keeps every value on the field', () => { + const document = projectDocument( + { + '@id': 'https://ex/work/1', + [workKey('about')]: [kessel, aligned, otherKessel], + }, + work, + schema, + ); + + // The field stays whole: the local places still display and still filter. + expect(document.about).toEqual([ + 'https://a/place/2', + `${GEONAMES}2751283`, + 'https://b/place/7', + ]); + // The policy is applied to the KEY the reference stores, never to the + // referent’s node IRI – the aligned place’s node is as local as Kessel’s. + expect(document.about_facet).toEqual([`${GEONAMES}2751283`]); + }); + + it('narrows a single-valued reference to a string companion, absent when excluded', () => { + const withAligned = projectDocument( + { '@id': 'https://ex/work/1', [workKey('locationCreated')]: aligned }, + work, + schema, + ); + const withLocal = projectDocument( + { '@id': 'https://ex/work/2', [workKey('locationCreated')]: kessel }, + work, + schema, + ); + + expect(withAligned.locationCreated_facet).toBe(`${GEONAMES}2751283`); + expect(withLocal.locationCreated).toBe('https://a/place/2'); + expect(withLocal).not.toHaveProperty('locationCreated_facet'); + }); + + it('narrows a single-valued reference from the value it stores, not from every value the graph offered', () => { + // The graph carries two values for a single-valued field; the field stores + // the first. A companion holding the admitted SECOND would count a bucket + // no filter on the field can reproduce. + const document = projectDocument( + { + '@id': 'https://ex/work/1', + [workKey('locationCreated')]: [kessel, aligned], + }, + work, + schema, + ); + + expect(document.locationCreated).toBe('https://a/place/2'); + expect(document).not.toHaveProperty('locationCreated_facet'); + }); + + it('leaves a derived reference naming a policy type without a companion', () => { + // A derive produces its own values rather than reading a referent, so it + // is re-keyed by nothing and narrowed by nothing – the engine facets the + // field itself (physicalFields agrees), never an unwritten companion. + const derived = defineSearchType({ + name: 'Derived', + class: 'urn:x:Derived', + fields: [ + { + name: 'place', + kind: 'reference', + facetable: true, + labelSource: 'Place', + derive: () => `${GEONAMES}2751283`, + }, + ], + }); + const derivedSchema = searchSchema(place, derived); + + const document = projectDocument( + { '@id': 'https://ex/thing/1' }, + derived, + derivedSchema, + ); + + expect(document.place).toBe(`${GEONAMES}2751283`); + expect(document).not.toHaveProperty('place_facet'); + }); + + it('writes no companion when the policy admits nothing', () => { + // Absent rather than empty – exactly as a path that matched nothing. + const document = projectDocument( + { + '@id': 'https://ex/work/1', + [workKey('about')]: [kessel, otherKessel], + }, + work, + schema, + ); + + expect(document.about).toHaveLength(2); + expect(document).not.toHaveProperty('about_facet'); + }); + + it('applies the policy to what the field stores: after the field’s own transform', () => { + // A place whose own node already lies in a covered source, spelled the + // way the publisher spelled it. Its key is the node IRI (no candidates); + // the referring field’s `transform` normalises it, and THAT is what the + // policy sees – a policy over the stored value, not over the raw one. + const document = projectDocument( + { + '@id': 'https://ex/work/1', + [workKey('about')]: [placeRef('http://sws.geonames.org/1/')], + }, + work, + schema, + ); + + expect(document.about).toEqual([`${GEONAMES}1`]); + expect(document.about_facet).toEqual([`${GEONAMES}1`]); + }); + + it('leaves a reference naming no type, and a keyword facet, without a companion', () => { + const document = projectDocument( + { + '@id': 'https://ex/work/1', + [workKey('unnamed')]: [aligned], + [workKey('genre')]: [{ '@value': 'map' }], + }, + work, + schema, + ); + + expect(document.unnamed).toEqual(['https://a/place/1']); + expect(document.genre).toEqual(['map']); + expect(document).not.toHaveProperty('unnamed_facet'); + expect(document).not.toHaveProperty('genre_facet'); + }); + + it('writes no companion without a schema, which cannot resolve the policy', () => { + const document = projectDocument( + { '@id': 'https://ex/work/1', [workKey('about')]: [aligned] }, + work, + ); + + expect(document.about).toEqual(['https://a/place/1']); + expect(document).not.toHaveProperty('about_facet'); + }); +}); diff --git a/packages/search/test/schema.test.ts b/packages/search/test/schema.test.ts index 851c8879..937abfb8 100644 --- a/packages/search/test/schema.test.ts +++ b/packages/search/test/schema.test.ts @@ -121,6 +121,7 @@ describe('physicalFields', () => { expect(physicalFields(keyword)).toEqual({ search: ['keyword_search'], sort: [], + facet: 'keyword', }); }); @@ -167,8 +168,105 @@ describe('physicalFields', () => { expect(physicalFields(publisher)).toEqual({ search: [], sort: [], + facet: 'publisher', }); }); + + it('facets a reference on its `_facet` companion only when the type it names declares a policy', () => { + const agent: SearchType = { + name: 'Agent', + class: 'urn:x:Agent', + labelField: 'label', + facetKeys: { only: (id) => id.startsWith('https://viaf.org/') }, + fields: [ + { + name: 'label', + kind: 'text', + locales: ['und'], + output: true, + searchable: { weight: 1 }, + }, + ], + }; + const publisher: SearchField = { + name: 'publisher', + kind: 'reference', + facetable: true, + ref: { strategy: 'lookup', target: 'Agent' }, + }; + const creator: SearchField = { + name: 'creator', + kind: 'reference', + facetable: true, + labelSource: 'Agent', + }; + const mentions: SearchField = { + // Names no type, so it inherits nothing – whatever its values point at. + name: 'mentions', + kind: 'reference', + facetable: true, + }; + const derived: SearchField = { + // Produces its own values rather than reading a referent: re-keyed by + // nothing, narrowed by nothing. + name: 'derived', + kind: 'reference', + facetable: true, + labelSource: 'Agent', + derive: () => 'https://viaf.org/1', + }; + const schema = searchSchema(agent, { + name: 'Dataset', + class: 'urn:x:Dataset', + fields: [publisher, creator, mentions, derived], + }); + + expect(physicalFields(publisher, schema).facet).toBe('publisher_facet'); + expect(physicalFields(creator, schema).facet).toBe('creator_facet'); + expect(physicalFields(mentions, schema).facet).toBe('mentions'); + expect(physicalFields(derived, schema).facet).toBe('derived'); + // Without the schema the target cannot be resolved – the same reading the + // projection makes without one. + expect(physicalFields(publisher).facet).toBe('publisher'); + // A facet companion is a role’s fanout, like a search companion: none + // without the role. + expect( + physicalFields({ ...publisher, facetable: undefined }, schema).facet, + ).toBeUndefined(); + }); +}); + +describe('facet policy', () => { + const place = (facetKeys: unknown) => + ({ + name: 'Place', + class: 'urn:x:Place', + facetKeys, + fields: [], + }) as unknown as SearchType; + + it('accepts a policy whose `only` is a function', () => { + expect(validateSearchType(place({ only: () => true }))).toEqual([]); + }); + + it('rejects an `only` that is not a function', () => { + // A declaration built outside TypeScript (a generator, plain JS) is what + // this guards; a typed one cannot express it. + expect(validateSearchType(place({ only: 'covered' }))).toEqual([ + { field: 'facetKeys', reason: 'facet-keys-only-not-a-function' }, + ]); + }); + + it('rejects a policy on a Reference Type, which nothing references by id', () => { + const nested = { + name: 'Marker', + facetKeys: { only: () => true }, + fields: [], + } as unknown as SearchType; + expect(validateSearchType(nested)).toEqual([ + { field: 'facetKeys', reason: 'facet-keys-not-allowed' }, + ]); + }); }); describe('irAlias', () => { diff --git a/packages/search/vite.config.ts b/packages/search/vite.config.ts index c070a10d..97d95d3e 100644 --- a/packages/search/vite.config.ts +++ b/packages/search/vite.config.ts @@ -12,7 +12,7 @@ export default mergeConfig( thresholds: { functions: 100, lines: 100, - branches: 99.61, + branches: 99.63, statements: 100, }, }, From 3023a6302956c9633bb5349cc76d9068b54d8b69 Mon Sep 17 00:00:00 2001 From: David de Boer Date: Sat, 22 Aug 2026 15:19:12 +0200 Subject: [PATCH 2/2] feat(search-typesense): facet the companion of a reference under a facet policy MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit For a facetable reference whose target declares `facetKeys`, the collection definition declares the field itself as a plain stored value and its `${name}_facet` companion as the facet. Only the `schema` option resolves the policy; without it the field itself stays the facet, the reading the projection makes without a schema. An in-place writer refuses a declared dataset field that inherits a policy: it enumerates the indexed datasets by faceting that field, and a narrowed facet would hide the excluded datasets from the sweep. The query compiler facets `physicalFields(field, schema).facet` and picks the membership operator off the field’s engine facet status rather than its declaration: the base field is still `facetable` in the schema but no longer a facet in the engine, so a filter on an excluded value – the case the policy promises stays whole – would otherwise compile to the tokenised operator and partial-match on a shared path prefix. Typesense reports a facet under the field it faceted, so the engine maps each `facet_counts[].field_name` back to the declared name once, at the response boundary, before the label lookup and the response parsing – either of which meeting the engine name would silently lose the facet’s labels or file its counts under the wrong key. An in-place rebuild over a collection that predates the policy fails at run open – after the lock, before any write – naming the missing companion and the drop that fixes it, as it does for a missing reference field: the documents would carry the companion, the collection would not declare it, and every facet on a reference to the policy’s type would fail until someone dropped the collection by hand. --- docs/reference/search-typesense.md | 27 +- package-lock.json | 15 + packages/search-typesense/package.json | 1 + .../src/collection-definition.ts | 29 +- .../search-typesense/src/in-place-rebuild.ts | 81 +++- .../search-typesense/src/query-compiler.ts | 47 +- .../search-typesense/src/rebuild-support.ts | 22 +- packages/search-typesense/src/search.ts | 69 ++- .../test/facet-policy.test.ts | 419 ++++++++++++++++++ .../test/query-compiler.test.ts | 10 + .../test/rebuild-support.test.ts | 48 +- packages/search-typesense/vite.config.ts | 8 +- 12 files changed, 728 insertions(+), 48 deletions(-) create mode 100644 packages/search-typesense/test/facet-policy.test.ts diff --git a/docs/reference/search-typesense.md b/docs/reference/search-typesense.md index 863a1ba4..de58b592 100644 --- a/docs/reference/search-typesense.md +++ b/docs/reference/search-typesense.md @@ -98,6 +98,21 @@ labels. Pass the `schema` option for a type that surfaces one – it is what resolves the Reference Type; a type declaring one without it throws here, rather than building a collection whose documents would all fail to import. +A **reference inheriting a [facet policy](./search#facet-policy)** – a facetable +`lookup`/`labelSource` reference to a type declaring `facetKeys` – is declared +as a plain stored field (`facet: false`) plus a `${name}_facet` companion +(`facet: true`, optional) holding the admitted subset, and the query compiler +facets the companion and filters the field itself with the exact `:=` +operator. Only the `schema` option can resolve the policy: built without it, +the field itself is declared the facet and no companion is – the projection +makes the same reading without a schema – so a deployment declaring a policy +passes the schema to its writers, as `createSearchIndexer` does. + +An `InPlaceRebuild` additionally refuses a type whose declared +[dataset field](#provenance) inherits a policy: it enumerates the indexed +datasets by faceting that field, and a facet narrowed to the admitted keys +would hide every dataset the policy excludes from the membership sweep. + ### Joins across collections A reference declaring [`joinable: true`](./search#filtering-across-collections) @@ -309,10 +324,14 @@ Document ids must be unique per (source, entity) – the caller keys them. `openRun` creates the collection on demand and otherwise leaves an existing one alone – with one exception. If the collection exists but does not carry every [reference field](#joins-across-collections) the declaration asks for (a -`joinable` added to a schema whose index predates it), the run **fails**, naming -the drop-and-rebuild that fixes it. Without that it would index and commit -happily and then 400 on every join query: the values would be there, the -reference would not. Scoped to reference fields only – every other schema +`joinable` added to a schema whose index predates it), or every +[facet companion](#collection-schema-and-engine) its facets read (a `facetKeys` +policy added to a type this one references), the run **fails** at open – after +the lock, before any write – naming the drop-and-rebuild that fixes it. Without +that it would index and commit happily and then fail on every join query, or +on every facet over that reference: the values would be there, the reference +or the facet field would not. Rotating a pipeline version reprocesses datasets; +it does not recreate collections. Scoped to those two – every other schema difference is self-correcting. ### The join component is the unit of rebuild diff --git a/package-lock.json b/package-lock.json index 0869b717..5bf4cc4c 100644 --- a/package-lock.json +++ b/package-lock.json @@ -41776,6 +41776,7 @@ "typesense": "^3.0.6" }, "devDependencies": { + "n3": "^2.1.2", "testcontainers": "^12.1.0" }, "peerDependencies": { @@ -41783,6 +41784,20 @@ "@lde/pipeline": "^0.36.0" } }, + "packages/search-typesense/node_modules/n3": { + "version": "2.2.5", + "resolved": "https://registry.npmjs.org/n3/-/n3-2.2.5.tgz", + "integrity": "sha512-lR/0N7zVayC6N0uWqrMkPpJn8Up4+qBsjWiBnoOWWEi/ldTcygs2Dw1iAgozzDXLW/fcxXFfbMzpGHlc/1MBLQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "buffer": "^6.0.3", + "readable-stream": "^4.0.0" + }, + "engines": { + "node": ">=12.0" + } + }, "packages/search/node_modules/n3": { "version": "2.1.2", "resolved": "https://registry.npmjs.org/n3/-/n3-2.1.2.tgz", diff --git a/packages/search-typesense/package.json b/packages/search-typesense/package.json index 8725fb1a..4dc2d187 100644 --- a/packages/search-typesense/package.json +++ b/packages/search-typesense/package.json @@ -31,6 +31,7 @@ "typesense": "^3.0.6" }, "devDependencies": { + "n3": "^2.1.2", "testcontainers": "^12.1.0" }, "peerDependencies": { diff --git a/packages/search-typesense/src/collection-definition.ts b/packages/search-typesense/src/collection-definition.ts index f5069b19..61386ea1 100644 --- a/packages/search-typesense/src/collection-definition.ts +++ b/packages/search-typesense/src/collection-definition.ts @@ -39,11 +39,19 @@ export interface CollectionDefinitionOptions { /** * The Search Schema the type belongs to – required when the type surfaces an * inline reference (whose nested fields are declared from the - * {@link ReferenceType} the schema resolves) or declares a joinable + * {@link ReferenceType} the schema resolves), declares a joinable * reference (whose target collection the schema’s {@link joinGraph} * resolves). A type doing neither needs no schema, so a caller passes none; * a type that does fails here rather than building a collection that * silently omits the nesting or the reference. + * + * Also what resolves the facet policy a facetable reference inherits from + * the type it names ({@link physicalFields}): without the schema the field + * itself is declared the facet and no `${name}_facet` companion is, the same + * reading the projection makes without one. Whether a policy applies cannot + * be told without the schema, so this is not guarded: a deployment that + * declares one and builds its collections schema-less meets the engine’s own + * error on the first facet query, not a silently narrowed-to-nothing facet. */ readonly schema?: SearchSchema; /** Snowball stemming locale for non-localized searchable fields (e.g. `en`). @@ -233,7 +241,7 @@ function typesenseFields( if (nested !== undefined) { return nestedFields(field.name, field, nested, schema as SearchSchema); } - const names = physicalFields(field); + const names = physicalFields(field, schema); if (field.kind === 'text') { const locales = field.locales; const displayPattern = displayFieldPattern(field); @@ -281,7 +289,10 @@ function typesenseFields( { name: field.name, type: valueType, - facet: field.facetable ?? false, + // A reference inheriting a facet policy facets its companion below, and + // the field itself stays a plain stored value – which is also what keeps + // a membership filter on it exact (`compileMembership`). + facet: names.facet === field.name, sort: field.sortable ?? false, // A `required` field is non-optional; so is the `default_sorting_field`, // which Typesense requires to be present. Everything else may be absent. @@ -302,6 +313,18 @@ function typesenseFields( }), }); } + // The `${name}_facet` companion of a reference inheriting a facet policy: + // the admitted subset of the field’s values, and the only one of the two + // the engine facets. Optional, because a document none of whose values the + // policy admits carries no companion at all. + if (names.facet !== undefined && names.facet !== field.name) { + fields.push({ + name: names.facet, + type: valueType, + facet: true, + optional: true, + }); + } return fields; } diff --git a/packages/search-typesense/src/in-place-rebuild.ts b/packages/search-typesense/src/in-place-rebuild.ts index 6baafcc3..c9646e13 100644 --- a/packages/search-typesense/src/in-place-rebuild.ts +++ b/packages/search-typesense/src/in-place-rebuild.ts @@ -1,6 +1,7 @@ import type { Client } from 'typesense'; import type { CollectionFieldSchema } from 'typesense/lib/Typesense/Collection.js'; -import type { SearchType } from '@lde/search'; +import type { SearchSchema, SearchType } from '@lde/search'; +import { facetableFields, physicalFields } from '@lde/search/adapter'; import type { Dataset } from '@lde/dataset'; import type { DatasetOutcome, @@ -75,9 +76,10 @@ export interface InPlaceRebuildOptions extends RebuildOptions { * `RebuildAlreadyRunning` when another rebuild holds it) and creates * the collection on demand from the {@link SearchType} plus the two * bookkeeping fields. A collection that already exists is used as it is, with - * one exception: it must carry every reference field the declaration asks for, - * or the run fails with the drop-and-rebuild that fixes it - * ({@link assertReferencesPresent}). + * one exception: it must carry every reference field the declaration asks for + * and every facet companion its facets read, or the run fails with the + * drop-and-rebuild that fixes it + * ({@link assertReferencesAndFacetCompanionsPresent}). * * Document ids must be unique per (source, entity) – the caller keys them – * or documents from different sources overwrite each other. @@ -109,7 +111,10 @@ export class InPlaceRebuild< options: InPlaceRebuildOptions = {}, ) { assertNoReservedFields(searchType, [SOURCE_FIELD, LAST_SEEN_FIELD]); - assertSweepableProvenanceField(searchType, { requireFacetable: true }); + assertSweepableProvenanceField(searchType, { + requireFacetable: true, + schema: options.schema, + }); this.sourceField = provenanceField(searchType); const { maxSweepableSources = DEFAULT_MAX_SWEEPABLE_SOURCES, @@ -151,9 +156,15 @@ export class InPlaceRebuild< }; }); // A collection this run did NOT create may predate a `joinable` being - // declared, and a reference cannot be added to it here. + // declared, or a facet policy on a type this one references – and + // neither a reference nor a facet companion can be added to it here. if (!created) { - await assertReferencesPresent(this.client, name, declaredFields); + await assertReferencesAndFacetCompanionsPresent( + this.client, + name, + declaredFields, + facetCompanionsOf(this.searchType, definitionOptions.schema), + ); } const importer = new BatchImporter>( @@ -285,35 +296,71 @@ export class InPlaceRebuild< * component’s collections come into existence WITH their references, and never * acquire them later. * - * Deliberately scoped to reference fields only. Every other schema difference - * is self-correcting (a new plain field simply starts being written) or + * The facet companion of a reference inheriting a facet policy + * (`RootType.facetKeys` on the type it names) is the other field a live + * collection cannot acquire: the documents carry `${name}_facet`, the + * collection does not declare it, and every facet on the field fails with + * *could not find a facet field* – on every type referencing the policy’s + * type, until someone drops the collection by hand. Rotating a pipeline + * version reprocesses datasets; it does not recreate collections. So it is + * checked here too, with the same instruction. + * + * Deliberately scoped to those two. Every other schema difference is + * self-correcting (a new plain field simply starts being written) or * harmless, and general drift detection is a feature of its own. */ -async function assertReferencesPresent( +async function assertReferencesAndFacetCompanionsPresent( client: Client, name: string, fields: readonly CollectionFieldSchema[], + facetCompanions: readonly string[], ): Promise { - const declared = fields.filter((field) => field.reference !== undefined); - if (declared.length === 0) { + const references = fields.filter((field) => field.reference !== undefined); + if (references.length === 0 && facetCompanions.length === 0) { return; } const existing = new Map( (await client.collections(name).retrieve()).fields.map((field) => [ field.name, - field.reference, + field, ]), ); - const missing = declared.filter( - (field) => existing.get(field.name) !== field.reference, + const missingReferences = references.filter( + (field) => existing.get(field.name)?.reference !== field.reference, ); - if (missing.length > 0) { + if (missingReferences.length > 0) { throw new Error( - `Typesense collection “${name}” exists without the reference field(s) ${missing + `Typesense collection “${name}” exists without the reference field(s) ${missingReferences .map((field) => `“${field.name}” → “${field.reference as string}”`) .join( ', ', )} its search type declares. A reference cannot be added to a live collection here: drop “${name}” (and the collections that join to it) and let the next run rebuild them.`, ); } + const missingCompanions = facetCompanions.filter( + (companion) => existing.get(companion)?.facet !== true, + ); + if (missingCompanions.length > 0) { + throw new Error( + `Typesense collection “${name}” exists without the facet field(s) ${missingCompanions + .map((companion) => `“${companion}”`) + .join( + ', ', + )} its search type facets on – the companion of a reference to a type that declares a facet policy (“facetKeys”). A facet field cannot be added to a live collection here: drop “${name}” (and the collections that join to it) and let the next run rebuild them.`, + ); + } +} + +/** The `${name}_facet` companions a type’s facets read instead of the fields + * themselves – one per facetable reference inheriting a facet policy + * ({@link physicalFields}). */ +function facetCompanionsOf( + searchType: SearchType, + schema: SearchSchema | undefined, +): readonly string[] { + return facetableFields(searchType).flatMap((field) => { + // A facetable field always has a facet field. + const facet = physicalFields(field, schema).facet as string; + return facet === field.name ? [] : [facet]; + }); } diff --git a/packages/search-typesense/src/query-compiler.ts b/packages/search-typesense/src/query-compiler.ts index 86ddbca4..e4720078 100644 --- a/packages/search-typesense/src/query-compiler.ts +++ b/packages/search-typesense/src/query-compiler.ts @@ -7,6 +7,7 @@ import { type RootType, type SearchField, type SearchQuery, + type SearchSchema, type SearchType, type Sort, } from '@lde/search'; @@ -64,6 +65,16 @@ export interface BuildSearchParamsOptions { * range count is still safe. */ readonly maxFacetValues?: number; + /** + * The Search Schema the searched type belongs to – what resolves the facet + * policy a reference inherits from the type it names + * (`physicalFields(field, schema).facet`): the `${name}_facet` companion a + * facet on such a field reads, and the membership operator a filter on it + * compiles to. Left unset, a facet reads the field itself – which, against a + * collection built with the schema, is a field the engine does not facet. + * Through the engine it is always set. + */ + readonly schema?: SearchSchema; /** * Called for each `where` clause that does not compile as the caller wrote * it – either because it states no constraint (an empty `in` list, a `range` @@ -119,7 +130,7 @@ export function buildSearchParams( params.sort_by = sortBy; } if (query.facets.length > 0) { - params.facet_by = compileFacetBy(query.facets, searchType); + params.facet_by = compileFacetBy(query.facets, searchType, options.schema); if (options.maxFacetValues !== undefined) { params.max_facet_values = options.maxFacetValues; } @@ -130,20 +141,26 @@ export function buildSearchParams( /** * The `facet_by` clause. A facet on a numeric field that declares * {@link SearchField.facetRanges} faceted into those fixed half-open `[min, max)` - * bins (a histogram); every other facet is a plain per-value facet on its field - * name. Typesense range syntax is already start-inclusive/end-exclusive, so the + * bins (a histogram); every other facet is a plain per-value facet on the + * field the declaration facets ({@link physicalFields} – the field itself, or + * the `${name}_facet` companion of a reference inheriting a facet policy). + * Typesense range syntax is already start-inclusive/end-exclusive, so the * declared bounds pass straight through with no boundary fix-up. */ function compileFacetBy( facets: readonly string[], searchType: SearchType, + schema: SearchSchema | undefined, ): string { return facets .map((name) => { const field = fieldNamed(searchType, name); - return field !== undefined && isRangeFacet(field) + if (field === undefined) { + return name; + } + return isRangeFacet(field) ? compileRangeFacet(field.name, field.facetRanges) - : name; + : (physicalFields(field, schema).facet ?? name); }) .join(','); } @@ -310,7 +327,7 @@ function compileCriterion( ): string | typeof VACUOUS | typeof UNUSABLE { const on = criterion.on ?? []; if (on.length === 0) { - return compileLeaf(criterion, searchType); + return compileLeaf(criterion, searchType, options.schema); } // A joined criterion constrains a document in ANOTHER collection, so the leaf // is compiled against the type that path reaches and then wrapped, one @@ -329,7 +346,7 @@ function compileCriterion( collections.push(resolved.collection); target = resolved.searchType; } - const leaf = compileLeaf(criterion, target); + const leaf = compileLeaf(criterion, target, options.schema); // A vacuous leaf states no constraint on the referent, so the join as a whole // states none either – the reading, and so the outcome, passes straight // through the hops. @@ -347,6 +364,7 @@ function compileCriterion( function compileLeaf( criterion: Criterion, searchType: SearchType, + schema: SearchSchema | undefined, ): string | typeof VACUOUS | typeof UNUSABLE { // `id` is the Typesense document key, not a declared field. Exact `:=` // membership, like a non-facet field ({@link compileMembership}), so an IRI @@ -370,7 +388,7 @@ function compileLeaf( } if ('in' in criterion) { return criterion.in.length > 0 - ? compileMembership(field, criterion.in) + ? compileMembership(field, criterion.in, schema) : VACUOUS; } if ('range' in criterion) { @@ -382,15 +400,22 @@ function compileLeaf( /** * A membership clause. A non-facet (tokenized) field uses the exact `:=` * operator so an IRI cannot partial-match on a shared path segment. + * + * *Non-facet* is the **engine’s** facet status, not the declaration’s: a + * reference inheriting a facet policy is `facetable` in the schema while the + * engine facets its companion and stores the field itself plain – and a + * filter on such a field is exactly what the policy promises stays whole, so + * it must compile exact, not tokenized. */ function compileMembership( field: SearchField, values: readonly string[], + schema: SearchSchema | undefined, ): string { const list = `[${values.map(escapeFilterValue).join(',')}]`; - return field.facetable !== true - ? `${field.name}:=${list}` - : `${field.name}:${list}`; + return physicalFields(field, schema).facet === field.name + ? `${field.name}:${list}` + : `${field.name}:=${list}`; } /** An inclusive Typesense range clause, or `undefined` when neither bound is diff --git a/packages/search-typesense/src/rebuild-support.ts b/packages/search-typesense/src/rebuild-support.ts index 8e449d56..261ec836 100644 --- a/packages/search-typesense/src/rebuild-support.ts +++ b/packages/search-typesense/src/rebuild-support.ts @@ -1,9 +1,10 @@ import type { Client } from 'typesense'; -import type { SearchType } from '@lde/search'; +import type { SearchSchema, SearchType } from '@lde/search'; import { datasetField, isInternalField, joinGraph, + physicalFields, referenceFields, } from '@lde/search/adapter'; import { @@ -148,14 +149,22 @@ export function assertNoReservedFields( * selection, which carries raw dataset IRIs, so a transformed value would * match nothing and the sweep would silently stop deleting; * - **`facetable`** where the writer enumerates the indexed sources by faceting - * it (the In-place writer does; Blue/green only ever filters by a known IRI). + * it (the In-place writer does; Blue/green only ever filters by a known IRI) + * – and faceted **on its own name**: a reference inheriting a facet policy + * from the type it names facets a companion holding only the admitted keys + * (`physicalFields(field, schema).facet`), so enumerating the sources + * through it would miss every dataset the policy excludes. `schema` is what + * resolves that; without one no policy is visible, as nowhere else. * * Thrown at writer construction, so a declaration that cannot be swept fails * before a run touches the index rather than at the sweep, after the writes. */ export function assertSweepableProvenanceField( searchType: SearchType, - options: { readonly requireFacetable: boolean }, + options: { + readonly requireFacetable: boolean; + readonly schema?: SearchSchema; + }, ): void { const field = datasetField(searchType); if (field === undefined || isInternalField(field)) { @@ -176,6 +185,13 @@ export function assertSweepableProvenanceField( problems.push( 'it is not facetable, and the membership sweep enumerates the indexed datasets by faceting it', ); + } else if ( + options.requireFacetable && + physicalFields(field, options.schema).facet !== field.name + ) { + problems.push( + 'it inherits a facet policy from the type it names, so the engine facets only the datasets the policy admits, and the membership sweep enumerates the indexed datasets by faceting it', + ); } if (problems.length > 0) { throw new Error( diff --git a/packages/search-typesense/src/search.ts b/packages/search-typesense/src/search.ts index 80554133..fc28f265 100644 --- a/packages/search-typesense/src/search.ts +++ b/packages/search-typesense/src/search.ts @@ -25,6 +25,7 @@ import { assertTypeInSchema, assertValidQuery, displayLangOf, + facetableFields, fieldNamed, isInlineReference, isRangeFacet, @@ -168,6 +169,7 @@ export function createTypesenseSearchEngine< const joins = joinGraph(schema); const searchParamsOptions: BuildSearchParamsOptions = { ...options, + schema, joinTargetFor: (from, path) => { // Both lookups always hit: `assertValidQuery` runs first and rejects // every path this could not resolve, and each type resolved a collection @@ -209,6 +211,26 @@ export function createTypesenseSearchEngine< ), ]), ); + // Typesense reports a facet under the field it faceted, which for a reference + // inheriting a facet policy is the `${name}_facet` companion, not the + // declared field. Resolved back to the declared name per type ONCE, from the + // same `physicalFields` that named it, and applied to every response at the + // boundary – before `labelLookupGroups` (which finds a facet’s label source + // by declared name) and `parseSearchResponse` (which files its counts under + // it). Either consumer meeting the engine name would silently lose the + // facet’s labels, or its counts, or both. + const declaredFacetNames = new Map>( + [...schema.values()].map((searchType) => [ + searchType.class, + new Map( + facetableFields(searchType).map((field) => [ + // A facetable field always has a facet field. + physicalFields(field, schema).facet as string, + field.name, + ]), + ), + ]), + ); // The distinct source collections per type, for the cached path – fixed at // construction, so no per-search dedup dance. const distinctLabelSources = new Map( @@ -386,10 +408,13 @@ export function createTypesenseSearchEngine< // would overflow Typesense's 4000-char query-string limit. POST puts it // in the body, so the filter is bounded by the caller's request, not by // the URL. - const response = await performSingleSearch( - client, - collections.get(searchType.class) as string, - params, + const response = withDeclaredFacetNames( + await performSingleSearch( + client, + collections.get(searchType.class) as string, + params, + ), + declaredFacetNames.get(searchType.class) as ReadonlyMap, ); const labels = await resolveLabels(cachedLabelsPromise, () => labelLookupGroups( @@ -446,7 +471,7 @@ export function createTypesenseSearchEngine< // compiles as facet-only regardless of what it carries: no hits // (per_page 0) and no ordering – nothing is transferred or sorted that // this method cannot return. - const { results } = (await client.multiSearch.perform({ + const { results: rawResults } = (await client.multiSearch.perform({ searches: dispatched.map(({ query }) => ({ collection, ...buildSearchParams( @@ -458,6 +483,14 @@ export function createTypesenseSearchEngine< })) as { results: readonly (TypesenseSearchResponse | TypesenseErrorEntry)[]; }; + const declaredNames = declaredFacetNames.get( + searchType.class, + ) as ReadonlyMap; + const results = rawResults.map((result) => + 'error' in result + ? result + : withDeclaredFacetNames(result, declaredNames), + ); // One label lookup serves every successful facet result in the batch. const responses = results.filter( (result): result is TypesenseSearchResponse => !('error' in result), @@ -632,6 +665,29 @@ function labelLookupGroups( })); } +/** + * The response with each facet filed under the declared field name its engine + * facet field stands for – the one place an engine facet name is translated, + * so that nothing downstream of it meets one. + */ +function withDeclaredFacetNames( + response: TypesenseSearchResponse, + declaredNames: ReadonlyMap, +): TypesenseSearchResponse { + if (response.facet_counts === undefined) { + return response; + } + return { + ...response, + facet_counts: response.facet_counts.map((facet) => ({ + ...facet, + // Every facet the engine reports was asked for by a declared, facetable + // field (`assertValidQuery`), and the map holds each of those. + field_name: declaredNames.get(facet.field_name) as string, + })), + }; +} + /** * Dispatch one search as a single-entry `multi_search` (POST). `multi_search` * reports a failed entry inline instead of rejecting the call, so the entry is @@ -765,6 +821,9 @@ export interface TypesenseSearchResponse { readonly found: number; readonly hits?: readonly { readonly document: Record }[]; readonly facet_counts?: readonly { + /** The field faceted – by the time a response reaches + * {@link parseSearchResponse}, the **declared** field name: the engine + * maps a facet companion’s name back at the response boundary. */ readonly field_name: string; readonly counts: readonly { readonly value: string; diff --git a/packages/search-typesense/test/facet-policy.test.ts b/packages/search-typesense/test/facet-policy.test.ts new file mode 100644 index 00000000..43488a12 --- /dev/null +++ b/packages/search-typesense/test/facet-policy.test.ts @@ -0,0 +1,419 @@ +import { afterAll, beforeAll, describe, expect, it } from 'vitest'; +import { Parser } from 'n3'; +import type { Client } from 'typesense'; +import { + defineSearchType, + projectRoots, + searchSchema, + type RootType, + type SearchDocument, + type SearchEngine, + type SearchQuery, +} from '@lde/search'; +import { describeSearchEngineContract } from '@lde/search/testing'; +import { buildCollectionDefinition } from '../src/collection-definition.js'; +import { InPlaceRebuild } from '../src/in-place-rebuild.js'; +import { buildSearchParams } from '../src/query-compiler.js'; +import { createTypesenseSearchEngine } from '../src/search.js'; +import { fakeTypesenseClient, labelLookup } from './fake-typesense-client.js'; +import { makeRunContext } from './helpers.js'; +import { TypesenseContainer } from './typesense-container.js'; + +// A facet policy (`RootType.facetKeys`) declared once on `Place` and inherited +// by every facetable reference that names it: the engine facets a +// `${name}_facet` companion holding the admitted keys, the field itself stays +// whole, and a filter on it stays exact. + +const SCHEMA_ORG = 'https://schema.org/'; +const GEONAMES = 'https://sws.geonames.org/'; +const VENLO = `${GEONAMES}2745706`; +/** The deployment’s one predicate, built into `pick` and the policy alike. */ +const isCovered = (iri: string) => iri.startsWith(GEONAMES); + +const place = defineSearchType({ + name: 'Place', + class: `${SCHEMA_ORG}Place`, + labelField: 'name', + key: { field: '_sameAs', pick: (candidates) => candidates.find(isCovered) }, + facetKeys: { only: isCovered }, + fields: [ + { + name: 'name', + kind: 'text', + path: `<${SCHEMA_ORG}name>`, + locales: ['und'], + output: true, + searchable: { weight: 1 }, + }, + { + name: '_sameAs', + kind: 'reference', + array: true, + path: `<${SCHEMA_ORG}sameAs>`, + }, + ], +}); + +const work = defineSearchType({ + name: 'CreativeWork', + class: `${SCHEMA_ORG}CreativeWork`, + fields: [ + { + name: 'title', + kind: 'text', + path: `<${SCHEMA_ORG}name>`, + locales: ['und'], + output: true, + searchable: { weight: 1 }, + }, + { + name: 'locationCreated', + kind: 'reference', + path: `<${SCHEMA_ORG}locationCreated>`, + facetable: true, + filterable: true, + output: true, + ref: { strategy: 'lookup', target: 'Place' }, + }, + { + name: 'about', + kind: 'reference', + array: true, + path: `<${SCHEMA_ORG}about>`, + facetable: true, + filterable: true, + output: true, + ref: { strategy: 'idOnly' }, + labelSource: 'Place', + }, + { + // Facetable, but names no type: inherits no policy. + name: 'genre', + kind: 'keyword', + path: `<${SCHEMA_ORG}genre>`, + facetable: true, + filterable: true, + }, + ], +}); + +const schema = searchSchema(place, work); + +const baseQuery: SearchQuery = { + where: [], + orderBy: [], + limit: 10, + offset: 0, + facets: [], + locale: 'und', +}; + +describe('collection definition', () => { + const definition = buildCollectionDefinition(work, { schema }); + + it('facets the `_facet` companion of a reference inheriting a policy, not the field', () => { + expect(definition.fields).toContainEqual({ + name: 'locationCreated', + type: 'string', + facet: false, + sort: false, + optional: true, + }); + expect(definition.fields).toContainEqual({ + name: 'locationCreated_facet', + type: 'string', + facet: true, + optional: true, + }); + expect(definition.fields).toContainEqual({ + name: 'about', + type: 'string[]', + facet: false, + sort: false, + optional: true, + }); + expect(definition.fields).toContainEqual({ + name: 'about_facet', + type: 'string[]', + facet: true, + optional: true, + }); + // A facet inheriting no policy is declared as before. + expect(definition.fields).toContainEqual({ + name: 'genre', + type: 'string', + facet: true, + sort: false, + optional: true, + }); + }); + + it('facets the field itself when built without the schema, as the projection reads it', () => { + // No schema, no visible policy – the same degraded reading the projection + // makes when it cannot re-key a reference. Whether a policy applies cannot + // be told here, so a schema-less build is not refused. + const schemaless = buildCollectionDefinition(work); + + expect(schemaless.fields).toContainEqual( + expect.objectContaining({ name: 'locationCreated', facet: true }), + ); + expect(schemaless.fields?.map((field) => field.name)).not.toContain( + 'locationCreated_facet', + ); + }); +}); + +describe('query compiler', () => { + it('facets the companion, and filters the field itself exactly', () => { + const params = buildSearchParams( + { + ...baseQuery, + facets: ['locationCreated', 'about', 'genre'], + where: [ + { or: [{ field: 'locationCreated', in: ['https://a/place/2'] }] }, + { or: [{ field: 'genre', in: ['map'] }] }, + ], + }, + work, + { schema }, + ); + + expect(params.facet_by).toBe('locationCreated_facet,about_facet,genre'); + // The field is `facetable` in the schema but not a facet in the engine, so + // the tokenized `:` operator – which would partial-match an IRI on a + // shared path prefix – is wrong for it; a plain facet keeps `:`. + expect(params.filter_by).toBe( + 'locationCreated:=[`https://a/place/2`] && genre:[`map`]', + ); + }); +}); + +describe('response boundary', () => { + const facetCounts = [ + { + // The engine reports the facet under the field it faceted. + field_name: 'locationCreated_facet', + counts: [{ value: VENLO, count: 2 }], + }, + { field_name: 'genre', counts: [{ value: 'map', count: 1 }] }, + ]; + const labels = labelLookup({ [VENLO]: { name_und: 'Venlo' } }); + + it('files a policy facet under the declared field, with its labels', async () => { + const fake = fakeTypesenseClient({ + searchResponse: { found: 0, hits: [], facet_counts: facetCounts }, + multiSearch: labels, + }); + const engine = createTypesenseSearchEngine(fake.client, schema); + + const result = await engine.search(work, { + ...baseQuery, + facets: ['locationCreated', 'genre'], + }); + + expect(result.facets).toEqual({ + locationCreated: [{ value: VENLO, count: 2, label: { und: ['Venlo'] } }], + genre: [{ value: 'map', count: 1 }], + }); + }); + + it('maps every response of a facet batch the same way', async () => { + const fake = fakeTypesenseClient({ + searchResponse: { found: 0, hits: [], facet_counts: facetCounts }, + multiSearch: labels, + }); + const engine = createTypesenseSearchEngine(fake.client, schema); + + const [outcome] = await engine.searchFacets(work, [ + { ...baseQuery, facets: ['locationCreated'] }, + ]); + + expect(outcome).toEqual({ + facets: { + locationCreated: [ + { value: VENLO, count: 2, label: { und: ['Venlo'] } }, + ], + genre: [{ value: 'map', count: 1 }], + }, + }); + }); +}); + +describe('facet policy end to end (integration)', () => { + const container = new TypesenseContainer(); + let client: Client; + let engine: SearchEngine; + + // The extraction’s output, as the pipeline hands it to the projection: each + // value under its field’s IR Alias, and – for a reference into the keyed + // Place – the referent’s key candidates one hop further, read by the + // referring type’s query. An aligned place, publisher A’s Kessel, a second + // place under it sharing its IRI as a path prefix, and publisher B’s own, + // unaligned Kessel. + const VENLO_NODE = 'https://a/place/1'; + const KESSEL = 'https://a/place/2'; + const KESSEL_CENTRUM = 'https://a/place/2/centrum'; + const OTHER_KESSEL = 'https://b/place/7'; + const placeQuads = ` + <${VENLO_NODE}> "Venlo" . + <${VENLO_NODE}> <${VENLO}> . + <${KESSEL}> "Kessel" . + <${KESSEL_CENTRUM}> "Kessel-Centrum" . + <${OTHER_KESSEL}> "Kessel" . + `; + const workQuads = ` + "Map of Venlo" . + <${VENLO_NODE}> . + <${VENLO_NODE}> . + <${KESSEL}> . + "Kessel church" . + <${KESSEL}> . + "Kessel square" . + <${KESSEL_CENTRUM}> . + <${VENLO_NODE}> . + "The other Kessel" . + <${OTHER_KESSEL}> . + <${VENLO_NODE}> <${VENLO}> . + `; + + async function project( + nTriples: string, + roots: readonly string[], + searchType: RootType, + ): Promise { + const quads = new Parser({ format: 'N-Triples' }).parse(nTriples); + const documents: SearchDocument[] = []; + for await (const document of projectRoots( + quads, + roots, + schema, + searchType, + )) { + documents.push(document); + } + return documents; + } + + beforeAll(async () => { + client = await container.start(); + await client + .collections() + .create(buildCollectionDefinition(place, { schema })); + await client + .collections() + .create(buildCollectionDefinition(work, { schema })); + await client + .collections('places') + .documents() + .import( + await project( + placeQuads, + [VENLO_NODE, KESSEL, KESSEL_CENTRUM, OTHER_KESSEL], + place, + ), + { action: 'create' }, + ); + await client + .collections('creative_works') + .documents() + .import( + await project( + workQuads, + [ + 'https://a/work/1', + 'https://a/work/2', + 'https://a/work/3', + 'https://b/work/4', + ], + work, + ), + { action: 'create' }, + ); + engine = createTypesenseSearchEngine(client, schema); + }, 120_000); + + afterAll(async () => { + await container.stop(); + }); + + describeSearchEngineContract( + 'TypesenseSearchEngine with a facet policy', + () => engine, + ); + + it('buckets every facet referencing Place on the admitted key only, labelled', async () => { + const result = await engine.search(work, { + ...baseQuery, + facets: ['locationCreated', 'about'], + }); + + expect(result.total).toBe(4); + // Three local Kessels – two of them one place by name – and not one bucket + // among them; the aligned place is the facet, labelled from its document. + expect(result.facets.locationCreated).toEqual([ + { value: VENLO, count: 1, label: { und: ['Venlo'] } }, + ]); + expect(result.facets.about).toEqual([ + { value: VENLO, count: 2, label: { und: ['Venlo'] } }, + ]); + }); + + it('keeps the field whole: a local place still displays and still filters exactly', async () => { + const result = await engine.search(work, { + ...baseQuery, + where: [{ or: [{ field: 'locationCreated', in: [KESSEL] }] }], + }); + + // Kessel, not Kessel-Centrum, whose IRI extends Kessel’s by a path segment. + expect(result.total).toBe(1); + expect(result.hits.map((hit) => hit.id)).toEqual(['https://a/work/2']); + expect(result.hits[0].document.locationCreated).toEqual({ + id: KESSEL, + label: { und: ['Kessel'] }, + }); + }); + + it('fails an in-place run at open over a collection that predates the policy', async () => { + // What a deployment that adds `facetKeys` to a type meets under in-place + // rebuild: the collections referencing that type were created without the + // companion, and the documents are about to carry it. Loud, at run open, + // naming the drop that fixes it – never a silently broken facet. + const prePolicy = buildCollectionDefinition(work, { schema }); + await client.collections().create({ + ...prePolicy, + name: 'pre_policy_works', + fields: (prePolicy.fields ?? []) + .filter((field) => !field.name.endsWith('_facet')) + .map((field) => + field.name === 'locationCreated' || field.name === 'about' + ? { ...field, facet: true } + : field, + ), + }); + const writer = new InPlaceRebuild(client, work, { + schema, + collectionNameFor: () => 'pre_policy_works', + }); + + await expect(writer.openRun(makeRunContext())).rejects.toThrow( + /exists without the facet field\(s\) “locationCreated_facet”, “about_facet”.*drop “pre_policy_works”/s, + ); + }); + + it('creates an in-place collection with the companions, and reopens it', async () => { + const writer = new InPlaceRebuild(client, work, { + schema, + collectionNameFor: () => 'fresh_works', + }); + + const run = await writer.openRun(makeRunContext()); + await run.abort(new Error('done')); + const created = await client.collections('fresh_works').retrieve(); + expect(created.fields.map((field) => field.name)).toContain( + 'locationCreated_facet', + ); + // The companion is there, so the next run opens. + const reopened = await writer.openRun(makeRunContext()); + await reopened.abort(new Error('done')); + }); +}); diff --git a/packages/search-typesense/test/query-compiler.test.ts b/packages/search-typesense/test/query-compiler.test.ts index feaed297..123f6236 100644 --- a/packages/search-typesense/test/query-compiler.test.ts +++ b/packages/search-typesense/test/query-compiler.test.ts @@ -533,6 +533,16 @@ describe('buildSearchParams', () => { ).toBe('keyword,format'); }); + it('passes a facet it cannot resolve through by name, as it does a where field', () => { + // The compiler does not validate: `assertValidQuery` rejects an unknown or + // non-facetable facet before a query is dispatched, so here the name + // simply travels as written rather than being dropped or re-derived. + expect( + buildSearchParams({ ...base, facets: ['nope', 'title'] }, schema) + .facet_by, + ).toBe('nope,title'); + }); + it('facets a range field into its declared half-open bins, open ends blank', () => { // Typesense range syntax is start-inclusive/end-exclusive, so the declared // `[min, max)` bounds pass straight through; the open-ended bin leaves the diff --git a/packages/search-typesense/test/rebuild-support.test.ts b/packages/search-typesense/test/rebuild-support.test.ts index d0b7c686..69689da1 100644 --- a/packages/search-typesense/test/rebuild-support.test.ts +++ b/packages/search-typesense/test/rebuild-support.test.ts @@ -1,5 +1,5 @@ import { describe, expect, it } from 'vitest'; -import type { SearchType } from '@lde/search'; +import { defineSearchType, searchSchema, type SearchType } from '@lde/search'; import { assertNoReservedFields, assertSweepableProvenanceField, @@ -183,6 +183,52 @@ describe('assertSweepableProvenanceField', () => { assertSweepableProvenanceField(notFacetable, { requireFacetable: false }), ).not.toThrow(); }); + + it('rejects a dataset field whose facet a policy narrows, where the writer enumerates by it', () => { + // The engine would facet only the admitted datasets, so the sweep would + // never see – and never reconcile – the ones the policy excludes. + const dataset = defineSearchType({ + name: 'Dataset', + class: 'https://example.org/Dataset', + facetKeys: { only: (id) => id.startsWith('https://registry/') }, + fields: [ + { + name: 'label', + kind: 'text', + locales: ['und'], + output: true, + searchable: { weight: 1 }, + }, + ], + }); + const object = defineSearchType({ + name: 'Object', + class: 'https://example.org/Object', + fields: [ + { + name: 'dataset', + kind: 'reference', + from: 'dataset', + facetable: true, + ref: { strategy: 'lookup', target: 'Dataset' }, + }, + ], + }); + const schema = searchSchema(dataset, object); + + expect(() => + assertSweepableProvenanceField(object, { + requireFacetable: true, + schema, + }), + ).toThrow(/inherits a facet policy/); + expect(() => + assertSweepableProvenanceField(object, { + requireFacetable: false, + schema, + }), + ).not.toThrow(); + }); }); describe('stampDocuments', () => { diff --git a/packages/search-typesense/vite.config.ts b/packages/search-typesense/vite.config.ts index c039f775..3129a3af 100644 --- a/packages/search-typesense/vite.config.ts +++ b/packages/search-typesense/vite.config.ts @@ -22,14 +22,14 @@ export default mergeConfig( // by-name map for the shared `rootTypeNamed`: one COVERED arrow fewer // over one function fewer, which lowers the ratio without uncovering // anything. - functions: 98.92, - lines: 99.31, + functions: 98.97, + lines: 99.33, // Re-anchored for the projection lookup: its guards against a // projection naming what no lookup reaches are unreachable through // the port, since `assertValidQuery` rejects such a query first. // They hold for a direct caller, and are exercised as one. - branches: 94.63, - statements: 99.33, + branches: 94.83, + statements: 99.35, }, }, },