From 7b5a4d30d6aa98927464c00ecddfd25590e9af16 Mon Sep 17 00:00:00 2001 From: David de Boer Date: Fri, 21 Aug 2026 15:46:33 +0200 Subject: [PATCH 1/4] feat(search): key a root type on a declared field MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A Root Type is keyed on the node’s IRI unless it names a `key` field to read the key from. A reference to such a type stores the target’s key, which is what a `lookup`’s `target` and an `idOnly`’s `labelSource` already promise. - `KeyField` and `RootType.key`: `field` names a declared, path-bearing, array, non-inline reference field whose values are the candidates; `pick` chooses among them, defaulting to the first, and `undefined` keeps the node’s own IRI. - `documentKeyOf` is that rule in one exported function – transform, IRI filter, dedupe, sort, pick – so the projection and a deployment’s transform read the same answer. A `pick` returning a value no candidate offered throws, naming the node and the candidates. - `rootTypeNamed` resolves the type a declaration points at, replacing the by-name map `assertResolvableLabelSources` kept for itself. - The projection keys a root through `documentKeyOf`, and re-keys the values of a reference that names a keyed target. A reference naming no target, a derived one, and a projection run without a schema are left alone. - Validation rejects a key naming an unknown, non-reference, path-less, single-valued or inline field, a non-function `pick`, and a key on a Reference Type. - ADR 22, plus a “Document key” section in the search reference and the re-emit rule for a replacing transform in the indexer reference. --- ...022-key-a-root-type-on-a-declared-field.md | 152 ++++++++ docs/reference/search-indexer.md | 8 + docs/reference/search.md | 97 ++++- packages/search/src/adapter.ts | 2 + packages/search/src/index.ts | 1 + packages/search/src/project.ts | 110 +++++- packages/search/src/schema.ts | 265 ++++++++++++- packages/search/test/project.test.ts | 347 ++++++++++++++++++ packages/search/test/schema.test.ts | 220 +++++++++++ packages/search/vite.config.ts | 2 +- 10 files changed, 1182 insertions(+), 22 deletions(-) create mode 100644 docs/decisions/0022-key-a-root-type-on-a-declared-field.md diff --git a/docs/decisions/0022-key-a-root-type-on-a-declared-field.md b/docs/decisions/0022-key-a-root-type-on-a-declared-field.md new file mode 100644 index 00000000..4c352e6e --- /dev/null +++ b/docs/decisions/0022-key-a-root-type-on-a-declared-field.md @@ -0,0 +1,152 @@ +# 22. Key a root type on a declared field + +Date: 2026-08-21 + +## Status + +Accepted + +Extends [ADR 20](./0020-resolve-a-references-fields-from-the-targets-own-collection.md), +whose contract – a reference holds ids of documents in the target’s collection – +is what makes reference rewriting a consequence rather than a new rule. Relates +to [ADR 11](./0011-decouple-rdf-depth-from-the-api-surface.md) (inline +references, the other way a referent reaches a document) and +[ADR 12](./0012-bound-memory-by-the-unit-of-work-not-the-input.md). + +## Context + +A search document is keyed on the IRI of the node it was projected from. That is +the right default and wrong for a whole class of profiles: where a publisher +models an entity as its own node and states, in the graph, that the node _is_ +some agreed term – SCHEMA-AP-NDE § 3.4 says exactly this for every +`DefinedTerm` – two publishers describing one place produce two documents, two +facet buckets and two entity pages, and neither can be reached from the other. + +The fact that decides the key is in the graph, one hop from every node that +needs it, and already readable as an ordinary declared field. What was missing +was a way for the schema to say **which field holds the key**. + +The alternative on the table was to move root selection into the deployment – a +`DISTINCT`/`BIND` selector minting roots that carry no triples, and a reader +transform told the batch’s bindings so it could mint content for them. That +works, but it puts a generic seam in `@lde/pipeline`, makes every keyed root an +empty CONSTRUCT, and writes the deployment’s rule twice: once as a SPARQL +`FILTER` in the selector and once in JavaScript in each transform that has to +rewrite a reference, with a standing obligation that the two never disagree. + +## Decision + +A Root Type is keyed on the node’s IRI unless it names a **`key` field** to read +the key from. A reference to such a type stores the target’s key. + +```ts +key: { + field: '_sameAs', + pick: (candidates) => candidates.find(isGeoNames) ?? candidates.find(isCovered), +} +``` + +`key.field` names a declared field of the type: a `path`-bearing, `array` +reference field that is not `inline`. Naming a **declared field** rather than a +path or a pseudo-field is the whole point – the extraction branch already exists, +a reader transform that repairs reference values covers the candidates the day +the field is declared, and the field’s own `transform` is where IRI +normalisation lives, so two spellings of one IRI become one candidate before +anything chooses between them. + +`key.pick` is the deployment’s choice among the candidates, defaulting to the +first. LDE never inspects an IRI’s shape – it asks. + +### The guards + +Candidates reach `pick` transformed, IRI-filtered, deduplicated and **sorted**, +so the default is deterministic whatever order the CONSTRUCT returned them in; +`pick` must return one of them or `undefined`, so a key is always either an IRI +the graph offered for that node or the node’s own, never one invented in +between. `pick` must be pure: the same function keys the document and every +reference to it, so an impure one could key those two differently and leave a +reference dangling. + +`documentKeyOf` is that whole rule in one exported function, so a transform that +needs a node’s key before the projection runs reads the same answer the +projection will. + +### The boundary, for keys and for joins alike + +Only a reference that **names** its target – a `lookup`’s `target`, an +`idOnly`’s `labelSource` – is re-keyed. That is the same line a join draws, and +for the same reason: naming the target is what asserts that the field holds ids +of that collection’s documents. An `idOnly` reference with no label source, and +a `derive`d reference over a raw internal path, never claimed as much, so +nothing rewrites them. + +## Consequences + +- **Several nodes with one key are one document.** That is what a document key + means; the writer upserts by `id`. A deployment that wants the merged document + to carry particular content attaches a transform; one that does not gets + last-writer-wins, exactly as a shared entity across datasets behaves today. + The projection still emits one document per distinct root – folding them is + the writer’s upsert, not the projection’s. +- **Shared documents become the norm rather than the edge case**, which + multiplies the exposure of the single-valued provenance stamp: when one + contributor leaves the run while another that still references the entity is + skipped, the membership sweep can delete a document that is still referenced. + The design does not make this worse per document, but it makes it common. +- **The CONSTRUCT grows** one `OPTIONAL` hop per reference into a keyed type, and + the frame carries the referent’s key-field values. `OPTIONAL` rather than + conjoined, so an unaligned referent keeps its row – and its own IRI – instead + of dropping out of the extraction. +- **A transform that replaces a root’s quads must re-emit the key field.** The + existing rule – a field the document needs must be in the stream – applied to + one more field; a transform that only adds never meets it. Left as a + convention rather than a guarantee: the structural alternative (reading the key + off the reader’s raw output before transforms run) touches `@lde/pipeline`’s + runner, the one package this design otherwise leaves alone. +- **The key is assigned before any `derive` runs**, so a derive sees the key and + never the node IRI. A deployment that wants the node IRI declares a plain + `idOnly` reference over the same path. +- **A cross-dataset node reference does not resolve.** A work in dataset A + pointing at a local node in dataset B gets no candidates – the hop runs against + A’s distribution – so it stores the node IRI and dangles against B’s keyed + document. Publishers reference other publishers through `sameAs` rather than + directly, and such a reference is already unresolvable today for every purpose + but labels. +- `@lde/pipeline`, `@lde/search-indexer`, `@lde/search-typesense` and the API + packages are untouched: the change is a schema member, the projection, and one + hop in the extraction generator. A schema declaring no `key` extracts, + projects, indexes and queries exactly as before. + +## Rejected + +**A deployment-supplied selector plus `bindings` on reader transforms.** Correct, +but it solves _ids come from the selector_ by moving the selector to the +deployment rather than by letting the schema say what a key is. It reaches the +goal at the cost of a generic seam in `@lde/pipeline`, an empty CONSTRUCT per +keyed root, and a rule every consumer must restate in two languages. + +**Framing the member in Linked Data sameness vocabulary** (`identity: { +alignment, canonical }`). Same mechanics, but it made LDE state rules – “several +nodes merge”, “references are rewritten” – that in search-document terms are +just what keys already do. _Identity_, _alignment_, _canonical_ and _authority_ +are the deployment’s words, and stay in the deployment’s schema. + +**`key: { path }` with a pseudo-field.** A pseudo-field is invisible to +everything that works on declared fields: a reader transform that enumerates +reference-field aliases would not see it, so a publisher writing the alignment as +a typed literal would silently yield no candidates; IRI normalisation would have +nowhere to live, so two spellings of one IRI would silently fail to merge; the +extraction would need an extra root branch; and the alias would become exported +surface. Naming a declared field removes all of it. + +**A derivable `id`.** A `derive` over the document key re-keys the document but +cannot reach a reference on another type: the hop to the referent’s key field has +to be _extracted_, and only a declaration the extraction generator can see makes +that happen. + +**Putting `key` in the indexer’s extensions** rather than in the schema. It keeps +the predicate in app code next to the deployment’s other logic, but the +extraction generator and the projection both need it, so it would have to be +threaded through the stage factory and the pipeline – the ripple the schema +placement avoids. A transform reads the declaration off the loaded schema, so +nothing is lost. diff --git a/docs/reference/search-indexer.md b/docs/reference/search-indexer.md index 124eb342..cdfb0afa 100644 --- a/docs/reference/search-indexer.md +++ b/docs/reference/search-indexer.md @@ -245,6 +245,14 @@ declare a `path`.** Projection skips a field with neither a `path` nor a `derive`, so a transform-minted IR Alias is otherwise never read and the field ships empty. +The same rule bites once more where a root type declares a +[document key](./search#document-key): **a transform that replaces a root’s +quads must re-emit the key field.** The key is read off the projected frame like +any other field, so a replaced root that drops it is keyed on its node IRI +instead – and every reference to it, which is keyed independently, then points +at a document that was never written. A transform that only adds quads (the +documented use) never meets this. + ## Compose it yourself Reach for this only when the deployment needs something `createSearchIndexer` diff --git a/docs/reference/search.md b/docs/reference/search.md index d8cfd5e3..7eea7e61 100644 --- a/docs/reference/search.md +++ b/docs/reference/search.md @@ -438,6 +438,100 @@ resolves labels from; without it, everything keeps serving `label`. A facet bucket’s `label` is unaffected: it is per-facet-field, and a per-type name would make the bucket shape non-uniform. +### Document key + +A Root Type is keyed on the node’s IRI unless it names a **`key` field** to read +the key from. `key` has the shape of `labelField` – _which field is the label_ – +and says _which field holds the key_: + +```ts +const place = defineSearchType({ + name: 'Place', + class: `${SCHEMA}Place`, + labelField: 'name', + key: { + field: '_sameAs', + // A preference order, not a filter: GeoNames first, then any other source + // an authority can resolve; nothing matched keeps the publisher’s node. + pick: (candidates) => + candidates.find(isGeoNames) ?? candidates.find(isCovered), + }, + fields: [ + { + name: 'name', + kind: 'text', + locales: ['nl', 'und'], + path: `<${SCHEMA}name>`, + output: true, + searchable: { weight: 3 }, + }, + { + // Internal (no role): read for the key, pruned before the writer. + name: '_sameAs', + kind: 'reference', + array: true, + path: `<${SCHEMA}sameAs>`, + transform: normaliseIri, + }, + ], +}); +``` + +- **`key.field`** names a declared field of the type: a `path`-bearing, `array` + reference field that is not `inline`. Its values are the key candidates. + Because it is an ordinary field, everything that already applies to fields + applies to the candidates – it is extracted like any field, a reader transform + that repairs reference values covers it, and the field’s own `transform` is + where IRI normalisation lives, so two spellings of one IRI become one + candidate before anything chooses between them. +- **`key.pick`** chooses among the candidates: `(candidates) => key | undefined`, + where `undefined` keeps the node’s own IRI. It defaults to the first candidate, + and is not consulted for a node whose key field is empty. +- **The guards.** Candidates reach `pick` transformed, IRI-filtered, deduplicated + and **sorted**, so the default is deterministic whatever order the CONSTRUCT + returned them in. `pick` must return one of them or `undefined` – anything else + throws, naming the node and its candidates – so a key is always an IRI the + graph offered for that node. And `pick` must be **pure**: the same function + keys the document and every reference to it. + +`documentKeyOf` (from `@lde/search/adapter`) is that whole rule in one function, +for a transform that needs to know a node’s key before the projection runs. + +Two consequences are not new rules, only what a document key already means: + +- **Several nodes with one key are one document.** The writer upserts by `id`. A + deployment that wants the merged document to carry particular content attaches + a transform; one that does not gets last-writer-wins, exactly as a shared + entity across datasets behaves today. +- **A reference stores the target’s key.** A `lookup`’s `target` and an + `idOnly`’s `labelSource` already mean _this field holds ids of documents in + that collection_ – the contract a label lookup and a join rely on – so a + reference that names a keyed target stores the referent’s key rather than its + node IRI. A reference that names no target is never rewritten: it claimed + nothing about a collection. The extraction adds one `OPTIONAL` hop per such + reference to read the referent’s key field, so an unaligned referent keeps its + row and its own IRI. + +What LDE deliberately does not know is _why_ one candidate is preferred over +another, and what a merged document should say. **LDE decides the key; the +deployment decides the content.** + +Two things to keep in mind when declaring one: + +- A transform that **replaces** a root’s quads must re-emit the key field – the + existing rule that a field the document needs must be in the stream, applied to + one more field. A transform that only adds never meets it. +- The key is assigned **before any `derive` runs**, so a derive sees the key and + never the node IRI. A deployment that wants the node IRI declares a plain + `idOnly` reference over the same path. +- A work in dataset A referencing a node **in dataset B** gets no candidates (the + hop runs against A’s distribution), so it stores the node IRI and does not + resolve against B’s keyed document. Publishers reference each other through + `sameAs` rather than directly, and such a reference is already unresolvable + today for every purpose but labels. + +See [ADR 22](../decisions/0022-key-a-root-type-on-a-declared-field). + ### Projecting what a lookup carries A `lookup` declares no field list. What it _fetches_ is named per query, by a @@ -728,7 +822,8 @@ reported to `onIgnoredFilter`, since neither compiled as written. ### Lookup by IRI -Every type is filterable on **`id`** – the document’s IRI – without declaring +Every type is filterable on **`id`** – the document’s key, which is the node’s +IRI unless the type declares a [`key` field](#document-key) – without declaring it, and every surface returns it. It is the one field no `SearchType` declares, because every indexed thing already carries it: it is the hit’s identity (`SearchHit.id`), not a value in its `ResultDocument`. `searchSchema()` rejects diff --git a/packages/search/src/adapter.ts b/packages/search/src/adapter.ts index c04ed658..137cd622 100644 --- a/packages/search/src/adapter.ts +++ b/packages/search/src/adapter.ts @@ -24,6 +24,7 @@ export { nestedFieldName, referenceFields, referenceTypeNamed, + rootTypeNamed, inlineFramingDepth, fieldNamed, datasetField, @@ -33,6 +34,7 @@ export { labelFieldOf, labelFieldNameOf, labelSourceNameOf, + documentKeyOf, DEFAULT_LABEL_FIELD, isRangeFacet, isAbsoluteIri, diff --git a/packages/search/src/index.ts b/packages/search/src/index.ts index 49c61e98..8debc101 100644 --- a/packages/search/src/index.ts +++ b/packages/search/src/index.ts @@ -49,6 +49,7 @@ export type { ReferenceType, RootTypeOf, SearchTypeIssue, + KeyField, SearchSchema, FacetRange, ProjectionValue, diff --git a/packages/search/src/project.ts b/packages/search/src/project.ts index 75a95716..83858ec4 100644 --- a/packages/search/src/project.ts +++ b/packages/search/src/project.ts @@ -8,14 +8,18 @@ import { import { assertTypeInSchema, displayFieldName, + documentKeyOf, + fieldNamed, inlineFramingDepth, irAlias, isAbsoluteIri, isInternalField, isInlineReference, isoToUnixSeconds, + labelSourceNameOf, physicalFields, referenceTypeNamed, + rootTypeNamed, type KeywordField, type ProjectionValue, type ReferenceField, @@ -147,7 +151,7 @@ function projectFields( schema: SearchSchema | undefined, context: ProjectionContext, ): ProjectedNode { - const id = documentKey(node); + const id = documentIdOf(node, searchType); const document: ProjectedNode = id === undefined ? {} : { id }; for (const field of searchType.fields) { applyField(document, node, field, searchType, schema, context); @@ -169,6 +173,46 @@ function documentKey(node: FramedNode): string | undefined { return typeof id === 'string' && isAbsoluteIri(id) ? id : undefined; } +/** + * The `id` a node is projected under: its own IRI, unless its type declares a + * {@link RootType.key} – then the key its key field carries + * ({@link documentKeyOf}). The candidates are read off the frame like any other + * field value, under the key field’s own {@link irAlias IR Alias}, because the + * key field *is* an ordinary declared field: the extraction already emits it, + * and a reader transform that repairs reference values has already run on it. + * + * The key is assigned before any `derive` runs, so a derive sees the key and + * never the node IRI; a deployment that wants the node IRI declares a plain + * `idOnly` reference over the same path. + */ +function documentIdOf( + node: FramedNode, + searchType: SearchType, +): string | undefined { + const nodeIri = documentKey(node); + if (nodeIri === undefined || searchType.key === undefined) { + return nodeIri; + } + return documentKeyOf( + searchType, + nodeIri, + keyCandidatesOf(node, searchType, searchType.key.field), + ); +} + +/** The raw values a node carries under a type’s key field – untransformed, as + * {@link documentKeyOf} expects them. Empty for a node the key field matched + * nothing on. The field itself is guaranteed declared by `searchSchema`, which + * is what validates a `key` at all. */ +function keyCandidatesOf( + node: FramedNode, + searchType: SearchType, + name: string, +): readonly string[] { + const keyField = fieldNamed(searchType, name) as SearchField; + return irisOf(node, irAlias(searchType, keyField)); +} + /** * Project a single type over a known set of `roots` – the per-type, roots-given * projection. The roots are supplied by the caller (the pipeline selector) @@ -193,8 +237,11 @@ export async function* projectRoots( const index = buildSubjectIndex(quads); // Distinct roots only. A selector may return an IRI more than once – a // non-`DISTINCT` `SELECT` over a one-to-many join yields the same subject per - // matched row – and a repeated root would otherwise frame and emit a - // duplicate document under the same `id`. + // matched row – and framing a root twice would do the same work twice for one + // node. Distinct roots may still legitimately share an `id`, where the type + // declares a `key` two of them carry: that is what a document key means, and + // it is the writer’s upsert – not the projection – that folds them into one + // document. const depth = inlineFramingDepth(schema, searchType); for await (const node of frameSubjects(index, [...new Set(roots)], depth)) { yield projectDocument(node, searchType, schema, context); @@ -277,7 +324,11 @@ function applyField( case 'keyword': return applyFacet(document, literalsOf(node, alias), field); case 'reference': - return applyFacet(document, irisOf(node, alias), field); + return applyFacet( + document, + referenceValues(node, alias, field, schema), + field, + ); case 'integer': return setNumber( document, @@ -309,6 +360,57 @@ function applyField( } } +/** + * The values a non-inline reference stores: the referents’ IRIs, **or their + * document keys** when the type the reference names declares a + * {@link RootType.key}. A `lookup`’s `target` and an `idOnly`’s `labelSource` + * both mean *this field holds ids of documents in that collection* (ADR 20) – + * the fact a label lookup and a join rely on – so storing the node IRI where + * the target keys on something else would break an invariant the schema already + * has. Rewriting a reference is LDE keeping it, not a new rule. + * + * That is also why the boundary is *naming the target*: a reference that names + * none ({@link labelSourceNameOf} → `undefined`) never claimed to hold a + * collection’s ids, and a `derive`d one produces its own values rather than + * reading a referent, so neither is re-keyed. + * + * The referent’s candidates are in the frame because framing already embeds a + * referenced node’s own triples at one hop, and the extraction adds the key + * field to that hop. A projection run without a `schema` cannot resolve the + * target, so it leaves the values as they are – exactly as it leaves an inline + * reference unprojected. + */ +function referenceValues( + node: FramedNode, + alias: string, + field: ReferenceField, + schema: SearchSchema | undefined, +): readonly string[] { + const targetName = labelSourceNameOf(field); + if (schema === undefined || targetName === undefined) { + return irisOf(node, alias); + } + // Guaranteed declared: `searchSchema` validates that every named target + // resolves, and the projection only ever runs against a type of its schema. + const target = rootTypeNamed(schema, targetName) as RootType; + if (target.key === undefined) { + return irisOf(node, alias); + } + const keyFieldName = target.key.field; + return valuesOf(node, alias) + .map((value) => { + const iri = iriString(value); + return iri === undefined + ? undefined + : documentKeyOf( + target, + iri, + isObject(value) ? keyCandidatesOf(value, target, keyFieldName) : [], + ); + }) + .filter((value): value is string => value !== undefined); +} + /** * Project a text field. **Display** (when `output`) preserves *every* language * present – one label per language (accents preserved, untagged under `und`), diff --git a/packages/search/src/schema.ts b/packages/search/src/schema.ts index bdfa755b..4b1b5dc5 100644 --- a/packages/search/src/schema.ts +++ b/packages/search/src/schema.ts @@ -353,6 +353,57 @@ export interface SearchTypeBase { readonly fields: readonly SearchField[]; } +/** + * Which declared field of a {@link RootType} holds its **document key**, when + * that key is not the node’s own IRI. It has the shape of + * {@link SearchTypeBase.labelField} – *which field is the label* – and says + * *which field holds the key*: a statement about the search document, never + * about the world, so the words a deployment reaches for (identity, alignment, + * canonical, authority) stay in the deployment’s own schema comments. + * + * Two things that already hold of keys then hold here too, without a new rule: + * several nodes sharing a key are **one document** (the writer upserts by `id`, + * so a deployment that wants particular content on the merged document attaches + * a transform, and one that does not gets last-writer-wins), and a reference + * into a keyed type stores the **target’s key** – a `lookup`/`labelSource` + * already means *this field holds ids of documents in that collection* + * (ADR 20), which storing the node IRI would break. + * + * See [ADR 22](../../docs/decisions/0022-key-a-root-type-on-a-declared-field.md). + */ +export interface KeyField { + /** + * The `name` of a declared field of this type whose values are the key + * candidates: a path-bearing, `array` {@link ReferenceField} that is not + * `inline` ({@link searchSchema} validates all four). Being an ordinary field + * is the point – its extraction branch already exists, a reader transform + * that repairs reference values covers it the day it is declared, and its own + * {@link ReferenceField.transform} is where IRI normalisation lives, so two + * spellings of one IRI become one candidate before anything chooses among + * them. + * + * A transform that **replaces** a root’s quads must re-emit this field, the + * existing rule that *a field the document needs must be in the stream* + * applied to one more field; a transform that only adds never meets it. + */ + readonly field: string; + /** + * Choose the key among the candidates – the key field’s values after its + * `transform`, IRI-filtered, deduplicated and sorted, so the default is + * deterministic whatever order the CONSTRUCT returned them in. Return one of + * them, or `undefined` to keep the node’s own IRI; anything else throws at + * projection. Not consulted for a node whose key field is empty. + * + * Must be **pure**: the same function keys the document and every reference + * to it, so a `pick` that consulted the network or the clock could key the + * two differently and leave a reference dangling. LDE never inspects an + * IRI’s shape – it asks. + * + * @default the first candidate + */ + readonly pick?: (candidates: readonly string[]) => string | undefined; +} + /** * A **Root Type**: a {@link SearchType} that is indexed. It declares a `class`, * roots are selected for it, a Writer owns a collection for it, and the @@ -365,6 +416,9 @@ export interface RootType extends SearchTypeBase { * key a {@link SearchSchema} maps this type under. Its presence is what makes * a type a Root Type – and so what gives it a collection. */ readonly class: string; + /** Which declared field holds this type’s document key, when it is not the + * node’s own IRI ({@link KeyField}, {@link documentKeyOf}). */ + readonly key?: KeyField; } /** @@ -383,6 +437,10 @@ export interface ReferenceType extends SearchTypeBase { * {@link SearchField} discriminates by `kind`: an indexed Reference Type * fails to compile, not at run time. */ readonly class?: never; + /** A Reference Type has no document key to declare: it is nested inside its + * referrer rather than keyed in a collection of its own. `never` for the + * same reason `class` is. */ + readonly key?: never; } /** @@ -452,6 +510,21 @@ const referenceTypesBySchema = new WeakMap< ReadonlyMap >(); +/** + * The Root Type name index each {@link SearchSchema} carries alongside its + * class-keyed map. A schema is keyed by `class` because that is what a selector + * and a Writer address it by, while every declaration that points at another + * type – a `lookup`’s `target`, an `idOnly`’s `labelSource`, a join edge – + * names it. Kept beside the map rather than rebuilt per consumer, so the + * projection, the label-source validation and an adapter’s lookup resolve a + * target the same way ({@link rootTypeNamed}) instead of each maintaining a + * by-name map of its own. + */ +const rootTypesBySchema = new WeakMap< + SearchSchema, + ReadonlyMap +>(); + /** Whether a declared type is a {@link RootType} (declares a `class`) rather * than a {@link ReferenceType}. */ function isRootType(searchType: SearchType): searchType is RootType { @@ -505,14 +578,21 @@ export function searchSchema( ); assertResolvableInlineReferences(types, referenceTypes); assertServiceableNestedFields(referenceTypes); - assertResolvableLabelSources(types); - // The one blessed cast: only this validated constructor mints the brand. + // The one blessed cast: only this validated constructor mints the brand. Built + // BEFORE the schema-wide assertions that resolve a type by name, so they can + // use `rootTypeNamed` – the same reading the projection and an adapter's + // lookup use – rather than a by-name map of their own. The schema stays local + // until every assertion has passed, so an invalid one is never observable. + const rootTypes = types.filter(isRootType); const schema = new Map( - types - .filter(isRootType) - .map((searchType) => [searchType.class, searchType]), + rootTypes.map((searchType) => [searchType.class, searchType]), ) as unknown as SearchSchema; referenceTypesBySchema.set(schema, referenceTypes); + rootTypesBySchema.set( + schema, + new Map(rootTypes.map((searchType) => [searchType.name, searchType])), + ); + assertResolvableLabelSources(types, schema); // Build the join graph eagerly and discard it: it is cached per schema, and // building it is what enforces the schema-wide join rules (one joinable // reference per target, every target an indexed Root Type, no cycles). A @@ -535,6 +615,25 @@ export function referenceTypeNamed( return referenceTypesBySchema.get(schema)?.get(name); } +/** + * The {@link RootType} a declaration names – a `lookup`’s `target`, an + * `idOnly`’s {@link ReferenceField.labelSource} ({@link labelSourceNameOf}), a + * join edge – or `undefined` when the schema declares no Root Type by that + * name. The one reading of *which type does this point at*, so the projection + * (which re-keys a reference through its target’s {@link KeyField}), the + * label-source validation and an adapter’s lookup cannot resolve a target + * differently. + * + * Complements {@link referenceTypeNamed}: the two name indexes are disjoint, + * because {@link searchSchema} rejects a name declared twice. + */ +export function rootTypeNamed( + schema: SearchSchema, + name: string, +): RootType | undefined { + return rootTypesBySchema.get(schema)?.get(name); +} + /** Whether a field is an inline reference – a {@link ReferenceField} whose * `ref` carries its referent’s projected fields ({@link ReferenceType}). */ export function isInlineReference( @@ -801,15 +900,16 @@ export function labelSourceNameOf(field: ReferenceField): string | undefined { * can actually serve labels ({@link labelFieldOf}). Checked schema-wide, * because a single declaration cannot see its siblings. * - * That leaves only a {@link RootType}, without naming one: a label field is - * `searchable`, and {@link assertServiceableNestedFields} already rejects a - * `searchable` field on a Reference Type – so a Reference Type can never serve - * labels, and a resolved label always has a collection to come from. + * Resolved through {@link rootTypeNamed}, which is exactly the set that can + * serve labels: a label field is `searchable`, and + * {@link assertServiceableNestedFields} – which runs first – already rejects a + * `searchable` field on a Reference Type, so a resolved label always has a + * collection to come from. */ -function assertResolvableLabelSources(types: readonly SearchType[]): void { - const byName = new Map( - types.map((searchType) => [searchType.name, searchType]), - ); +function assertResolvableLabelSources( + types: readonly SearchType[], + schema: SearchSchema, +): void { for (const searchType of types) { for (const field of searchType.fields) { const labelSource = (field as { readonly labelSource?: string }) @@ -836,7 +936,7 @@ function assertResolvableLabelSources(types: readonly SearchType[]): void { if (sourceName === undefined) { continue; } - const source = byName.get(sourceName); + const source = rootTypeNamed(schema, sourceName); if (source === undefined) { throw new Error( `Reference “${searchType.name}.${field.name}” names unknown label source “${sourceName}”; declare a SearchType with that name.`, @@ -884,7 +984,14 @@ export interface SearchTypeIssue { | 'joinable-not-allowed' | 'joinable-without-label-source' | 'joinable-with-inline-ref' - | 'reserved-field-name'; + | 'reserved-field-name' + | 'key-field-unknown' + | 'key-field-not-reference' + | 'key-field-without-path' + | 'key-field-not-array' + | 'key-field-inline' + | 'key-pick-not-a-function' + | 'key-not-allowed'; } /** @@ -996,7 +1103,10 @@ const LOCALE_PATTERN = /^[A-Za-z0-9]+(-[A-Za-z0-9]+)*$/; * field has exactly one), is not an `inline` reference (a projection value is * a bare IRI, with no referent to carry fields from), and no two fields * declare the same projection value – which would leave a consumer reading - * the dataset off a declaration no rule picks between. + * the dataset off a declaration no rule picks between; + * - a {@link RootType.key} names a declared, `path`-bearing, `array`, + * non-`inline` `reference` field of this type, and its `pick` is a function + * ({@link keyIssues}); a {@link ReferenceType} declares no key at all. * * Pure and total: returns every issue rather than throwing; * {@link assertValidSearchType} is the throwing entry point. @@ -1159,6 +1269,59 @@ export function validateSearchType( } } } + issues.push(...keyIssues(searchType)); + return issues; +} + +/** + * The issues a type’s {@link RootType.key} declaration carries, each filed + * under the field name the declaration names – so a message reads like every + * other one, naming the field the rule is about rather than a member. + * + * The key field must be an ordinary declared field the extraction already + * reads and the projection already applies a `transform` to: a `reference` + * (candidates are IRIs), with a `path` (a key read from the graph, not one + * derived from a document that is keyed already), `array` (a node may offer + * several candidates, and a single-valued field would drop all but the first + * before `pick` ever saw them), and not `inline` (which carries the referent’s + * fields rather than its IRI, so it offers no candidate at all). + */ +function keyIssues(searchType: SearchType): readonly SearchTypeIssue[] { + const key = (searchType as RootType).key; + if (key === undefined) { + return []; + } + const issue = (reason: SearchTypeIssue['reason']) => ({ + field: key.field, + reason, + }); + // A Reference Type is nested inside its referrer, never keyed in a collection + // of its own – so a key on one states a rule nothing could apply. + if (searchType.class === undefined) { + return [issue('key-not-allowed')]; + } + const issues: SearchTypeIssue[] = []; + if (key.pick !== undefined && typeof key.pick !== 'function') { + issues.push(issue('key-pick-not-a-function')); + } + const field = fieldNamed(searchType, key.field); + if (field === undefined) { + issues.push(issue('key-field-unknown')); + return issues; + } + if (field.kind !== 'reference') { + issues.push(issue('key-field-not-reference')); + return issues; + } + if (field.path === undefined) { + issues.push(issue('key-field-without-path')); + } + if (field.array !== true) { + issues.push(issue('key-field-not-array')); + } + if (field.ref?.strategy === 'inline') { + issues.push(issue('key-field-inline')); + } return issues; } @@ -1386,6 +1549,76 @@ export function fieldNamed( return searchType.fields.find((field) => field.name === name); } +/** + * The **document key** of one node of `searchType`: what the projection writes + * as the document’s `id`, and what a reference into this type stores. + * + * The whole rule, in one place, so the projection and any transform that needs + * to know a node’s key read the same answer: + * + * 1. a type declaring no {@link RootType.key} keys on `nodeIri` – the node’s own + * IRI, as every type does today; + * 2. otherwise the key field’s `rawValues` become **candidates**: its + * {@link ReferenceField.transform} (where IRI normalisation lives, so two + * spellings of one IRI become one candidate), then the + * {@link isAbsoluteIri} filter, then dedupe and sort – so the default is + * deterministic whatever order the CONSTRUCT returned them in; + * 3. no candidate keys on `nodeIri`, without consulting {@link KeyField.pick}; + * 4. otherwise {@link KeyField.pick} chooses (defaulting to the first + * candidate), and `undefined` keeps `nodeIri`. + * + * So a key is always either the node’s own IRI or an IRI the graph offered for + * that node – never one invented in between. A `pick` returning anything else + * throws, naming the node and the candidates: it is a bug in a pure function + * that keys both the document and every reference to it, and letting it through + * would key those two differently. + * + * @param nodeIri the node’s own IRI – the key it falls back to + * @param rawValues the key field’s values as the frame carries them, untransformed + */ +export function documentKeyOf( + searchType: RootType, + nodeIri: string, + rawValues: readonly string[], +): string { + const key = searchType.key; + if (key === undefined) { + return nodeIri; + } + // Guaranteed a declared reference field by `searchSchema`, which is what + // validates a `key` at all. + const { transform } = fieldNamed(searchType, key.field) as ReferenceField; + const candidates = [ + ...new Set( + (transform === undefined ? rawValues : rawValues.map(transform)).filter( + isAbsoluteIri, + ), + ), + ].sort(); + if (candidates.length === 0) { + return nodeIri; + } + const picked = (key.pick ?? firstCandidate)(candidates); + if (picked === undefined) { + return nodeIri; + } + if (!candidates.includes(picked)) { + throw new Error( + `The “pick” of “${searchType.name}.key” returned “${picked}” for <${nodeIri}>, which is not among its candidates (${candidates + .map((candidate) => `<${candidate}>`) + .join( + ', ', + )}); a key must be one of them, or “undefined” to keep the node’s own IRI.`, + ); + } + return picked; +} + +/** The {@link KeyField.pick} a type that declares none falls back to. */ +function firstCandidate(candidates: readonly string[]): string { + return candidates[0]; +} + /** * The **IR Alias** predicate for a field: `urn:lde:‹SearchType.name›/‹field.name›`. * The extraction CONSTRUCT emits a field’s value under this minted predicate, and diff --git a/packages/search/test/project.test.ts b/packages/search/test/project.test.ts index ff347185..75a16517 100644 --- a/packages/search/test/project.test.ts +++ b/packages/search/test/project.test.ts @@ -1646,3 +1646,350 @@ describe('projection-time values', () => { }); }); }); + +describe('document keys (a type keyed on a declared field)', () => { + const SCHEMA_ORG = 'https://schema.org/'; + const GEONAMES = 'https://sws.geonames.org/'; + const WIKIDATA = 'https://www.wikidata.org/entity/'; + + /** A deployment’s IRI normalisation: one spelling per alignment target, so two + * publishers writing the same place alike key it alike. */ + const normaliseIri = (iri: string) => + iri.replace(/^http:/, 'https:').replace(/\/$/, ''); + + const place = defineSearchType({ + name: 'Place', + class: `${SCHEMA_ORG}Place`, + labelField: 'name', + // A preference order, not a filter: GeoNames first, then any other covered + // source; nothing matched keeps the publisher’s own node. + key: { + field: '_sameAs', + pick: (candidates) => + candidates.find((iri) => iri.startsWith(GEONAMES)) ?? + candidates.find((iri) => iri.startsWith(WIKIDATA)), + }, + fields: [ + { + name: 'name', + kind: 'text', + path: `<${SCHEMA_ORG}name>`, + locales: ['und'], + output: true, + searchable: { weight: 1 }, + }, + { + // Internal: extracted and read for the key, then pruned before the writer. + name: '_sameAs', + kind: 'reference', + array: true, + path: `<${SCHEMA_ORG}sameAs>`, + transform: normaliseIri, + }, + ], + }); + + const work = defineSearchType({ + name: 'CreativeWork', + class: `${SCHEMA_ORG}CreativeWork`, + fields: [ + { + name: 'locationCreated', + kind: 'reference', + path: `<${SCHEMA_ORG}locationCreated>`, + facetable: true, + output: true, + ref: { strategy: 'lookup', target: 'Place' }, + }, + { + name: 'about', + kind: 'reference', + array: true, + path: `<${SCHEMA_ORG}about>`, + facetable: true, + labelSource: 'Place', + }, + { + // Names no target, so it never claimed to hold a collection’s ids. + name: 'unnamed', + kind: 'reference', + array: true, + path: `<${SCHEMA_ORG}mentions>`, + facetable: true, + }, + ], + }); + + const keyedSchema = searchSchema(place, work); + const placeKey = (field: string) => alias('Place', field); + const workKey = (field: string) => alias('CreativeWork', field); + + /** A framed place node, with the raw values the CONSTRUCT emitted for it. */ + const placeNode = (id: string, ...sameAs: string[]) => ({ + '@id': id, + [placeKey('name')]: [{ '@value': 'Kessel' }], + ...(sameAs.length === 0 + ? {} + : { [placeKey('_sameAs')]: sameAs.map((iri) => ({ '@id': iri })) }), + }); + + it('keys a root on the field’s value, not on the node IRI', () => { + const document = projectDocument( + placeNode('https://ex/place/1', `${GEONAMES}2751283`), + place, + keyedSchema, + ); + + expect(document.id).toBe(`${GEONAMES}2751283`); + // The key field itself is internal, so it never reaches the writer. + expect(document).not.toHaveProperty('_sameAs'); + }); + + it('keeps the node’s own IRI when the key field is empty', () => { + expect( + projectDocument(placeNode('https://ex/place/1'), place, keyedSchema).id, + ).toBe('https://ex/place/1'); + }); + + it('keeps the node’s own IRI when pick declines every candidate', () => { + // A candidate in no source `pick` prefers: the publisher keeps its node. + expect( + projectDocument( + placeNode('https://ex/place/1', 'https://ex/other/1'), + place, + keyedSchema, + ).id, + ).toBe('https://ex/place/1'); + }); + + it('applies the key field’s transform before pick, so spellings merge', () => { + // Two publishers spelling one GeoNames IRI differently – `http://` and a + // trailing slash – project to ONE document key, which is what makes the + // writer’s upsert merge them. + const first = projectDocument( + placeNode('https://a/place/1', 'http://sws.geonames.org/2751283/'), + place, + keyedSchema, + ); + const second = projectDocument( + placeNode('https://b/place/9', `${GEONAMES}2751283`), + place, + keyedSchema, + ); + + expect(first.id).toBe(`${GEONAMES}2751283`); + expect(second.id).toBe(first.id); + }); + + it('sorts and dedupes the candidates, whatever order the CONSTRUCT returned', () => { + // `pick` here takes the FIRST candidate (the default), so the answer is + // stable only because the candidates are sorted before it sees them. + const unordered = defineSearchType({ + ...place, + name: 'AnyPlace', + class: 'urn:x:AnyPlace', + key: { field: '_sameAs' }, + }); + const anySchema = searchSchema(unordered); + const candidates = [`${WIKIDATA}Q1`, `${GEONAMES}1`, `${WIKIDATA}Q1`]; + const forward = { + '@id': 'https://ex/place/1', + [alias('AnyPlace', '_sameAs')]: candidates.map((iri) => ({ '@id': iri })), + }; + const reversed = { + '@id': 'https://ex/place/1', + [alias('AnyPlace', '_sameAs')]: [...candidates] + .reverse() + .map((iri) => ({ '@id': iri })), + }; + + expect(projectDocument(forward, unordered, anySchema).id).toBe( + `${GEONAMES}1`, + ); + expect(projectDocument(reversed, unordered, anySchema).id).toBe( + `${GEONAMES}1`, + ); + }); + + it('throws when pick returns a value that is not a candidate', () => { + // A key must be an IRI the graph offered for that node – the same pure + // function keys the document AND every reference to it, so a `pick` that + // invents one would key the two differently. + const invented = defineSearchType({ + ...place, + name: 'Invented', + class: 'urn:x:Invented', + key: { field: '_sameAs', pick: () => 'https://elsewhere/1' }, + }); + + expect(() => + projectDocument( + { + '@id': 'https://ex/place/1', + [alias('Invented', '_sameAs')]: [{ '@id': `${GEONAMES}1` }], + }, + invented, + searchSchema(invented), + ), + ).toThrow( + /returned “https:\/\/elsewhere\/1” for , which is not among its candidates \(\)/, + ); + }); + + it('stores a referent’s key on a lookup and on a labelSource reference', () => { + // Both name their target, which is what a `lookup`/`labelSource` already + // means: this field holds ids of documents in that collection. + const document = projectDocument( + { + '@id': 'https://ex/work/1', + [workKey('locationCreated')]: { + '@id': 'https://ex/place/1', + [placeKey('_sameAs')]: [{ '@id': `${GEONAMES}2751283` }], + }, + [workKey('about')]: [ + { + '@id': 'https://ex/place/2', + [placeKey('_sameAs')]: [{ '@id': `${WIKIDATA}Q1` }], + }, + ], + }, + work, + keyedSchema, + ); + + expect(document.locationCreated).toBe(`${GEONAMES}2751283`); + expect(document.about).toEqual([`${WIKIDATA}Q1`]); + }); + + it('stores the referent’s own IRI when it offers no key', () => { + // An unaligned place: the reference still resolves, against the document + // that place is written under. + const document = projectDocument( + { + '@id': 'https://ex/work/1', + [workKey('locationCreated')]: { '@id': 'https://ex/place/1' }, + // A reference naming no target is never re-keyed, even into a keyed type. + [workKey('unnamed')]: [ + { + '@id': 'https://ex/place/2', + [placeKey('_sameAs')]: [{ '@id': `${GEONAMES}1` }], + }, + ], + }, + work, + keyedSchema, + ); + + expect(document.locationCreated).toBe('https://ex/place/1'); + expect(document.unnamed).toEqual(['https://ex/place/2']); + }); + + it('re-keys a referent given as a bare IRI, and drops a value that is none', () => { + // `schema:sameAs` and friends range on `schema:URL`, which a source may emit + // as a literal – so a reference value may arrive as a bare string, carrying + // no candidates of its own; a value that is no IRI at all yields nothing, + // exactly as it does for an unkeyed reference. + const document = projectDocument( + { + '@id': 'https://ex/work/1', + [workKey('about')]: ['https://ex/place/3', { '@value': 'Kessel' }], + }, + work, + keyedSchema, + ); + + expect(document.about).toEqual(['https://ex/place/3']); + }); + + it('leaves a reference into an unkeyed target alone', () => { + const unkeyed = defineSearchType({ + name: 'Place', + class: `${SCHEMA_ORG}Place`, + labelField: 'name', + fields: place.fields, + }); + const document = projectDocument( + { + '@id': 'https://ex/work/1', + [workKey('locationCreated')]: { + '@id': 'https://ex/place/1', + [placeKey('_sameAs')]: [{ '@id': `${GEONAMES}1` }], + }, + }, + work, + searchSchema(unkeyed, work), + ); + + expect(document.locationCreated).toBe('https://ex/place/1'); + }); + + it('leaves references as they are when projected without a schema', () => { + // No schema, no way to resolve the target – the same graceful degradation + // an inline reference makes. + const document = projectDocument( + { + '@id': 'https://ex/work/1', + [workKey('locationCreated')]: { + '@id': 'https://ex/place/1', + [placeKey('_sameAs')]: [{ '@id': `${GEONAMES}1` }], + }, + }, + work, + ); + + expect(document.locationCreated).toBe('https://ex/place/1'); + }); + + it('assigns the key before any derive runs', () => { + const withDerive = defineSearchType({ + ...place, + name: 'DerivedPlace', + class: 'urn:x:DerivedPlace', + key: { field: '_sameAs' }, + fields: [ + ...place.fields, + { + name: 'keyedOn', + kind: 'keyword', + output: true, + derive: (document) => document.id, + }, + ], + }); + + expect( + projectDocument( + { + '@id': 'https://ex/place/1', + [alias('DerivedPlace', '_sameAs')]: [{ '@id': `${GEONAMES}1` }], + }, + withDerive, + searchSchema(withDerive), + ).keyedOn, + ).toBe(`${GEONAMES}1`); + }); + + it('lets two distinct roots project to one document key', async () => { + // Two publishers’ nodes for one place. The projection emits both documents, + // under one `id`; folding them is the writer’s upsert, not the projection’s. + const quads = new Parser({ format: 'N-Triples' }).parse(` + <${placeKey('_sameAs')}> <${GEONAMES}2751283> . + <${placeKey('_sameAs')}> <${GEONAMES}2751283> . + `); + + const documents: SearchDocument[] = []; + for await (const document of projectRoots( + quads, + ['https://a/place/1', 'https://b/place/9'], + keyedSchema, + place, + )) { + documents.push(document); + } + + expect(documents.map((document) => document.id)).toEqual([ + `${GEONAMES}2751283`, + `${GEONAMES}2751283`, + ]); + }); +}); diff --git a/packages/search/test/schema.test.ts b/packages/search/test/schema.test.ts index 67166e7e..50407fa8 100644 --- a/packages/search/test/schema.test.ts +++ b/packages/search/test/schema.test.ts @@ -4,6 +4,7 @@ import { assertValidSearchType, datasetField, displayFieldName, + documentKeyOf, displayFieldPattern, displayLangOf, facetableFields, @@ -22,12 +23,15 @@ import { physicalFields, referenceFields, referenceTypeNamed, + rootTypeNamed, searchableFields, searchSchema, sortableFields, unixSecondsToIso, validateSearchType, + type KeyField, type ReferenceStrategy, + type RootType, type SearchField, type SearchType, type TextField, @@ -1510,3 +1514,219 @@ describe('date storage codec', () => { expect(bce).toBeLessThan(ce); }); }); + +describe('key fields', () => { + const SCHEMA_ORG = 'https://schema.org/'; + const GEONAMES = 'https://sws.geonames.org/'; + + const sameAsField: SearchField = { + name: '_sameAs', + kind: 'reference', + array: true, + path: `<${SCHEMA_ORG}sameAs>`, + }; + const nameField: SearchField = { + name: 'name', + kind: 'text', + path: `<${SCHEMA_ORG}name>`, + locales: ['und'], + output: true, + searchable: { weight: 1 }, + }; + /** A `Place` keyed on its alignment target, with an overridable key. */ + const keyedPlace = (key: RootType['key'] = { field: '_sameAs' }) => + ({ + name: 'Place', + class: `${SCHEMA_ORG}Place`, + labelField: 'name', + key, + fields: [nameField, sameAsField], + }) as const satisfies SearchType; + + describe('declaration', () => { + it('accepts a key naming a path-bearing, array, non-inline reference field', () => { + expect(() => searchSchema(keyedPlace())).not.toThrow(); + expect(validateSearchType(keyedPlace())).toEqual([]); + }); + + it('rejects a key naming a field the type does not declare', () => { + expect(validateSearchType(keyedPlace({ field: 'absent' }))).toEqual([ + { field: 'absent', reason: 'key-field-unknown' }, + ]); + }); + + it('rejects a key on a field that is not a reference', () => { + // Candidates are IRIs, and the projection reads them as such. + expect(validateSearchType(keyedPlace({ field: 'name' }))).toEqual([ + { field: 'name', reason: 'key-field-not-reference' }, + ]); + }); + + it('rejects a key field with no path: a key is read from the graph', () => { + expect( + validateSearchType({ + ...keyedPlace(), + fields: [ + nameField, + { + name: '_sameAs', + kind: 'reference', + array: true, + derive: () => [], + }, + ], + }), + ).toEqual([{ field: '_sameAs', reason: 'key-field-without-path' }]); + }); + + it('rejects a single-valued key field: it would drop candidates unseen', () => { + // A node may offer several; a single-valued field keeps the first, + // whatever the CONSTRUCT’s order, so `pick` would never see the rest. + expect( + validateSearchType({ + ...keyedPlace(), + fields: [nameField, { ...sameAsField, array: undefined }], + }), + ).toEqual([{ field: '_sameAs', reason: 'key-field-not-array' }]); + }); + + it('rejects an inline key field: it carries fields, not an IRI', () => { + const marker: SearchType = { + name: 'Marker', + fields: [nameField], + }; + const inlineKeyed: SearchType = { + ...keyedPlace(), + fields: [ + nameField, + { ...sameAsField, ref: { strategy: 'inline', typeName: 'Marker' } }, + ], + }; + expect(validateSearchType(inlineKeyed)).toEqual([ + { field: '_sameAs', reason: 'key-field-inline' }, + ]); + expect(() => searchSchema(inlineKeyed, marker)).toThrow( + /“_sameAs” \(key-field-inline\)/, + ); + }); + + it('rejects a pick that is not a function', () => { + // A declaration built outside TypeScript (a generator, plain JS) is what + // this guards; a typed one cannot express it. + const key = { field: '_sameAs', pick: 'first' } as unknown as KeyField; + expect(validateSearchType(keyedPlace(key))).toEqual([ + { field: '_sameAs', reason: 'key-pick-not-a-function' }, + ]); + }); + + it('rejects a key on a Reference Type, which is never keyed at all', () => { + // Nested inside its referrer rather than keyed in a collection of its own. + const nested = { + name: 'Marker', + key: { field: '_sameAs' }, + fields: [sameAsField], + } as unknown as SearchType; + expect(validateSearchType(nested)).toEqual([ + { field: '_sameAs', reason: 'key-not-allowed' }, + ]); + }); + }); + + describe('documentKeyOf', () => { + it('keys on the node’s own IRI for a type that declares no key', () => { + const unkeyed: RootType = { + name: 'Place', + class: `${SCHEMA_ORG}Place`, + fields: [sameAsField], + }; + expect( + documentKeyOf(unkeyed, 'https://ex/place/1', [`${GEONAMES}1`]), + ).toBe('https://ex/place/1'); + }); + + it('defaults to the first candidate, sorted and deduplicated', () => { + // Sorting is what makes the default deterministic whatever order the + // CONSTRUCT returned the values in. + const place = keyedPlace(); + expect( + documentKeyOf(place, 'https://ex/place/1', [ + `${GEONAMES}2`, + `${GEONAMES}1`, + `${GEONAMES}1`, + ]), + ).toBe(`${GEONAMES}1`); + }); + + it('applies the key field’s transform, then filters non-IRIs', () => { + const place = { + ...keyedPlace(), + fields: [ + nameField, + { + ...sameAsField, + transform: (iri: string) => iri.replace(/\/$/, ''), + }, + ], + } as const satisfies SearchType; + expect( + documentKeyOf(place, 'https://ex/place/1', [ + 'boerenbont', + `${GEONAMES}1/`, + ]), + ).toBe(`${GEONAMES}1`); + }); + + it('keeps the node’s IRI when nothing survives, without consulting pick', () => { + let consulted = false; + const place = keyedPlace({ + field: '_sameAs', + pick: (candidates) => { + consulted = true; + return candidates[0]; + }, + }); + + expect(documentKeyOf(place, 'https://ex/place/1', ['_:b0'])).toBe( + 'https://ex/place/1', + ); + expect(consulted).toBe(false); + }); + + it('keeps the node’s IRI when pick declines', () => { + const place = keyedPlace({ field: '_sameAs', pick: () => undefined }); + expect(documentKeyOf(place, 'https://ex/place/1', [`${GEONAMES}1`])).toBe( + 'https://ex/place/1', + ); + }); + + it('throws when pick returns something no candidate offered', () => { + const place = keyedPlace({ + field: '_sameAs', + pick: () => 'https://elsewhere/1', + }); + expect(() => + documentKeyOf(place, 'https://ex/place/1', [`${GEONAMES}1`]), + ).toThrow(/is not among its candidates/); + }); + }); +}); + +describe('rootTypeNamed', () => { + const place: SearchType = { + name: 'Place', + class: 'https://schema.org/Place', + fields: [], + }; + const marker: SearchType = { name: 'Marker', fields: [] }; + + it('resolves a Root Type by the name a declaration points at', () => { + expect(rootTypeNamed(searchSchema(place, marker), 'Place')).toBe(place); + }); + + it('resolves neither a Reference Type nor an undeclared name', () => { + // The two name indexes are disjoint: `searchSchema` rejects a name twice. + const schema = searchSchema(place, marker); + expect(rootTypeNamed(schema, 'Marker')).toBeUndefined(); + expect(rootTypeNamed(schema, 'Absent')).toBeUndefined(); + }); +}); diff --git a/packages/search/vite.config.ts b/packages/search/vite.config.ts index e64239d4..c070a10d 100644 --- a/packages/search/vite.config.ts +++ b/packages/search/vite.config.ts @@ -12,7 +12,7 @@ export default mergeConfig( thresholds: { functions: 100, lines: 100, - branches: 99.58, + branches: 99.61, statements: 100, }, }, From 9116e5169818d1fadc75ab38b08409d9ea3ebcd7 Mon Sep 17 00:00:00 2001 From: David de Boer Date: Fri, 21 Aug 2026 15:49:12 +0200 Subject: [PATCH 2/4] refactor(search-typesense): resolve a lookup target through rootTypeNamed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The projection lookup kept a by-name map of its own, rebuilt per call. Read the schema’s own index instead, so the projection, the label-source validation and this lookup resolve a target the same way. - Re-anchor the function-coverage threshold: dropping one covered arrow lowers the ratio without uncovering anything. --- packages/search-typesense/src/lookup.ts | 6 ++---- packages/search-typesense/vite.config.ts | 10 +++++++--- 2 files changed, 9 insertions(+), 7 deletions(-) diff --git a/packages/search-typesense/src/lookup.ts b/packages/search-typesense/src/lookup.ts index d09be180..1a9ffffb 100644 --- a/packages/search-typesense/src/lookup.ts +++ b/packages/search-typesense/src/lookup.ts @@ -9,6 +9,7 @@ import { displayFieldName, fieldNamed, labelFieldOf, + rootTypeNamed, } from '@lde/search/adapter'; import { escapeFilterValue } from './query-compiler.js'; @@ -55,9 +56,6 @@ export async function resolveProjection( if (projection === undefined || parents.length === 0) { return resolved; } - const targetsByName = new Map( - [...schema.values()].map((rootType) => [rootType.name, rootType]), - ); // The level's fields resolve CONCURRENTLY: they read from different // collections and nothing links them, so running them in turn would make the // stated bound – one round-trip per level – one per field instead. @@ -73,7 +71,7 @@ export async function resolveProjection( // hand-built query from throwing here rather than at the port’s guard. return undefined; } - const target = targetsByName.get(field.ref.target); + const target = rootTypeNamed(schema, field.ref.target); const collection = target === undefined ? undefined : collections.get(target.class); if (target === undefined || collection === undefined) { diff --git a/packages/search-typesense/vite.config.ts b/packages/search-typesense/vite.config.ts index 73d30aa6..c039f775 100644 --- a/packages/search-typesense/vite.config.ts +++ b/packages/search-typesense/vite.config.ts @@ -17,14 +17,18 @@ export default mergeConfig( // exercised, which is why branch coverage is lower. thresholds: { // Honest full-suite baseline (autoUpdate raises it from here); a - // partial vitest run must never rewrite these – see AGENTS.md. - functions: 98.93, + // partial vitest run must never rewrite these – see AGENTS.md. The + // function figure moved when the projection lookup dropped its own + // by-name map for the shared `rootTypeNamed`: one COVERED arrow fewer + // over one function fewer, which lowers the ratio without uncovering + // anything. + functions: 98.92, lines: 99.31, // Re-anchored for the projection lookup: its guards against a // projection naming what no lookup reaches are unreachable through // the port, since `assertValidQuery` rejects such a query first. // They hold for a direct caller, and are exercised as one. - branches: 94.58, + branches: 94.63, statements: 99.33, }, }, From c0cc672c561a4a0f04e97907932e6ac4cef6f374 Mon Sep 17 00:00:00 2001 From: David de Boer Date: Fri, 21 Aug 2026 15:49:25 +0200 Subject: [PATCH 3/4] =?UTF-8?q?feat(search-pipeline):=20extract=20a=20refe?= =?UTF-8?q?rent=E2=80=99s=20key=20for=20a=20keyed=20target?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A reference naming a target that declares a `key` needs the referent’s key field in the frame, or the projection has nothing to re-key it from. - Extend such a reference’s UNION branch with an `OPTIONAL` hop reading the key field, emitted under the target’s own IR Alias. Inside the branch, so it multiplies against nothing; `OPTIONAL`, so a referent with no candidate keeps its row and its own IRI. - The root side is unchanged: a key field is a declared field, so its branch and template triple are already there. - Extend the extraction ⟷ projection alias drift guard to the key hop, and add an end-to-end run over a fixture of aligned, unaligned, multiply aligned and declined places with a work referencing each. --- AGENTS.md | 2 +- docs/reference/search-pipeline.md | 8 + packages/search-pipeline/src/extraction.ts | 105 +++++++++- .../search-pipeline/test/extraction.test.ts | 184 +++++++++++++++++- .../test/fixtures/keyed-places-sample.ttl | 48 +++++ .../test/keyed-roots.integration.test.ts | 173 ++++++++++++++++ 6 files changed, 507 insertions(+), 13 deletions(-) create mode 100644 packages/search-pipeline/test/fixtures/keyed-places-sample.ttl create mode 100644 packages/search-pipeline/test/keyed-roots.integration.test.ts diff --git a/AGENTS.md b/AGENTS.md index 6cd788c5..97c1343b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -85,7 +85,7 @@ Each package uses conditional exports with a `development` condition for local d - Test files use `.test.ts` suffix in `test/` directory - Fixtures in `test/fixtures/` - HTTP mocking with Nock -- Tests that start a local SPARQL endpoint (`@lde/local-sparql-endpoint`) must use unique ports across packages to avoid conflicts when Nx runs tests in parallel. Current port allocations: `dataset-registry-client` (3002), `pipeline` sparqlQuery (3001), `pipeline` executor (3003), `pipeline` provenance store (3004), `pipeline-void` namespace-normalization (3005–3006), `search-pipeline` extraction round-trip (3007), `search-pipeline` searchIndexerPipeline end-to-end (3008), `search-pipeline` registry-sourced extraction (3009), `search-pipeline` joins end-to-end (3010) +- Tests that start a local SPARQL endpoint (`@lde/local-sparql-endpoint`) must use unique ports across packages to avoid conflicts when Nx runs tests in parallel. Current port allocations: `dataset-registry-client` (3002), `pipeline` sparqlQuery (3001), `pipeline` executor (3003), `pipeline` provenance store (3004), `pipeline-void` namespace-normalization (3005–3006), `search-pipeline` extraction round-trip (3007), `search-pipeline` searchIndexerPipeline end-to-end (3008), `search-pipeline` registry-sourced extraction (3009), `search-pipeline` joins end-to-end (3010), `search-pipeline` keyed roots (3011) ### Key Dependencies diff --git a/docs/reference/search-pipeline.md b/docs/reference/search-pipeline.md index 18f87228..0401a83a 100644 --- a/docs/reference/search-pipeline.md +++ b/docs/reference/search-pipeline.md @@ -412,6 +412,14 @@ guarantees one output triple per genuine value: fields are UNION’d off the referent variable, so even a multi-hop nested template never conjoins independent multi-valued fields – only the intermediate link triple repeats, and only linearly. +- **A key hop stays inside its branch.** A reference naming a target that + declares a [document key](./search#document-key) gets its branch extended with + an `OPTIONAL` hop reading the referent’s key field, emitted under the target’s + own alias – so the projection can store the referent’s key rather than its node + IRI. It sits inside that reference’s UNION branch, so it multiplies against + nothing, and it is `OPTIONAL` so a referent with no key candidate keeps its + row. The root side is unchanged: a key field is a declared field, so its own + branch and template triple are already there. Because the queries are duplicate-free by construction, correctness and bounded output volume do **not** depend on a client-side **post-processing deduplication diff --git a/packages/search-pipeline/src/extraction.ts b/packages/search-pipeline/src/extraction.ts index 9f97b9ec..51e1e613 100644 --- a/packages/search-pipeline/src/extraction.ts +++ b/packages/search-pipeline/src/extraction.ts @@ -11,11 +11,20 @@ import { import { Parser } from '@traqula/parser-sparql-1-1'; import { Generator } from '@traqula/generator-sparql-1-1'; import { + fieldNamed, irAlias, isInlineReference, + labelSourceNameOf, referenceTypeNamed, + rootTypeNamed, } from '@lde/search/adapter'; -import type { SearchSchema, SearchType } from '@lde/search'; +import type { + ReferenceField, + RootType, + SearchField, + SearchSchema, + SearchType, +} from '@lde/search'; const factory = new AstFactory(); const parser = new Parser(); @@ -55,6 +64,13 @@ export interface ExtractionOptions { * nested template (`{ ?root <…/ref> ?r . ?r <…/field> ?v }`), recursing into * the reference type to the schema’s declared depth. The referent-binding hop * uses the source `path`; the emitted triples use the minted aliases. + * - **references into a keyed type**: a reference naming a target that declares + * a `key` gets its branch extended with an `OPTIONAL` hop reading the + * referent’s key field, emitted under the **target’s** alias for it – so the + * projection can store the referent’s document key rather than its node IRI. + * `OPTIONAL`, so a referent with no key candidate keeps its row. The root side + * needs nothing: a key field is a declared field, so its own branch and + * template triple are already there. * * Wire the result into a `SparqlConstructReader` (see `searchStages`), which * runs it per batch with the roots injected as VALUES. @@ -155,22 +171,91 @@ function buildFor( } else { const value = factory.termVariable(`v${counter.next++}`, factory.gen()); template.push(factory.triple(subject, alias, value)); - branches.push( - factory.patternGroup( - [ - factory.patternBgp( - [factory.triple(subject, sourcePath, value)], - factory.gen(), - ), - ], + const patterns: Pattern[] = [ + factory.patternBgp( + [factory.triple(subject, sourcePath, value)], factory.gen(), ), - ); + ]; + const keyed = keyedTargetOf(field, schema); + if (keyed !== undefined) { + const built = buildKeyHop(keyed.target, keyed.keyField, value, counter); + template.push(built.triple); + patterns.push(built.pattern); + } + branches.push(factory.patternGroup(patterns, factory.gen())); } } return { template, branches }; } +/** + * The keyed Root Type a reference points at, with the field its key is read + * from: a `lookup`’s `target` or an `idOnly`’s `labelSource` + * ({@link labelSourceNameOf}) that declares a {@link RootType.key}, or + * `undefined` for every other field. Naming the target is exactly the boundary + * the projection re-keys along, so the extraction reads the same declarations + * rather than a rule of its own: a reference that names no target keeps the + * node IRI, and needs no hop. + */ +function keyedTargetOf( + field: SearchField, + schema: SearchSchema, +): { readonly target: RootType; readonly keyField: KeyedField } | undefined { + if (field.kind !== 'reference') { + return undefined; + } + const targetName = labelSourceNameOf(field as ReferenceField); + const target = + targetName === undefined ? undefined : rootTypeNamed(schema, targetName); + if (target?.key === undefined) { + return undefined; + } + // `searchSchema` guarantees a declared, path-bearing key field, so the target + // – which came out of the schema – always has one. + const keyField = fieldNamed(target, target.key.field) as KeyedField; + return { target, keyField }; +} + +/** A key field as the schema guarantees it: path-bearing. */ +type KeyedField = SearchField & { readonly path: string }; + +/** + * The one-field hop that reads a referent’s key: a template triple emitting the + * key field under the **target’s** IR Alias, and an `OPTIONAL` branch binding + * it. The referent recursion an inline reference already performs, for a single + * field and wrapped in `OPTIONAL` rather than conjoined – so a referent with no + * key candidate keeps its row and still stores its own IRI, instead of the + * whole reference dropping out of the CONSTRUCT. + * + * It sits inside its own UNION branch (the one the reference contributes), so it + * multiplies against nothing. + */ +function buildKeyHop( + target: RootType, + keyField: KeyedField, + referent: TermVariable, + counter: VariableCounter, +): { readonly triple: TripleNesting; readonly pattern: Pattern } { + const key = factory.termVariable(`k${counter.next++}`, factory.gen()); + return { + triple: factory.triple( + referent, + factory.termNamed(factory.gen(), irAlias(target, keyField)), + key, + ), + pattern: factory.patternOptional( + [ + factory.patternBgp( + [factory.triple(referent, liftPath(keyField.path), key)], + factory.gen(), + ), + ], + factory.gen(), + ), + }; +} + /** * Lift a field’s `path` – written in the SPARQL reader adapter’s grammar (a * property path) – into a predicate AST node, by parsing it inside a throwaway diff --git a/packages/search-pipeline/test/extraction.test.ts b/packages/search-pipeline/test/extraction.test.ts index 727138d7..cc26b37b 100644 --- a/packages/search-pipeline/test/extraction.test.ts +++ b/packages/search-pipeline/test/extraction.test.ts @@ -8,7 +8,13 @@ import { type TripleNesting, } from '@traqula/rules-sparql-1-1'; import { defineSearchType, searchSchema } from '@lde/search'; -import { irAlias, referenceTypeNamed } from '@lde/search/adapter'; +import { + fieldNamed, + irAlias, + labelSourceNameOf, + referenceTypeNamed, + rootTypeNamed, +} from '@lde/search/adapter'; import type { SearchSchema, SearchType } from '@lde/search'; import { extractionQuery, extractionQueryString } from '../src/extraction.js'; @@ -296,6 +302,121 @@ describe('inline references (nested template)', () => { }); }); +describe('references into a keyed type', () => { + // A SCHEMA-AP-NDE-shaped place: keyed on its alignment target, so a work + // referencing it must store that key rather than the publisher’s node IRI. + const place = defineSearchType({ + name: 'Place', + class: `${SCHEMA}Place`, + labelField: 'name', + key: { field: '_sameAs' }, + fields: [ + { + name: 'name', + kind: 'text', + path: `<${SCHEMA}name>`, + locales: ['nl', 'und'], + output: true, + searchable: { weight: 3 }, + }, + { + name: '_sameAs', + kind: 'reference', + array: true, + path: `<${SCHEMA}sameAs>`, + }, + ], + }); + const work = defineSearchType({ + name: 'CreativeWork', + class: `${SCHEMA}CreativeWork`, + fields: [ + { + name: 'locationCreated', + kind: 'reference', + path: `<${SCHEMA}locationCreated>`, + facetable: true, + output: true, + ref: { strategy: 'lookup', target: 'Place' }, + }, + ], + }); + const keyedSchema = searchSchema(work, place); + + it('extends the reference branch with an OPTIONAL hop under the target’s alias', () => { + const query = extractionQuery(work, keyedSchema); + + // ?root <…/locationCreated> ?v ; ?v ?k – the + // referent’s key is minted against PLACE, the type that declares the field. + expect(templatePredicates(query)).toEqual([ + irAlias(work, work.fields[0]), + irAlias(place, place.fields[1]), + ]); + const [referenceTriple, keyTriple] = templateTriples(query); + expect(keyTriple.subject).toEqual(referenceTriple.object); + + // One branch, with the hop OPTIONAL inside it: an unaligned referent keeps + // its row (and so its own IRI) instead of dropping out of the CONSTRUCT. + const [group] = unionBranches(query); + expect(group.patterns).toHaveLength(2); + expect(group.patterns[0]).toMatchObject({ subType: 'bgp' }); + expect(group.patterns[1]).toMatchObject({ subType: 'optional' }); + expect(extractionQueryString(work, keyedSchema)).toContain('OPTIONAL'); + }); + + it('leaves the keyed type’s own extraction unchanged', () => { + // The key field is an ordinary declared field, so its branch and template + // triple are already there; keying adds nothing on the root side. + expect(templatePredicates(extractionQuery(place, keyedSchema))).toEqual([ + irAlias(place, place.fields[0]), + irAlias(place, place.fields[1]), + ]); + expect(unionBranches(extractionQuery(place, keyedSchema))).toHaveLength(2); + }); + + it('adds no hop for a reference that names no target', () => { + // An idOnly reference with no label source never claimed to hold a + // collection’s ids, so nothing re-keys it and nothing is extracted for it. + const unnamed = defineSearchType({ + name: 'Other', + class: 'urn:x:Other', + fields: [ + { + name: 'sameAs', + kind: 'reference', + array: true, + path: `<${SCHEMA}sameAs>`, + facetable: true, + }, + ], + }); + const query = extractionQuery(unnamed, searchSchema(unnamed, place)); + expect(templatePredicates(query)).toEqual([ + irAlias(unnamed, unnamed.fields[0]), + ]); + const [group] = unionBranches(query); + expect(group.patterns).toHaveLength(1); + }); + + it('adds no hop for a target the given schema does not declare', () => { + // Generated against a foreign schema that omits `Place` – the same graceful + // degradation an unresolvable inline reference makes. + const foreignSchema = searchSchema({ + name: 'Other', + class: 'urn:x:Other', + fields: [], + }); + const [group] = unionBranches(extractionQuery(work, foreignSchema)); + expect(group.patterns).toHaveLength(1); + }); + + it('adds no hop for a reference into a target that declares no key', () => { + const query = extractionQuery(creativeWork, drapoSchema); + const [, , creatorBranch] = unionBranches(query); + expect(creatorBranch.patterns).toHaveLength(1); + }); +}); + describe('extraction ⟷ projection contract', () => { // The drift guard: every IR Alias the generator mints is one the projection // reads, and vice versa. Both derive from the same rule – a path-bearing field, @@ -311,13 +432,31 @@ describe('extraction ⟷ projection contract', () => { continue; } aliases.add(irAlias(searchType, field)); - if (field.kind === 'reference' && field.ref?.strategy === 'inline') { + if (field.kind !== 'reference') { + continue; + } + if (field.ref?.strategy === 'inline') { const referenceType = referenceTypeNamed(schema, field.ref.typeName); if (referenceType !== undefined) { for (const alias of projectionReads(referenceType, schema)) { aliases.add(alias); } } + continue; + } + // A reference into a keyed type: the projection reads the referent’s key + // candidates off the frame, under the TARGET’s alias for its key field. + const targetName = labelSourceNameOf(field); + const target = + targetName === undefined + ? undefined + : rootTypeNamed(schema, targetName); + const keyField = + target?.key === undefined + ? undefined + : fieldNamed(target, target.key.field); + if (target !== undefined && keyField !== undefined) { + aliases.add(irAlias(target, keyField)); } } return aliases; @@ -356,4 +495,45 @@ describe('extraction ⟷ projection contract', () => { ); expect(minted).toEqual(projectionReads(dataset, schema)); }); + + it('mints exactly the aliases the projection reads, into a keyed target', () => { + const place = defineSearchType({ + name: 'Place', + class: `${SCHEMA}Place`, + labelField: 'name', + key: { field: '_sameAs' }, + fields: [ + { + name: 'name', + kind: 'text', + path: `<${SCHEMA}name>`, + locales: ['und'], + output: true, + searchable: { weight: 1 }, + }, + { + name: '_sameAs', + kind: 'reference', + array: true, + path: `<${SCHEMA}sameAs>`, + }, + ], + }); + const work = defineSearchType({ + name: 'CreativeWork', + class: `${SCHEMA}CreativeWork`, + fields: [ + { + name: 'locationCreated', + kind: 'reference', + path: `<${SCHEMA}locationCreated>`, + facetable: true, + ref: { strategy: 'lookup', target: 'Place' }, + }, + ], + }); + const schema = searchSchema(work, place); + const minted = new Set(templatePredicates(extractionQuery(work, schema))); + expect(minted).toEqual(projectionReads(work, schema)); + }); }); diff --git a/packages/search-pipeline/test/fixtures/keyed-places-sample.ttl b/packages/search-pipeline/test/fixtures/keyed-places-sample.ttl new file mode 100644 index 00000000..ec15e8ce --- /dev/null +++ b/packages/search-pipeline/test/fixtures/keyed-places-sample.ttl @@ -0,0 +1,48 @@ +@prefix schema: . + +# Two publishers’ places, in the SCHEMA-AP-NDE § 3.4 shape: a publisher’s own +# node, optionally aligned to a term in a source an authority can resolve. + +# Unaligned – nothing to key on, so it keeps the publisher’s own node. + a schema:Place ; + schema:name "Kessel" . + +# Aligned, spelled with a trailing slash. + a schema:Place ; + schema:name "Venlo" ; + schema:sameAs . + +# The same place from another publisher, spelled with http: – one key after the +# key field’s transform, so the two nodes become one document. + a schema:Place ; + schema:name "Venlo (b)" ; + schema:sameAs . + +# Two alignments. Sorted, the GND one comes first; `pick` prefers GeoNames, so +# the key is the SECOND candidate – not whichever the CONSTRUCT happened to +# return first. + a schema:Place ; + schema:name "Roermond" ; + schema:sameAs , . + +# One alignment, into a source `pick` declines: the publisher keeps its node. + a schema:Place ; + schema:name "Buurtschap" ; + schema:sameAs . + +# A work referencing each of them. + a schema:CreativeWork ; + schema:name "Werk 1"@nl ; + schema:locationCreated . + + a schema:CreativeWork ; + schema:name "Werk 2"@nl ; + schema:locationCreated . + + a schema:CreativeWork ; + schema:name "Werk 3"@nl ; + schema:locationCreated . + + a schema:CreativeWork ; + schema:name "Werk 4"@nl ; + schema:locationCreated . diff --git a/packages/search-pipeline/test/keyed-roots.integration.test.ts b/packages/search-pipeline/test/keyed-roots.integration.test.ts new file mode 100644 index 00000000..2b102027 --- /dev/null +++ b/packages/search-pipeline/test/keyed-roots.integration.test.ts @@ -0,0 +1,173 @@ +import { fileURLToPath } from 'node:url'; +import { afterAll, beforeAll, describe, expect, it } from 'vitest'; +import { Dataset, Distribution } from '@lde/dataset'; +import type { DatasetWriter } from '@lde/pipeline'; +import { defineSearchType, searchSchema, type RootType } from '@lde/search'; +import { + startSparqlEndpoint, + teardownSparqlEndpoint, +} from '@lde/local-sparql-endpoint'; +import { searchStages, selectByClass } from '../src/search-stages.js'; +import type { TypedSearchDocument } from '../src/typed-search-document.js'; + +const SCHEMA = 'https://schema.org/'; +const GEONAMES = 'https://sws.geonames.org/'; +const GND = 'https://d-nb.info/gnd/'; + +/** The deployment’s one predicate: an IRI an authority can resolve. */ +const isCovered = (iri: string) => + iri.startsWith(GEONAMES) || iri.startsWith(GND); + +/** The deployment’s IRI normalisation, on the key field – so two publishers + * spelling one alignment differently produce one candidate. */ +const normaliseIri = (iri: string) => + iri.replace(/^http:/, 'https:').replace(/\/$/, ''); + +const place = defineSearchType({ + name: 'Place', + class: `${SCHEMA}Place`, + labelField: 'name', + // A preference order, not a filter: GeoNames first (for its coordinates), + // then any other resolvable source; nothing matched keeps the publisher’s node. + key: { + field: '_sameAs', + pick: (candidates) => + candidates.find((iri) => iri.startsWith(GEONAMES)) ?? + candidates.find(isCovered), + }, + fields: [ + { + name: 'name', + kind: 'text', + path: `<${SCHEMA}name>`, + locales: ['und'], + output: true, + searchable: { weight: 3 }, + }, + { + // Internal: read for the key, then pruned before the writer. + name: '_sameAs', + kind: 'reference', + array: true, + path: `<${SCHEMA}sameAs>`, + transform: normaliseIri, + }, + ], +}); + +const creativeWork = defineSearchType({ + name: 'CreativeWork', + class: `${SCHEMA}CreativeWork`, + fields: [ + { + name: 'name', + kind: 'text', + path: `<${SCHEMA}name>`, + locales: ['nl'], + output: true, + searchable: { weight: 5 }, + }, + { + name: 'locationCreated', + kind: 'reference', + path: `<${SCHEMA}locationCreated>`, + facetable: true, + output: true, + ref: { strategy: 'lookup', target: 'Place' }, + }, + ], +}); + +const schema = searchSchema(creativeWork, place); + +describe('a root type keyed on a declared field, end to end', () => { + const port = 3011; + const distribution = Distribution.sparql( + new URL(`http://localhost:${port}/sparql`), + ); + const dataset = new Dataset({ + iri: new URL('http://example.org/dataset/keyed'), + distributions: [distribution], + }); + + beforeAll(async () => { + const fixture = fileURLToPath( + new URL('./fixtures/keyed-places-sample.ttl', import.meta.url), + ); + await startSparqlEndpoint(port, fixture); + }, 60_000); + + afterAll(async () => { + await teardownSparqlEndpoint(); + }); + + /** Run the generated Extraction CONSTRUCT for one type against the endpoint, + * over roots selected by class, and collect the projected documents. */ + async function project( + searchType: RootType, + ): Promise[]> { + const [stage] = searchStages({ + schema, + types: [ + { + searchType, + rootVariable: 'root', + itemSelector: selectByClass(searchType), + }, + ], + }); + const documents: Record[] = []; + const writer: DatasetWriter = { + write: async (_dataset, items) => { + for await (const item of items) { + documents.push(item.document); + } + }, + }; + await stage.run(dataset, distribution, writer); + return documents; + } + + it('keys each place on its alignment, falling back to the publisher’s node', async () => { + const documents = await project(place); + + expect(documents.map((document) => document.id).sort()).toEqual( + [ + // Unaligned, and aligned into a source `pick` declines: their own nodes. + 'https://a/place/buurtschap', + 'https://a/place/kessel', + // Two alignments, `pick` preferring the GeoNames one – the second + // candidate once they are sorted. + `${GEONAMES}2748812`, + // Two publishers’ nodes for one place, spelled `http://…/` and + // `https://…`: ONE key, because the key field’s transform ran first. + `${GEONAMES}2745707`, + `${GEONAMES}2745707`, + ].sort(), + ); + + // The key field is internal – it keys the document and never reaches the + // writer. + for (const document of documents) { + expect(document).not.toHaveProperty('_sameAs'); + } + }); + + it('stores each work’s place reference under that place’s key', async () => { + const byName = Object.fromEntries( + (await project(creativeWork)).map((document) => [ + document.name_nl, + document, + ]), + ); + + // Aligned referents: the work points at the place’s document, not at the + // publisher’s node – which is what makes the lookup resolve. + expect(byName['Werk 1'].locationCreated).toBe(`${GEONAMES}2745707`); + expect(byName['Werk 3'].locationCreated).toBe(`${GEONAMES}2748812`); + // Unaligned and declined referents keep their node IRI, and still resolve: + // that is the id their own document is written under. + expect(byName['Werk 2'].locationCreated).toBe('https://a/place/kessel'); + expect(byName['Werk 4'].locationCreated).toBe('https://a/place/buurtschap'); + }); +}); From 020c47aeda91c0260069fe25063f724779e7b816 Mon Sep 17 00:00:00 2001 From: David de Boer Date: Fri, 21 Aug 2026 20:19:40 +0200 Subject: [PATCH 4/4] fix(search): name a Reference Type label source for what it is MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Resolving a label source through the Root Type index turned a name that IS declared, just not as a Root Type, into “unknown label source … declare a SearchType with that name” – which sends an author looking in the wrong place. Say it is a Reference Type, and why only a Root Type can serve. Documentation, from review of the key feature: - A transform SUPPLYING key candidates must reach every referring type: a transform is attached to one type’s reader, and a reference’s key is read in the referring type’s query, so candidates minted on the target alone leave every reference keyed on the node IRI. Repairing candidates the graph already carries is unaffected. - The referring field’s own `transform` runs on what it stores, which for a keyed target is the key rather than the referent’s node IRI – pinned by a test, so the ordering is deliberate. - `documentKeyOf` states which shape its `rawValues` are, so a transform that reads them off quads cannot compute a key from a literal-valued object the projection drops. - The key hop repeats a reference’s template triple once per candidate – linearly, like an inline reference’s link triple – rather than “multiplying against nothing”. - ADR 22 no longer claims `@lde/search-typesense` is untouched. --- ...022-key-a-root-type-on-a-declared-field.md | 12 +++++- docs/reference/search-indexer.md | 15 ++++++- docs/reference/search-pipeline.md | 8 ++-- docs/reference/search.md | 12 +++++- packages/search/src/project.ts | 6 +++ packages/search/src/schema.ts | 14 ++++++- packages/search/test/project.test.ts | 42 +++++++++++++++++++ packages/search/test/schema.test.ts | 30 +++++++++++++ 8 files changed, 128 insertions(+), 11 deletions(-) diff --git a/docs/decisions/0022-key-a-root-type-on-a-declared-field.md b/docs/decisions/0022-key-a-root-type-on-a-declared-field.md index 4c352e6e..4aeb6c39 100644 --- a/docs/decisions/0022-key-a-root-type-on-a-declared-field.md +++ b/docs/decisions/0022-key-a-root-type-on-a-declared-field.md @@ -103,6 +103,13 @@ nothing rewrites them. convention rather than a guarantee: the structural alternative (reading the key off the reader’s raw output before transforms run) touches `@lde/pipeline`’s runner, the one package this design otherwise leaves alone. +- **A transform that supplies key candidates must reach every referring type.** + A transform is attached to one type’s reader, and a reference’s key is read in + the _referring_ type’s extraction query – so candidates minted on the target + alone key the target’s own document while every reference to it still stores + the node IRI. Repairing candidates the graph already carries is unaffected + (a reader transform on the referring type covers its own hop); supplying them + is what has to reach both, or be supplied upstream. - **The key is assigned before any `derive` runs**, so a derive sees the key and never the node IRI. A deployment that wants the node IRI declares a plain `idOnly` reference over the same path. @@ -112,8 +119,9 @@ nothing rewrites them. document. Publishers reference other publishers through `sameAs` rather than directly, and such a reference is already unresolvable today for every purpose but labels. -- `@lde/pipeline`, `@lde/search-indexer`, `@lde/search-typesense` and the API - packages are untouched: the change is a schema member, the projection, and one +- `@lde/pipeline`, `@lde/search-indexer` and the API packages are untouched, and + `@lde/search-typesense` only adopts the shared `rootTypeNamed` in place of a + by-name map of its own: the change is a schema member, the projection, and one hop in the extraction generator. A schema declaring no `key` extracts, projects, indexes and queries exactly as before. diff --git a/docs/reference/search-indexer.md b/docs/reference/search-indexer.md index cdfb0afa..9a720318 100644 --- a/docs/reference/search-indexer.md +++ b/docs/reference/search-indexer.md @@ -250,8 +250,19 @@ The same rule bites once more where a root type declares a quads must re-emit the key field.** The key is read off the projected frame like any other field, so a replaced root that drops it is keyed on its node IRI instead – and every reference to it, which is keyed independently, then points -at a document that was never written. A transform that only adds quads (the -documented use) never meets this. +at a document that was never written. A transform that only adds quads never +meets _this_ rule. + +An adding transform has its own version of the same trap, because **a transform +is attached to one type’s reader**, and a reference’s key is read in the +_referring_ type’s extraction query. Mint key candidates on `Place` – a +reconciliation step adding `schema:sameAs` – and `Place` documents are keyed on +them, while the `CreativeWork` stage’s hop still runs against the untransformed +endpoint, finds nothing, and stores the publisher’s node IRI. Every reference +then points at a document that was never written. Where a transform supplies key +candidates rather than repairing them, attach it to **every type that references +the keyed one** as well, or supply them upstream (in the import, or in a reader +of your own) so both queries see them. ## Compose it yourself diff --git a/docs/reference/search-pipeline.md b/docs/reference/search-pipeline.md index 0401a83a..fe0789fc 100644 --- a/docs/reference/search-pipeline.md +++ b/docs/reference/search-pipeline.md @@ -416,9 +416,11 @@ guarantees one output triple per genuine value: declares a [document key](./search#document-key) gets its branch extended with an `OPTIONAL` hop reading the referent’s key field, emitted under the target’s own alias – so the projection can store the referent’s key rather than its node - IRI. It sits inside that reference’s UNION branch, so it multiplies against - nothing, and it is `OPTIONAL` so a referent with no key candidate keeps its - row. The root side is unchanged: a key field is a declared field, so its own + IRI. It sits inside that reference’s UNION branch, so it never cross-multiplies + against another field; like an inline reference’s link triple, the reference’s + own template triple then repeats once per key candidate – linearly, and only + for a referent that carries several. `OPTIONAL`, so a referent with no key + candidate keeps its row. The root side is unchanged: a key field is a declared field, so its own branch and template triple are already there. Because the queries are duplicate-free by construction, correctness and bounded diff --git a/docs/reference/search.md b/docs/reference/search.md index 7eea7e61..aa6d9951 100644 --- a/docs/reference/search.md +++ b/docs/reference/search.md @@ -516,14 +516,22 @@ What LDE deliberately does not know is _why_ one candidate is preferred over another, and what a merged document should say. **LDE decides the key; the deployment decides the content.** -Two things to keep in mind when declaring one: +Things to keep in mind when declaring one: - A transform that **replaces** a root’s quads must re-emit the key field – the existing rule that a field the document needs must be in the stream, applied to - one more field. A transform that only adds never meets it. + one more field. A transform that only adds never meets that rule, but a + transform that **supplies** key candidates has a mirror of it: a transform is + attached to one type’s reader, and a reference’s key is read in the referring + type’s query, so candidates minted on the target alone leave every reference + keyed on the node IRI. See + [Add a transform](./search-indexer#add-a-transform). - The key is assigned **before any `derive` runs**, so a derive sees the key and never the node IRI. A deployment that wants the node IRI declares a plain `idOnly` reference over the same path. +- The **referring** field’s own `transform` runs on what it stores, which for a + keyed target is the key rather than the referent’s node IRI. Declare the two + together only deliberately. - A work in dataset A referencing a node **in dataset B** gets no candidates (the hop runs against A’s distribution), so it stores the node IRI and does not resolve against B’s keyed document. Publishers reference each other through diff --git a/packages/search/src/project.ts b/packages/search/src/project.ts index 83858ec4..a68cea09 100644 --- a/packages/search/src/project.ts +++ b/packages/search/src/project.ts @@ -379,6 +379,12 @@ function applyField( * field to that hop. A projection run without a `schema` cannot resolve the * target, so it leaves the values as they are – exactly as it leaves an inline * reference unprojected. + * + * The referring field’s own {@link ReferenceField.transform} still runs after + * this, in {@link applyFacet}: a `transform` transforms what the field stores, + * and for a keyed target that is the key. A `transform` written to repair the + * referent’s node IRIs therefore sees a key instead, which is why the two are + * worth declaring together only deliberately. */ function referenceValues( node: FramedNode, diff --git a/packages/search/src/schema.ts b/packages/search/src/schema.ts index 4b1b5dc5..a5c3e243 100644 --- a/packages/search/src/schema.ts +++ b/packages/search/src/schema.ts @@ -938,8 +938,14 @@ function assertResolvableLabelSources( } const source = rootTypeNamed(schema, sourceName); if (source === undefined) { + // A name that IS declared, just not as a Root Type, is the confusing + // case: telling the author to declare a type they already declared + // would send them looking in the wrong place. Only a Root Type has a + // collection to resolve against, so name that instead. throw new Error( - `Reference “${searchType.name}.${field.name}” names unknown label source “${sourceName}”; declare a SearchType with that name.`, + referenceTypeNamed(schema, sourceName) === undefined + ? `Reference “${searchType.name}.${field.name}” names unknown label source “${sourceName}”; declare a SearchType with that name.` + : `Reference “${searchType.name}.${field.name}” names label source “${sourceName}”, which is a Reference Type; a label source must be a Root Type, since a resolved label is read from that type’s own collection.`, ); } if (labelFieldOf(source) === undefined) { @@ -1574,7 +1580,11 @@ export function fieldNamed( * would key those two differently. * * @param nodeIri the node’s own IRI – the key it falls back to - * @param rawValues the key field’s values as the frame carries them, untransformed + * @param rawValues the key field’s values **as a reference field reads them** – + * a node’s `@id` or a bare-string value, before the field’s `transform`. A + * caller that reads them itself (a transform working on quads) must apply the + * same rule and skip a literal-valued object, or it will compute a key for a + * value the projection never saw and key the two differently. */ export function documentKeyOf( searchType: RootType, diff --git a/packages/search/test/project.test.ts b/packages/search/test/project.test.ts index 75a16517..387c199d 100644 --- a/packages/search/test/project.test.ts +++ b/packages/search/test/project.test.ts @@ -1901,6 +1901,48 @@ describe('document keys (a type keyed on a declared field)', () => { expect(document.about).toEqual(['https://ex/place/3']); }); + it('runs the referring field’s transform on the key, not the node IRI', () => { + // A `transform` transforms what the field STORES, and for a keyed target + // that is the key – so a transform written against the referent’s node IRIs + // sees a key instead. Pinned because the two are only worth declaring + // together deliberately. + const seen: string[] = []; + const withTransform = defineSearchType({ + ...work, + name: 'TransformingWork', + class: 'urn:x:TransformingWork', + fields: [ + { + name: 'about', + kind: 'reference', + array: true, + path: `<${SCHEMA_ORG}about>`, + facetable: true, + labelSource: 'Place', + transform: (value) => { + seen.push(value); + return value; + }, + }, + ], + }); + projectDocument( + { + '@id': 'https://ex/work/1', + [alias('TransformingWork', 'about')]: [ + { + '@id': 'https://ex/place/1', + [placeKey('_sameAs')]: [{ '@id': `${GEONAMES}1` }], + }, + ], + }, + withTransform, + searchSchema(place, withTransform), + ); + + expect(seen).toEqual([`${GEONAMES}1`]); + }); + it('leaves a reference into an unkeyed target alone', () => { const unkeyed = defineSearchType({ name: 'Place', diff --git a/packages/search/test/schema.test.ts b/packages/search/test/schema.test.ts index 50407fa8..851c8879 100644 --- a/packages/search/test/schema.test.ts +++ b/packages/search/test/schema.test.ts @@ -1334,6 +1334,36 @@ describe('searchSchema validation', () => { ).toThrow(/must declare an output, searchable text field “name”/); }); + it('names a Reference Type label source for what it is, not as unknown', () => { + // The name IS declared, just not as a Root Type – so “declare a + // SearchType with that name” would send an author looking in the wrong + // place. Reachable only through a lookup: `labelSource` on a nested field + // is rejected earlier, and a Reference Type carrying a label field is + // rejected for being searchable. + expect(() => + searchSchema( + { + name: 'AddressRef', + fields: [{ name: 'street', kind: 'keyword', output: true }], + }, + { + name: 'Organization', + class: 'https://example.org/Organization', + fields: [ + { + name: 'address', + kind: 'reference', + output: true, + ref: { strategy: 'lookup', target: 'AddressRef' }, + }, + ], + }, + ), + ).toThrow( + /names label source “AddressRef”, which is a Reference Type; a label source must be a Root Type/, + ); + }); + it('rejects a Reference Type as a label source: it cannot be searchable', () => { // Why a label source is always a Root Type, and so always has a // collection to resolve from: a Reference Type carries `output` only.