diff --git a/AGENTS.md b/AGENTS.md index 6cd788c5..97c1343b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -85,7 +85,7 @@ Each package uses conditional exports with a `development` condition for local d - Test files use `.test.ts` suffix in `test/` directory - Fixtures in `test/fixtures/` - HTTP mocking with Nock -- Tests that start a local SPARQL endpoint (`@lde/local-sparql-endpoint`) must use unique ports across packages to avoid conflicts when Nx runs tests in parallel. Current port allocations: `dataset-registry-client` (3002), `pipeline` sparqlQuery (3001), `pipeline` executor (3003), `pipeline` provenance store (3004), `pipeline-void` namespace-normalization (3005–3006), `search-pipeline` extraction round-trip (3007), `search-pipeline` searchIndexerPipeline end-to-end (3008), `search-pipeline` registry-sourced extraction (3009), `search-pipeline` joins end-to-end (3010) +- Tests that start a local SPARQL endpoint (`@lde/local-sparql-endpoint`) must use unique ports across packages to avoid conflicts when Nx runs tests in parallel. Current port allocations: `dataset-registry-client` (3002), `pipeline` sparqlQuery (3001), `pipeline` executor (3003), `pipeline` provenance store (3004), `pipeline-void` namespace-normalization (3005–3006), `search-pipeline` extraction round-trip (3007), `search-pipeline` searchIndexerPipeline end-to-end (3008), `search-pipeline` registry-sourced extraction (3009), `search-pipeline` joins end-to-end (3010), `search-pipeline` keyed roots (3011) ### Key Dependencies diff --git a/docs/decisions/0022-key-a-root-type-on-a-declared-field.md b/docs/decisions/0022-key-a-root-type-on-a-declared-field.md new file mode 100644 index 00000000..4aeb6c39 --- /dev/null +++ b/docs/decisions/0022-key-a-root-type-on-a-declared-field.md @@ -0,0 +1,160 @@ +# 22. Key a root type on a declared field + +Date: 2026-08-21 + +## Status + +Accepted + +Extends [ADR 20](./0020-resolve-a-references-fields-from-the-targets-own-collection.md), +whose contract – a reference holds ids of documents in the target’s collection – +is what makes reference rewriting a consequence rather than a new rule. Relates +to [ADR 11](./0011-decouple-rdf-depth-from-the-api-surface.md) (inline +references, the other way a referent reaches a document) and +[ADR 12](./0012-bound-memory-by-the-unit-of-work-not-the-input.md). + +## Context + +A search document is keyed on the IRI of the node it was projected from. That is +the right default and wrong for a whole class of profiles: where a publisher +models an entity as its own node and states, in the graph, that the node _is_ +some agreed term – SCHEMA-AP-NDE § 3.4 says exactly this for every +`DefinedTerm` – two publishers describing one place produce two documents, two +facet buckets and two entity pages, and neither can be reached from the other. + +The fact that decides the key is in the graph, one hop from every node that +needs it, and already readable as an ordinary declared field. What was missing +was a way for the schema to say **which field holds the key**. + +The alternative on the table was to move root selection into the deployment – a +`DISTINCT`/`BIND` selector minting roots that carry no triples, and a reader +transform told the batch’s bindings so it could mint content for them. That +works, but it puts a generic seam in `@lde/pipeline`, makes every keyed root an +empty CONSTRUCT, and writes the deployment’s rule twice: once as a SPARQL +`FILTER` in the selector and once in JavaScript in each transform that has to +rewrite a reference, with a standing obligation that the two never disagree. + +## Decision + +A Root Type is keyed on the node’s IRI unless it names a **`key` field** to read +the key from. A reference to such a type stores the target’s key. + +```ts +key: { + field: '_sameAs', + pick: (candidates) => candidates.find(isGeoNames) ?? candidates.find(isCovered), +} +``` + +`key.field` names a declared field of the type: a `path`-bearing, `array` +reference field that is not `inline`. Naming a **declared field** rather than a +path or a pseudo-field is the whole point – the extraction branch already exists, +a reader transform that repairs reference values covers the candidates the day +the field is declared, and the field’s own `transform` is where IRI +normalisation lives, so two spellings of one IRI become one candidate before +anything chooses between them. + +`key.pick` is the deployment’s choice among the candidates, defaulting to the +first. LDE never inspects an IRI’s shape – it asks. + +### The guards + +Candidates reach `pick` transformed, IRI-filtered, deduplicated and **sorted**, +so the default is deterministic whatever order the CONSTRUCT returned them in; +`pick` must return one of them or `undefined`, so a key is always either an IRI +the graph offered for that node or the node’s own, never one invented in +between. `pick` must be pure: the same function keys the document and every +reference to it, so an impure one could key those two differently and leave a +reference dangling. + +`documentKeyOf` is that whole rule in one exported function, so a transform that +needs a node’s key before the projection runs reads the same answer the +projection will. + +### The boundary, for keys and for joins alike + +Only a reference that **names** its target – a `lookup`’s `target`, an +`idOnly`’s `labelSource` – is re-keyed. That is the same line a join draws, and +for the same reason: naming the target is what asserts that the field holds ids +of that collection’s documents. An `idOnly` reference with no label source, and +a `derive`d reference over a raw internal path, never claimed as much, so +nothing rewrites them. + +## Consequences + +- **Several nodes with one key are one document.** That is what a document key + means; the writer upserts by `id`. A deployment that wants the merged document + to carry particular content attaches a transform; one that does not gets + last-writer-wins, exactly as a shared entity across datasets behaves today. + The projection still emits one document per distinct root – folding them is + the writer’s upsert, not the projection’s. +- **Shared documents become the norm rather than the edge case**, which + multiplies the exposure of the single-valued provenance stamp: when one + contributor leaves the run while another that still references the entity is + skipped, the membership sweep can delete a document that is still referenced. + The design does not make this worse per document, but it makes it common. +- **The CONSTRUCT grows** one `OPTIONAL` hop per reference into a keyed type, and + the frame carries the referent’s key-field values. `OPTIONAL` rather than + conjoined, so an unaligned referent keeps its row – and its own IRI – instead + of dropping out of the extraction. +- **A transform that replaces a root’s quads must re-emit the key field.** The + existing rule – a field the document needs must be in the stream – applied to + one more field; a transform that only adds never meets it. Left as a + convention rather than a guarantee: the structural alternative (reading the key + off the reader’s raw output before transforms run) touches `@lde/pipeline`’s + runner, the one package this design otherwise leaves alone. +- **A transform that supplies key candidates must reach every referring type.** + A transform is attached to one type’s reader, and a reference’s key is read in + the _referring_ type’s extraction query – so candidates minted on the target + alone key the target’s own document while every reference to it still stores + the node IRI. Repairing candidates the graph already carries is unaffected + (a reader transform on the referring type covers its own hop); supplying them + is what has to reach both, or be supplied upstream. +- **The key is assigned before any `derive` runs**, so a derive sees the key and + never the node IRI. A deployment that wants the node IRI declares a plain + `idOnly` reference over the same path. +- **A cross-dataset node reference does not resolve.** A work in dataset A + pointing at a local node in dataset B gets no candidates – the hop runs against + A’s distribution – so it stores the node IRI and dangles against B’s keyed + document. Publishers reference other publishers through `sameAs` rather than + directly, and such a reference is already unresolvable today for every purpose + but labels. +- `@lde/pipeline`, `@lde/search-indexer` and the API packages are untouched, and + `@lde/search-typesense` only adopts the shared `rootTypeNamed` in place of a + by-name map of its own: the change is a schema member, the projection, and one + hop in the extraction generator. A schema declaring no `key` extracts, + projects, indexes and queries exactly as before. + +## Rejected + +**A deployment-supplied selector plus `bindings` on reader transforms.** Correct, +but it solves _ids come from the selector_ by moving the selector to the +deployment rather than by letting the schema say what a key is. It reaches the +goal at the cost of a generic seam in `@lde/pipeline`, an empty CONSTRUCT per +keyed root, and a rule every consumer must restate in two languages. + +**Framing the member in Linked Data sameness vocabulary** (`identity: { +alignment, canonical }`). Same mechanics, but it made LDE state rules – “several +nodes merge”, “references are rewritten” – that in search-document terms are +just what keys already do. _Identity_, _alignment_, _canonical_ and _authority_ +are the deployment’s words, and stay in the deployment’s schema. + +**`key: { path }` with a pseudo-field.** A pseudo-field is invisible to +everything that works on declared fields: a reader transform that enumerates +reference-field aliases would not see it, so a publisher writing the alignment as +a typed literal would silently yield no candidates; IRI normalisation would have +nowhere to live, so two spellings of one IRI would silently fail to merge; the +extraction would need an extra root branch; and the alias would become exported +surface. Naming a declared field removes all of it. + +**A derivable `id`.** A `derive` over the document key re-keys the document but +cannot reach a reference on another type: the hop to the referent’s key field has +to be _extracted_, and only a declaration the extraction generator can see makes +that happen. + +**Putting `key` in the indexer’s extensions** rather than in the schema. It keeps +the predicate in app code next to the deployment’s other logic, but the +extraction generator and the projection both need it, so it would have to be +threaded through the stage factory and the pipeline – the ripple the schema +placement avoids. A transform reads the declaration off the loaded schema, so +nothing is lost. diff --git a/docs/reference/search-indexer.md b/docs/reference/search-indexer.md index 124eb342..9a720318 100644 --- a/docs/reference/search-indexer.md +++ b/docs/reference/search-indexer.md @@ -245,6 +245,25 @@ declare a `path`.** Projection skips a field with neither a `path` nor a `derive`, so a transform-minted IR Alias is otherwise never read and the field ships empty. +The same rule bites once more where a root type declares a +[document key](./search#document-key): **a transform that replaces a root’s +quads must re-emit the key field.** The key is read off the projected frame like +any other field, so a replaced root that drops it is keyed on its node IRI +instead – and every reference to it, which is keyed independently, then points +at a document that was never written. A transform that only adds quads never +meets _this_ rule. + +An adding transform has its own version of the same trap, because **a transform +is attached to one type’s reader**, and a reference’s key is read in the +_referring_ type’s extraction query. Mint key candidates on `Place` – a +reconciliation step adding `schema:sameAs` – and `Place` documents are keyed on +them, while the `CreativeWork` stage’s hop still runs against the untransformed +endpoint, finds nothing, and stores the publisher’s node IRI. Every reference +then points at a document that was never written. Where a transform supplies key +candidates rather than repairing them, attach it to **every type that references +the keyed one** as well, or supply them upstream (in the import, or in a reader +of your own) so both queries see them. + ## Compose it yourself Reach for this only when the deployment needs something `createSearchIndexer` diff --git a/docs/reference/search-pipeline.md b/docs/reference/search-pipeline.md index 18f87228..fe0789fc 100644 --- a/docs/reference/search-pipeline.md +++ b/docs/reference/search-pipeline.md @@ -412,6 +412,16 @@ guarantees one output triple per genuine value: fields are UNION’d off the referent variable, so even a multi-hop nested template never conjoins independent multi-valued fields – only the intermediate link triple repeats, and only linearly. +- **A key hop stays inside its branch.** A reference naming a target that + declares a [document key](./search#document-key) gets its branch extended with + an `OPTIONAL` hop reading the referent’s key field, emitted under the target’s + own alias – so the projection can store the referent’s key rather than its node + IRI. It sits inside that reference’s UNION branch, so it never cross-multiplies + against another field; like an inline reference’s link triple, the reference’s + own template triple then repeats once per key candidate – linearly, and only + for a referent that carries several. `OPTIONAL`, so a referent with no key + candidate keeps its row. The root side is unchanged: a key field is a declared field, so its own + branch and template triple are already there. Because the queries are duplicate-free by construction, correctness and bounded output volume do **not** depend on a client-side **post-processing deduplication diff --git a/docs/reference/search.md b/docs/reference/search.md index d8cfd5e3..aa6d9951 100644 --- a/docs/reference/search.md +++ b/docs/reference/search.md @@ -438,6 +438,108 @@ resolves labels from; without it, everything keeps serving `label`. A facet bucket’s `label` is unaffected: it is per-facet-field, and a per-type name would make the bucket shape non-uniform. +### Document key + +A Root Type is keyed on the node’s IRI unless it names a **`key` field** to read +the key from. `key` has the shape of `labelField` – _which field is the label_ – +and says _which field holds the key_: + +```ts +const place = defineSearchType({ + name: 'Place', + class: `${SCHEMA}Place`, + labelField: 'name', + key: { + field: '_sameAs', + // A preference order, not a filter: GeoNames first, then any other source + // an authority can resolve; nothing matched keeps the publisher’s node. + pick: (candidates) => + candidates.find(isGeoNames) ?? candidates.find(isCovered), + }, + fields: [ + { + name: 'name', + kind: 'text', + locales: ['nl', 'und'], + path: `<${SCHEMA}name>`, + output: true, + searchable: { weight: 3 }, + }, + { + // Internal (no role): read for the key, pruned before the writer. + name: '_sameAs', + kind: 'reference', + array: true, + path: `<${SCHEMA}sameAs>`, + transform: normaliseIri, + }, + ], +}); +``` + +- **`key.field`** names a declared field of the type: a `path`-bearing, `array` + reference field that is not `inline`. Its values are the key candidates. + Because it is an ordinary field, everything that already applies to fields + applies to the candidates – it is extracted like any field, a reader transform + that repairs reference values covers it, and the field’s own `transform` is + where IRI normalisation lives, so two spellings of one IRI become one + candidate before anything chooses between them. +- **`key.pick`** chooses among the candidates: `(candidates) => key | undefined`, + where `undefined` keeps the node’s own IRI. It defaults to the first candidate, + and is not consulted for a node whose key field is empty. +- **The guards.** Candidates reach `pick` transformed, IRI-filtered, deduplicated + and **sorted**, so the default is deterministic whatever order the CONSTRUCT + returned them in. `pick` must return one of them or `undefined` – anything else + throws, naming the node and its candidates – so a key is always an IRI the + graph offered for that node. And `pick` must be **pure**: the same function + keys the document and every reference to it. + +`documentKeyOf` (from `@lde/search/adapter`) is that whole rule in one function, +for a transform that needs to know a node’s key before the projection runs. + +Two consequences are not new rules, only what a document key already means: + +- **Several nodes with one key are one document.** The writer upserts by `id`. A + deployment that wants the merged document to carry particular content attaches + a transform; one that does not gets last-writer-wins, exactly as a shared + entity across datasets behaves today. +- **A reference stores the target’s key.** A `lookup`’s `target` and an + `idOnly`’s `labelSource` already mean _this field holds ids of documents in + that collection_ – the contract a label lookup and a join rely on – so a + reference that names a keyed target stores the referent’s key rather than its + node IRI. A reference that names no target is never rewritten: it claimed + nothing about a collection. The extraction adds one `OPTIONAL` hop per such + reference to read the referent’s key field, so an unaligned referent keeps its + row and its own IRI. + +What LDE deliberately does not know is _why_ one candidate is preferred over +another, and what a merged document should say. **LDE decides the key; the +deployment decides the content.** + +Things to keep in mind when declaring one: + +- A transform that **replaces** a root’s quads must re-emit the key field – the + existing rule that a field the document needs must be in the stream, applied to + one more field. A transform that only adds never meets that rule, but a + transform that **supplies** key candidates has a mirror of it: a transform is + attached to one type’s reader, and a reference’s key is read in the referring + type’s query, so candidates minted on the target alone leave every reference + keyed on the node IRI. See + [Add a transform](./search-indexer#add-a-transform). +- The key is assigned **before any `derive` runs**, so a derive sees the key and + never the node IRI. A deployment that wants the node IRI declares a plain + `idOnly` reference over the same path. +- The **referring** field’s own `transform` runs on what it stores, which for a + keyed target is the key rather than the referent’s node IRI. Declare the two + together only deliberately. +- A work in dataset A referencing a node **in dataset B** gets no candidates (the + hop runs against A’s distribution), so it stores the node IRI and does not + resolve against B’s keyed document. Publishers reference each other through + `sameAs` rather than directly, and such a reference is already unresolvable + today for every purpose but labels. + +See [ADR 22](../decisions/0022-key-a-root-type-on-a-declared-field). + ### Projecting what a lookup carries A `lookup` declares no field list. What it _fetches_ is named per query, by a @@ -728,7 +830,8 @@ reported to `onIgnoredFilter`, since neither compiled as written. ### Lookup by IRI -Every type is filterable on **`id`** – the document’s IRI – without declaring +Every type is filterable on **`id`** – the document’s key, which is the node’s +IRI unless the type declares a [`key` field](#document-key) – without declaring it, and every surface returns it. It is the one field no `SearchType` declares, because every indexed thing already carries it: it is the hit’s identity (`SearchHit.id`), not a value in its `ResultDocument`. `searchSchema()` rejects diff --git a/packages/search-pipeline/src/extraction.ts b/packages/search-pipeline/src/extraction.ts index 9f97b9ec..51e1e613 100644 --- a/packages/search-pipeline/src/extraction.ts +++ b/packages/search-pipeline/src/extraction.ts @@ -11,11 +11,20 @@ import { import { Parser } from '@traqula/parser-sparql-1-1'; import { Generator } from '@traqula/generator-sparql-1-1'; import { + fieldNamed, irAlias, isInlineReference, + labelSourceNameOf, referenceTypeNamed, + rootTypeNamed, } from '@lde/search/adapter'; -import type { SearchSchema, SearchType } from '@lde/search'; +import type { + ReferenceField, + RootType, + SearchField, + SearchSchema, + SearchType, +} from '@lde/search'; const factory = new AstFactory(); const parser = new Parser(); @@ -55,6 +64,13 @@ export interface ExtractionOptions { * nested template (`{ ?root <…/ref> ?r . ?r <…/field> ?v }`), recursing into * the reference type to the schema’s declared depth. The referent-binding hop * uses the source `path`; the emitted triples use the minted aliases. + * - **references into a keyed type**: a reference naming a target that declares + * a `key` gets its branch extended with an `OPTIONAL` hop reading the + * referent’s key field, emitted under the **target’s** alias for it – so the + * projection can store the referent’s document key rather than its node IRI. + * `OPTIONAL`, so a referent with no key candidate keeps its row. The root side + * needs nothing: a key field is a declared field, so its own branch and + * template triple are already there. * * Wire the result into a `SparqlConstructReader` (see `searchStages`), which * runs it per batch with the roots injected as VALUES. @@ -155,22 +171,91 @@ function buildFor( } else { const value = factory.termVariable(`v${counter.next++}`, factory.gen()); template.push(factory.triple(subject, alias, value)); - branches.push( - factory.patternGroup( - [ - factory.patternBgp( - [factory.triple(subject, sourcePath, value)], - factory.gen(), - ), - ], + const patterns: Pattern[] = [ + factory.patternBgp( + [factory.triple(subject, sourcePath, value)], factory.gen(), ), - ); + ]; + const keyed = keyedTargetOf(field, schema); + if (keyed !== undefined) { + const built = buildKeyHop(keyed.target, keyed.keyField, value, counter); + template.push(built.triple); + patterns.push(built.pattern); + } + branches.push(factory.patternGroup(patterns, factory.gen())); } } return { template, branches }; } +/** + * The keyed Root Type a reference points at, with the field its key is read + * from: a `lookup`’s `target` or an `idOnly`’s `labelSource` + * ({@link labelSourceNameOf}) that declares a {@link RootType.key}, or + * `undefined` for every other field. Naming the target is exactly the boundary + * the projection re-keys along, so the extraction reads the same declarations + * rather than a rule of its own: a reference that names no target keeps the + * node IRI, and needs no hop. + */ +function keyedTargetOf( + field: SearchField, + schema: SearchSchema, +): { readonly target: RootType; readonly keyField: KeyedField } | undefined { + if (field.kind !== 'reference') { + return undefined; + } + const targetName = labelSourceNameOf(field as ReferenceField); + const target = + targetName === undefined ? undefined : rootTypeNamed(schema, targetName); + if (target?.key === undefined) { + return undefined; + } + // `searchSchema` guarantees a declared, path-bearing key field, so the target + // – which came out of the schema – always has one. + const keyField = fieldNamed(target, target.key.field) as KeyedField; + return { target, keyField }; +} + +/** A key field as the schema guarantees it: path-bearing. */ +type KeyedField = SearchField & { readonly path: string }; + +/** + * The one-field hop that reads a referent’s key: a template triple emitting the + * key field under the **target’s** IR Alias, and an `OPTIONAL` branch binding + * it. The referent recursion an inline reference already performs, for a single + * field and wrapped in `OPTIONAL` rather than conjoined – so a referent with no + * key candidate keeps its row and still stores its own IRI, instead of the + * whole reference dropping out of the CONSTRUCT. + * + * It sits inside its own UNION branch (the one the reference contributes), so it + * multiplies against nothing. + */ +function buildKeyHop( + target: RootType, + keyField: KeyedField, + referent: TermVariable, + counter: VariableCounter, +): { readonly triple: TripleNesting; readonly pattern: Pattern } { + const key = factory.termVariable(`k${counter.next++}`, factory.gen()); + return { + triple: factory.triple( + referent, + factory.termNamed(factory.gen(), irAlias(target, keyField)), + key, + ), + pattern: factory.patternOptional( + [ + factory.patternBgp( + [factory.triple(referent, liftPath(keyField.path), key)], + factory.gen(), + ), + ], + factory.gen(), + ), + }; +} + /** * Lift a field’s `path` – written in the SPARQL reader adapter’s grammar (a * property path) – into a predicate AST node, by parsing it inside a throwaway diff --git a/packages/search-pipeline/test/extraction.test.ts b/packages/search-pipeline/test/extraction.test.ts index 727138d7..cc26b37b 100644 --- a/packages/search-pipeline/test/extraction.test.ts +++ b/packages/search-pipeline/test/extraction.test.ts @@ -8,7 +8,13 @@ import { type TripleNesting, } from '@traqula/rules-sparql-1-1'; import { defineSearchType, searchSchema } from '@lde/search'; -import { irAlias, referenceTypeNamed } from '@lde/search/adapter'; +import { + fieldNamed, + irAlias, + labelSourceNameOf, + referenceTypeNamed, + rootTypeNamed, +} from '@lde/search/adapter'; import type { SearchSchema, SearchType } from '@lde/search'; import { extractionQuery, extractionQueryString } from '../src/extraction.js'; @@ -296,6 +302,121 @@ describe('inline references (nested template)', () => { }); }); +describe('references into a keyed type', () => { + // A SCHEMA-AP-NDE-shaped place: keyed on its alignment target, so a work + // referencing it must store that key rather than the publisher’s node IRI. + const place = defineSearchType({ + name: 'Place', + class: `${SCHEMA}Place`, + labelField: 'name', + key: { field: '_sameAs' }, + fields: [ + { + name: 'name', + kind: 'text', + path: `<${SCHEMA}name>`, + locales: ['nl', 'und'], + output: true, + searchable: { weight: 3 }, + }, + { + name: '_sameAs', + kind: 'reference', + array: true, + path: `<${SCHEMA}sameAs>`, + }, + ], + }); + const work = defineSearchType({ + name: 'CreativeWork', + class: `${SCHEMA}CreativeWork`, + fields: [ + { + name: 'locationCreated', + kind: 'reference', + path: `<${SCHEMA}locationCreated>`, + facetable: true, + output: true, + ref: { strategy: 'lookup', target: 'Place' }, + }, + ], + }); + const keyedSchema = searchSchema(work, place); + + it('extends the reference branch with an OPTIONAL hop under the target’s alias', () => { + const query = extractionQuery(work, keyedSchema); + + // ?root <…/locationCreated> ?v ; ?v ?k – the + // referent’s key is minted against PLACE, the type that declares the field. + expect(templatePredicates(query)).toEqual([ + irAlias(work, work.fields[0]), + irAlias(place, place.fields[1]), + ]); + const [referenceTriple, keyTriple] = templateTriples(query); + expect(keyTriple.subject).toEqual(referenceTriple.object); + + // One branch, with the hop OPTIONAL inside it: an unaligned referent keeps + // its row (and so its own IRI) instead of dropping out of the CONSTRUCT. + const [group] = unionBranches(query); + expect(group.patterns).toHaveLength(2); + expect(group.patterns[0]).toMatchObject({ subType: 'bgp' }); + expect(group.patterns[1]).toMatchObject({ subType: 'optional' }); + expect(extractionQueryString(work, keyedSchema)).toContain('OPTIONAL'); + }); + + it('leaves the keyed type’s own extraction unchanged', () => { + // The key field is an ordinary declared field, so its branch and template + // triple are already there; keying adds nothing on the root side. + expect(templatePredicates(extractionQuery(place, keyedSchema))).toEqual([ + irAlias(place, place.fields[0]), + irAlias(place, place.fields[1]), + ]); + expect(unionBranches(extractionQuery(place, keyedSchema))).toHaveLength(2); + }); + + it('adds no hop for a reference that names no target', () => { + // An idOnly reference with no label source never claimed to hold a + // collection’s ids, so nothing re-keys it and nothing is extracted for it. + const unnamed = defineSearchType({ + name: 'Other', + class: 'urn:x:Other', + fields: [ + { + name: 'sameAs', + kind: 'reference', + array: true, + path: `<${SCHEMA}sameAs>`, + facetable: true, + }, + ], + }); + const query = extractionQuery(unnamed, searchSchema(unnamed, place)); + expect(templatePredicates(query)).toEqual([ + irAlias(unnamed, unnamed.fields[0]), + ]); + const [group] = unionBranches(query); + expect(group.patterns).toHaveLength(1); + }); + + it('adds no hop for a target the given schema does not declare', () => { + // Generated against a foreign schema that omits `Place` – the same graceful + // degradation an unresolvable inline reference makes. + const foreignSchema = searchSchema({ + name: 'Other', + class: 'urn:x:Other', + fields: [], + }); + const [group] = unionBranches(extractionQuery(work, foreignSchema)); + expect(group.patterns).toHaveLength(1); + }); + + it('adds no hop for a reference into a target that declares no key', () => { + const query = extractionQuery(creativeWork, drapoSchema); + const [, , creatorBranch] = unionBranches(query); + expect(creatorBranch.patterns).toHaveLength(1); + }); +}); + describe('extraction ⟷ projection contract', () => { // The drift guard: every IR Alias the generator mints is one the projection // reads, and vice versa. Both derive from the same rule – a path-bearing field, @@ -311,13 +432,31 @@ describe('extraction ⟷ projection contract', () => { continue; } aliases.add(irAlias(searchType, field)); - if (field.kind === 'reference' && field.ref?.strategy === 'inline') { + if (field.kind !== 'reference') { + continue; + } + if (field.ref?.strategy === 'inline') { const referenceType = referenceTypeNamed(schema, field.ref.typeName); if (referenceType !== undefined) { for (const alias of projectionReads(referenceType, schema)) { aliases.add(alias); } } + continue; + } + // A reference into a keyed type: the projection reads the referent’s key + // candidates off the frame, under the TARGET’s alias for its key field. + const targetName = labelSourceNameOf(field); + const target = + targetName === undefined + ? undefined + : rootTypeNamed(schema, targetName); + const keyField = + target?.key === undefined + ? undefined + : fieldNamed(target, target.key.field); + if (target !== undefined && keyField !== undefined) { + aliases.add(irAlias(target, keyField)); } } return aliases; @@ -356,4 +495,45 @@ describe('extraction ⟷ projection contract', () => { ); expect(minted).toEqual(projectionReads(dataset, schema)); }); + + it('mints exactly the aliases the projection reads, into a keyed target', () => { + const place = defineSearchType({ + name: 'Place', + class: `${SCHEMA}Place`, + labelField: 'name', + key: { field: '_sameAs' }, + fields: [ + { + name: 'name', + kind: 'text', + path: `<${SCHEMA}name>`, + locales: ['und'], + output: true, + searchable: { weight: 1 }, + }, + { + name: '_sameAs', + kind: 'reference', + array: true, + path: `<${SCHEMA}sameAs>`, + }, + ], + }); + const work = defineSearchType({ + name: 'CreativeWork', + class: `${SCHEMA}CreativeWork`, + fields: [ + { + name: 'locationCreated', + kind: 'reference', + path: `<${SCHEMA}locationCreated>`, + facetable: true, + ref: { strategy: 'lookup', target: 'Place' }, + }, + ], + }); + const schema = searchSchema(work, place); + const minted = new Set(templatePredicates(extractionQuery(work, schema))); + expect(minted).toEqual(projectionReads(work, schema)); + }); }); diff --git a/packages/search-pipeline/test/fixtures/keyed-places-sample.ttl b/packages/search-pipeline/test/fixtures/keyed-places-sample.ttl new file mode 100644 index 00000000..ec15e8ce --- /dev/null +++ b/packages/search-pipeline/test/fixtures/keyed-places-sample.ttl @@ -0,0 +1,48 @@ +@prefix schema: . + +# Two publishers’ places, in the SCHEMA-AP-NDE § 3.4 shape: a publisher’s own +# node, optionally aligned to a term in a source an authority can resolve. + +# Unaligned – nothing to key on, so it keeps the publisher’s own node. + a schema:Place ; + schema:name "Kessel" . + +# Aligned, spelled with a trailing slash. + a schema:Place ; + schema:name "Venlo" ; + schema:sameAs . + +# The same place from another publisher, spelled with http: – one key after the +# key field’s transform, so the two nodes become one document. + a schema:Place ; + schema:name "Venlo (b)" ; + schema:sameAs . + +# Two alignments. Sorted, the GND one comes first; `pick` prefers GeoNames, so +# the key is the SECOND candidate – not whichever the CONSTRUCT happened to +# return first. + a schema:Place ; + schema:name "Roermond" ; + schema:sameAs , . + +# One alignment, into a source `pick` declines: the publisher keeps its node. + a schema:Place ; + schema:name "Buurtschap" ; + schema:sameAs . + +# A work referencing each of them. + a schema:CreativeWork ; + schema:name "Werk 1"@nl ; + schema:locationCreated . + + a schema:CreativeWork ; + schema:name "Werk 2"@nl ; + schema:locationCreated . + + a schema:CreativeWork ; + schema:name "Werk 3"@nl ; + schema:locationCreated . + + a schema:CreativeWork ; + schema:name "Werk 4"@nl ; + schema:locationCreated . diff --git a/packages/search-pipeline/test/keyed-roots.integration.test.ts b/packages/search-pipeline/test/keyed-roots.integration.test.ts new file mode 100644 index 00000000..2b102027 --- /dev/null +++ b/packages/search-pipeline/test/keyed-roots.integration.test.ts @@ -0,0 +1,173 @@ +import { fileURLToPath } from 'node:url'; +import { afterAll, beforeAll, describe, expect, it } from 'vitest'; +import { Dataset, Distribution } from '@lde/dataset'; +import type { DatasetWriter } from '@lde/pipeline'; +import { defineSearchType, searchSchema, type RootType } from '@lde/search'; +import { + startSparqlEndpoint, + teardownSparqlEndpoint, +} from '@lde/local-sparql-endpoint'; +import { searchStages, selectByClass } from '../src/search-stages.js'; +import type { TypedSearchDocument } from '../src/typed-search-document.js'; + +const SCHEMA = 'https://schema.org/'; +const GEONAMES = 'https://sws.geonames.org/'; +const GND = 'https://d-nb.info/gnd/'; + +/** The deployment’s one predicate: an IRI an authority can resolve. */ +const isCovered = (iri: string) => + iri.startsWith(GEONAMES) || iri.startsWith(GND); + +/** The deployment’s IRI normalisation, on the key field – so two publishers + * spelling one alignment differently produce one candidate. */ +const normaliseIri = (iri: string) => + iri.replace(/^http:/, 'https:').replace(/\/$/, ''); + +const place = defineSearchType({ + name: 'Place', + class: `${SCHEMA}Place`, + labelField: 'name', + // A preference order, not a filter: GeoNames first (for its coordinates), + // then any other resolvable source; nothing matched keeps the publisher’s node. + key: { + field: '_sameAs', + pick: (candidates) => + candidates.find((iri) => iri.startsWith(GEONAMES)) ?? + candidates.find(isCovered), + }, + fields: [ + { + name: 'name', + kind: 'text', + path: `<${SCHEMA}name>`, + locales: ['und'], + output: true, + searchable: { weight: 3 }, + }, + { + // Internal: read for the key, then pruned before the writer. + name: '_sameAs', + kind: 'reference', + array: true, + path: `<${SCHEMA}sameAs>`, + transform: normaliseIri, + }, + ], +}); + +const creativeWork = defineSearchType({ + name: 'CreativeWork', + class: `${SCHEMA}CreativeWork`, + fields: [ + { + name: 'name', + kind: 'text', + path: `<${SCHEMA}name>`, + locales: ['nl'], + output: true, + searchable: { weight: 5 }, + }, + { + name: 'locationCreated', + kind: 'reference', + path: `<${SCHEMA}locationCreated>`, + facetable: true, + output: true, + ref: { strategy: 'lookup', target: 'Place' }, + }, + ], +}); + +const schema = searchSchema(creativeWork, place); + +describe('a root type keyed on a declared field, end to end', () => { + const port = 3011; + const distribution = Distribution.sparql( + new URL(`http://localhost:${port}/sparql`), + ); + const dataset = new Dataset({ + iri: new URL('http://example.org/dataset/keyed'), + distributions: [distribution], + }); + + beforeAll(async () => { + const fixture = fileURLToPath( + new URL('./fixtures/keyed-places-sample.ttl', import.meta.url), + ); + await startSparqlEndpoint(port, fixture); + }, 60_000); + + afterAll(async () => { + await teardownSparqlEndpoint(); + }); + + /** Run the generated Extraction CONSTRUCT for one type against the endpoint, + * over roots selected by class, and collect the projected documents. */ + async function project( + searchType: RootType, + ): Promise[]> { + const [stage] = searchStages({ + schema, + types: [ + { + searchType, + rootVariable: 'root', + itemSelector: selectByClass(searchType), + }, + ], + }); + const documents: Record[] = []; + const writer: DatasetWriter = { + write: async (_dataset, items) => { + for await (const item of items) { + documents.push(item.document); + } + }, + }; + await stage.run(dataset, distribution, writer); + return documents; + } + + it('keys each place on its alignment, falling back to the publisher’s node', async () => { + const documents = await project(place); + + expect(documents.map((document) => document.id).sort()).toEqual( + [ + // Unaligned, and aligned into a source `pick` declines: their own nodes. + 'https://a/place/buurtschap', + 'https://a/place/kessel', + // Two alignments, `pick` preferring the GeoNames one – the second + // candidate once they are sorted. + `${GEONAMES}2748812`, + // Two publishers’ nodes for one place, spelled `http://…/` and + // `https://…`: ONE key, because the key field’s transform ran first. + `${GEONAMES}2745707`, + `${GEONAMES}2745707`, + ].sort(), + ); + + // The key field is internal – it keys the document and never reaches the + // writer. + for (const document of documents) { + expect(document).not.toHaveProperty('_sameAs'); + } + }); + + it('stores each work’s place reference under that place’s key', async () => { + const byName = Object.fromEntries( + (await project(creativeWork)).map((document) => [ + document.name_nl, + document, + ]), + ); + + // Aligned referents: the work points at the place’s document, not at the + // publisher’s node – which is what makes the lookup resolve. + expect(byName['Werk 1'].locationCreated).toBe(`${GEONAMES}2745707`); + expect(byName['Werk 3'].locationCreated).toBe(`${GEONAMES}2748812`); + // Unaligned and declined referents keep their node IRI, and still resolve: + // that is the id their own document is written under. + expect(byName['Werk 2'].locationCreated).toBe('https://a/place/kessel'); + expect(byName['Werk 4'].locationCreated).toBe('https://a/place/buurtschap'); + }); +}); diff --git a/packages/search-typesense/src/lookup.ts b/packages/search-typesense/src/lookup.ts index d09be180..1a9ffffb 100644 --- a/packages/search-typesense/src/lookup.ts +++ b/packages/search-typesense/src/lookup.ts @@ -9,6 +9,7 @@ import { displayFieldName, fieldNamed, labelFieldOf, + rootTypeNamed, } from '@lde/search/adapter'; import { escapeFilterValue } from './query-compiler.js'; @@ -55,9 +56,6 @@ export async function resolveProjection( if (projection === undefined || parents.length === 0) { return resolved; } - const targetsByName = new Map( - [...schema.values()].map((rootType) => [rootType.name, rootType]), - ); // The level's fields resolve CONCURRENTLY: they read from different // collections and nothing links them, so running them in turn would make the // stated bound – one round-trip per level – one per field instead. @@ -73,7 +71,7 @@ export async function resolveProjection( // hand-built query from throwing here rather than at the port’s guard. return undefined; } - const target = targetsByName.get(field.ref.target); + const target = rootTypeNamed(schema, field.ref.target); const collection = target === undefined ? undefined : collections.get(target.class); if (target === undefined || collection === undefined) { diff --git a/packages/search-typesense/vite.config.ts b/packages/search-typesense/vite.config.ts index 73d30aa6..c039f775 100644 --- a/packages/search-typesense/vite.config.ts +++ b/packages/search-typesense/vite.config.ts @@ -17,14 +17,18 @@ export default mergeConfig( // exercised, which is why branch coverage is lower. thresholds: { // Honest full-suite baseline (autoUpdate raises it from here); a - // partial vitest run must never rewrite these – see AGENTS.md. - functions: 98.93, + // partial vitest run must never rewrite these – see AGENTS.md. The + // function figure moved when the projection lookup dropped its own + // by-name map for the shared `rootTypeNamed`: one COVERED arrow fewer + // over one function fewer, which lowers the ratio without uncovering + // anything. + functions: 98.92, lines: 99.31, // Re-anchored for the projection lookup: its guards against a // projection naming what no lookup reaches are unreachable through // the port, since `assertValidQuery` rejects such a query first. // They hold for a direct caller, and are exercised as one. - branches: 94.58, + branches: 94.63, statements: 99.33, }, }, diff --git a/packages/search/src/adapter.ts b/packages/search/src/adapter.ts index c04ed658..137cd622 100644 --- a/packages/search/src/adapter.ts +++ b/packages/search/src/adapter.ts @@ -24,6 +24,7 @@ export { nestedFieldName, referenceFields, referenceTypeNamed, + rootTypeNamed, inlineFramingDepth, fieldNamed, datasetField, @@ -33,6 +34,7 @@ export { labelFieldOf, labelFieldNameOf, labelSourceNameOf, + documentKeyOf, DEFAULT_LABEL_FIELD, isRangeFacet, isAbsoluteIri, diff --git a/packages/search/src/index.ts b/packages/search/src/index.ts index 49c61e98..8debc101 100644 --- a/packages/search/src/index.ts +++ b/packages/search/src/index.ts @@ -49,6 +49,7 @@ export type { ReferenceType, RootTypeOf, SearchTypeIssue, + KeyField, SearchSchema, FacetRange, ProjectionValue, diff --git a/packages/search/src/project.ts b/packages/search/src/project.ts index 75a95716..a68cea09 100644 --- a/packages/search/src/project.ts +++ b/packages/search/src/project.ts @@ -8,14 +8,18 @@ import { import { assertTypeInSchema, displayFieldName, + documentKeyOf, + fieldNamed, inlineFramingDepth, irAlias, isAbsoluteIri, isInternalField, isInlineReference, isoToUnixSeconds, + labelSourceNameOf, physicalFields, referenceTypeNamed, + rootTypeNamed, type KeywordField, type ProjectionValue, type ReferenceField, @@ -147,7 +151,7 @@ function projectFields( schema: SearchSchema | undefined, context: ProjectionContext, ): ProjectedNode { - const id = documentKey(node); + const id = documentIdOf(node, searchType); const document: ProjectedNode = id === undefined ? {} : { id }; for (const field of searchType.fields) { applyField(document, node, field, searchType, schema, context); @@ -169,6 +173,46 @@ function documentKey(node: FramedNode): string | undefined { return typeof id === 'string' && isAbsoluteIri(id) ? id : undefined; } +/** + * The `id` a node is projected under: its own IRI, unless its type declares a + * {@link RootType.key} – then the key its key field carries + * ({@link documentKeyOf}). The candidates are read off the frame like any other + * field value, under the key field’s own {@link irAlias IR Alias}, because the + * key field *is* an ordinary declared field: the extraction already emits it, + * and a reader transform that repairs reference values has already run on it. + * + * The key is assigned before any `derive` runs, so a derive sees the key and + * never the node IRI; a deployment that wants the node IRI declares a plain + * `idOnly` reference over the same path. + */ +function documentIdOf( + node: FramedNode, + searchType: SearchType, +): string | undefined { + const nodeIri = documentKey(node); + if (nodeIri === undefined || searchType.key === undefined) { + return nodeIri; + } + return documentKeyOf( + searchType, + nodeIri, + keyCandidatesOf(node, searchType, searchType.key.field), + ); +} + +/** The raw values a node carries under a type’s key field – untransformed, as + * {@link documentKeyOf} expects them. Empty for a node the key field matched + * nothing on. The field itself is guaranteed declared by `searchSchema`, which + * is what validates a `key` at all. */ +function keyCandidatesOf( + node: FramedNode, + searchType: SearchType, + name: string, +): readonly string[] { + const keyField = fieldNamed(searchType, name) as SearchField; + return irisOf(node, irAlias(searchType, keyField)); +} + /** * Project a single type over a known set of `roots` – the per-type, roots-given * projection. The roots are supplied by the caller (the pipeline selector) @@ -193,8 +237,11 @@ export async function* projectRoots( const index = buildSubjectIndex(quads); // Distinct roots only. A selector may return an IRI more than once – a // non-`DISTINCT` `SELECT` over a one-to-many join yields the same subject per - // matched row – and a repeated root would otherwise frame and emit a - // duplicate document under the same `id`. + // matched row – and framing a root twice would do the same work twice for one + // node. Distinct roots may still legitimately share an `id`, where the type + // declares a `key` two of them carry: that is what a document key means, and + // it is the writer’s upsert – not the projection – that folds them into one + // document. const depth = inlineFramingDepth(schema, searchType); for await (const node of frameSubjects(index, [...new Set(roots)], depth)) { yield projectDocument(node, searchType, schema, context); @@ -277,7 +324,11 @@ function applyField( case 'keyword': return applyFacet(document, literalsOf(node, alias), field); case 'reference': - return applyFacet(document, irisOf(node, alias), field); + return applyFacet( + document, + referenceValues(node, alias, field, schema), + field, + ); case 'integer': return setNumber( document, @@ -309,6 +360,63 @@ function applyField( } } +/** + * The values a non-inline reference stores: the referents’ IRIs, **or their + * document keys** when the type the reference names declares a + * {@link RootType.key}. A `lookup`’s `target` and an `idOnly`’s `labelSource` + * both mean *this field holds ids of documents in that collection* (ADR 20) – + * the fact a label lookup and a join rely on – so storing the node IRI where + * the target keys on something else would break an invariant the schema already + * has. Rewriting a reference is LDE keeping it, not a new rule. + * + * That is also why the boundary is *naming the target*: a reference that names + * none ({@link labelSourceNameOf} → `undefined`) never claimed to hold a + * collection’s ids, and a `derive`d one produces its own values rather than + * reading a referent, so neither is re-keyed. + * + * The referent’s candidates are in the frame because framing already embeds a + * referenced node’s own triples at one hop, and the extraction adds the key + * field to that hop. A projection run without a `schema` cannot resolve the + * target, so it leaves the values as they are – exactly as it leaves an inline + * reference unprojected. + * + * The referring field’s own {@link ReferenceField.transform} still runs after + * this, in {@link applyFacet}: a `transform` transforms what the field stores, + * and for a keyed target that is the key. A `transform` written to repair the + * referent’s node IRIs therefore sees a key instead, which is why the two are + * worth declaring together only deliberately. + */ +function referenceValues( + node: FramedNode, + alias: string, + field: ReferenceField, + schema: SearchSchema | undefined, +): readonly string[] { + const targetName = labelSourceNameOf(field); + if (schema === undefined || targetName === undefined) { + return irisOf(node, alias); + } + // Guaranteed declared: `searchSchema` validates that every named target + // resolves, and the projection only ever runs against a type of its schema. + const target = rootTypeNamed(schema, targetName) as RootType; + if (target.key === undefined) { + return irisOf(node, alias); + } + const keyFieldName = target.key.field; + return valuesOf(node, alias) + .map((value) => { + const iri = iriString(value); + return iri === undefined + ? undefined + : documentKeyOf( + target, + iri, + isObject(value) ? keyCandidatesOf(value, target, keyFieldName) : [], + ); + }) + .filter((value): value is string => value !== undefined); +} + /** * Project a text field. **Display** (when `output`) preserves *every* language * present – one label per language (accents preserved, untagged under `und`), diff --git a/packages/search/src/schema.ts b/packages/search/src/schema.ts index bdfa755b..a5c3e243 100644 --- a/packages/search/src/schema.ts +++ b/packages/search/src/schema.ts @@ -353,6 +353,57 @@ export interface SearchTypeBase { readonly fields: readonly SearchField[]; } +/** + * Which declared field of a {@link RootType} holds its **document key**, when + * that key is not the node’s own IRI. It has the shape of + * {@link SearchTypeBase.labelField} – *which field is the label* – and says + * *which field holds the key*: a statement about the search document, never + * about the world, so the words a deployment reaches for (identity, alignment, + * canonical, authority) stay in the deployment’s own schema comments. + * + * Two things that already hold of keys then hold here too, without a new rule: + * several nodes sharing a key are **one document** (the writer upserts by `id`, + * so a deployment that wants particular content on the merged document attaches + * a transform, and one that does not gets last-writer-wins), and a reference + * into a keyed type stores the **target’s key** – a `lookup`/`labelSource` + * already means *this field holds ids of documents in that collection* + * (ADR 20), which storing the node IRI would break. + * + * See [ADR 22](../../docs/decisions/0022-key-a-root-type-on-a-declared-field.md). + */ +export interface KeyField { + /** + * The `name` of a declared field of this type whose values are the key + * candidates: a path-bearing, `array` {@link ReferenceField} that is not + * `inline` ({@link searchSchema} validates all four). Being an ordinary field + * is the point – its extraction branch already exists, a reader transform + * that repairs reference values covers it the day it is declared, and its own + * {@link ReferenceField.transform} is where IRI normalisation lives, so two + * spellings of one IRI become one candidate before anything chooses among + * them. + * + * A transform that **replaces** a root’s quads must re-emit this field, the + * existing rule that *a field the document needs must be in the stream* + * applied to one more field; a transform that only adds never meets it. + */ + readonly field: string; + /** + * Choose the key among the candidates – the key field’s values after its + * `transform`, IRI-filtered, deduplicated and sorted, so the default is + * deterministic whatever order the CONSTRUCT returned them in. Return one of + * them, or `undefined` to keep the node’s own IRI; anything else throws at + * projection. Not consulted for a node whose key field is empty. + * + * Must be **pure**: the same function keys the document and every reference + * to it, so a `pick` that consulted the network or the clock could key the + * two differently and leave a reference dangling. LDE never inspects an + * IRI’s shape – it asks. + * + * @default the first candidate + */ + readonly pick?: (candidates: readonly string[]) => string | undefined; +} + /** * A **Root Type**: a {@link SearchType} that is indexed. It declares a `class`, * roots are selected for it, a Writer owns a collection for it, and the @@ -365,6 +416,9 @@ export interface RootType extends SearchTypeBase { * key a {@link SearchSchema} maps this type under. Its presence is what makes * a type a Root Type – and so what gives it a collection. */ readonly class: string; + /** Which declared field holds this type’s document key, when it is not the + * node’s own IRI ({@link KeyField}, {@link documentKeyOf}). */ + readonly key?: KeyField; } /** @@ -383,6 +437,10 @@ export interface ReferenceType extends SearchTypeBase { * {@link SearchField} discriminates by `kind`: an indexed Reference Type * fails to compile, not at run time. */ readonly class?: never; + /** A Reference Type has no document key to declare: it is nested inside its + * referrer rather than keyed in a collection of its own. `never` for the + * same reason `class` is. */ + readonly key?: never; } /** @@ -452,6 +510,21 @@ const referenceTypesBySchema = new WeakMap< ReadonlyMap >(); +/** + * The Root Type name index each {@link SearchSchema} carries alongside its + * class-keyed map. A schema is keyed by `class` because that is what a selector + * and a Writer address it by, while every declaration that points at another + * type – a `lookup`’s `target`, an `idOnly`’s `labelSource`, a join edge – + * names it. Kept beside the map rather than rebuilt per consumer, so the + * projection, the label-source validation and an adapter’s lookup resolve a + * target the same way ({@link rootTypeNamed}) instead of each maintaining a + * by-name map of its own. + */ +const rootTypesBySchema = new WeakMap< + SearchSchema, + ReadonlyMap +>(); + /** Whether a declared type is a {@link RootType} (declares a `class`) rather * than a {@link ReferenceType}. */ function isRootType(searchType: SearchType): searchType is RootType { @@ -505,14 +578,21 @@ export function searchSchema( ); assertResolvableInlineReferences(types, referenceTypes); assertServiceableNestedFields(referenceTypes); - assertResolvableLabelSources(types); - // The one blessed cast: only this validated constructor mints the brand. + // The one blessed cast: only this validated constructor mints the brand. Built + // BEFORE the schema-wide assertions that resolve a type by name, so they can + // use `rootTypeNamed` – the same reading the projection and an adapter's + // lookup use – rather than a by-name map of their own. The schema stays local + // until every assertion has passed, so an invalid one is never observable. + const rootTypes = types.filter(isRootType); const schema = new Map( - types - .filter(isRootType) - .map((searchType) => [searchType.class, searchType]), + rootTypes.map((searchType) => [searchType.class, searchType]), ) as unknown as SearchSchema; referenceTypesBySchema.set(schema, referenceTypes); + rootTypesBySchema.set( + schema, + new Map(rootTypes.map((searchType) => [searchType.name, searchType])), + ); + assertResolvableLabelSources(types, schema); // Build the join graph eagerly and discard it: it is cached per schema, and // building it is what enforces the schema-wide join rules (one joinable // reference per target, every target an indexed Root Type, no cycles). A @@ -535,6 +615,25 @@ export function referenceTypeNamed( return referenceTypesBySchema.get(schema)?.get(name); } +/** + * The {@link RootType} a declaration names – a `lookup`’s `target`, an + * `idOnly`’s {@link ReferenceField.labelSource} ({@link labelSourceNameOf}), a + * join edge – or `undefined` when the schema declares no Root Type by that + * name. The one reading of *which type does this point at*, so the projection + * (which re-keys a reference through its target’s {@link KeyField}), the + * label-source validation and an adapter’s lookup cannot resolve a target + * differently. + * + * Complements {@link referenceTypeNamed}: the two name indexes are disjoint, + * because {@link searchSchema} rejects a name declared twice. + */ +export function rootTypeNamed( + schema: SearchSchema, + name: string, +): RootType | undefined { + return rootTypesBySchema.get(schema)?.get(name); +} + /** Whether a field is an inline reference – a {@link ReferenceField} whose * `ref` carries its referent’s projected fields ({@link ReferenceType}). */ export function isInlineReference( @@ -801,15 +900,16 @@ export function labelSourceNameOf(field: ReferenceField): string | undefined { * can actually serve labels ({@link labelFieldOf}). Checked schema-wide, * because a single declaration cannot see its siblings. * - * That leaves only a {@link RootType}, without naming one: a label field is - * `searchable`, and {@link assertServiceableNestedFields} already rejects a - * `searchable` field on a Reference Type – so a Reference Type can never serve - * labels, and a resolved label always has a collection to come from. + * Resolved through {@link rootTypeNamed}, which is exactly the set that can + * serve labels: a label field is `searchable`, and + * {@link assertServiceableNestedFields} – which runs first – already rejects a + * `searchable` field on a Reference Type, so a resolved label always has a + * collection to come from. */ -function assertResolvableLabelSources(types: readonly SearchType[]): void { - const byName = new Map( - types.map((searchType) => [searchType.name, searchType]), - ); +function assertResolvableLabelSources( + types: readonly SearchType[], + schema: SearchSchema, +): void { for (const searchType of types) { for (const field of searchType.fields) { const labelSource = (field as { readonly labelSource?: string }) @@ -836,10 +936,16 @@ function assertResolvableLabelSources(types: readonly SearchType[]): void { if (sourceName === undefined) { continue; } - const source = byName.get(sourceName); + const source = rootTypeNamed(schema, sourceName); if (source === undefined) { + // A name that IS declared, just not as a Root Type, is the confusing + // case: telling the author to declare a type they already declared + // would send them looking in the wrong place. Only a Root Type has a + // collection to resolve against, so name that instead. throw new Error( - `Reference “${searchType.name}.${field.name}” names unknown label source “${sourceName}”; declare a SearchType with that name.`, + referenceTypeNamed(schema, sourceName) === undefined + ? `Reference “${searchType.name}.${field.name}” names unknown label source “${sourceName}”; declare a SearchType with that name.` + : `Reference “${searchType.name}.${field.name}” names label source “${sourceName}”, which is a Reference Type; a label source must be a Root Type, since a resolved label is read from that type’s own collection.`, ); } if (labelFieldOf(source) === undefined) { @@ -884,7 +990,14 @@ export interface SearchTypeIssue { | 'joinable-not-allowed' | 'joinable-without-label-source' | 'joinable-with-inline-ref' - | 'reserved-field-name'; + | 'reserved-field-name' + | 'key-field-unknown' + | 'key-field-not-reference' + | 'key-field-without-path' + | 'key-field-not-array' + | 'key-field-inline' + | 'key-pick-not-a-function' + | 'key-not-allowed'; } /** @@ -996,7 +1109,10 @@ const LOCALE_PATTERN = /^[A-Za-z0-9]+(-[A-Za-z0-9]+)*$/; * field has exactly one), is not an `inline` reference (a projection value is * a bare IRI, with no referent to carry fields from), and no two fields * declare the same projection value – which would leave a consumer reading - * the dataset off a declaration no rule picks between. + * the dataset off a declaration no rule picks between; + * - a {@link RootType.key} names a declared, `path`-bearing, `array`, + * non-`inline` `reference` field of this type, and its `pick` is a function + * ({@link keyIssues}); a {@link ReferenceType} declares no key at all. * * Pure and total: returns every issue rather than throwing; * {@link assertValidSearchType} is the throwing entry point. @@ -1159,6 +1275,59 @@ export function validateSearchType( } } } + issues.push(...keyIssues(searchType)); + return issues; +} + +/** + * The issues a type’s {@link RootType.key} declaration carries, each filed + * under the field name the declaration names – so a message reads like every + * other one, naming the field the rule is about rather than a member. + * + * The key field must be an ordinary declared field the extraction already + * reads and the projection already applies a `transform` to: a `reference` + * (candidates are IRIs), with a `path` (a key read from the graph, not one + * derived from a document that is keyed already), `array` (a node may offer + * several candidates, and a single-valued field would drop all but the first + * before `pick` ever saw them), and not `inline` (which carries the referent’s + * fields rather than its IRI, so it offers no candidate at all). + */ +function keyIssues(searchType: SearchType): readonly SearchTypeIssue[] { + const key = (searchType as RootType).key; + if (key === undefined) { + return []; + } + const issue = (reason: SearchTypeIssue['reason']) => ({ + field: key.field, + reason, + }); + // A Reference Type is nested inside its referrer, never keyed in a collection + // of its own – so a key on one states a rule nothing could apply. + if (searchType.class === undefined) { + return [issue('key-not-allowed')]; + } + const issues: SearchTypeIssue[] = []; + if (key.pick !== undefined && typeof key.pick !== 'function') { + issues.push(issue('key-pick-not-a-function')); + } + const field = fieldNamed(searchType, key.field); + if (field === undefined) { + issues.push(issue('key-field-unknown')); + return issues; + } + if (field.kind !== 'reference') { + issues.push(issue('key-field-not-reference')); + return issues; + } + if (field.path === undefined) { + issues.push(issue('key-field-without-path')); + } + if (field.array !== true) { + issues.push(issue('key-field-not-array')); + } + if (field.ref?.strategy === 'inline') { + issues.push(issue('key-field-inline')); + } return issues; } @@ -1386,6 +1555,80 @@ export function fieldNamed( return searchType.fields.find((field) => field.name === name); } +/** + * The **document key** of one node of `searchType`: what the projection writes + * as the document’s `id`, and what a reference into this type stores. + * + * The whole rule, in one place, so the projection and any transform that needs + * to know a node’s key read the same answer: + * + * 1. a type declaring no {@link RootType.key} keys on `nodeIri` – the node’s own + * IRI, as every type does today; + * 2. otherwise the key field’s `rawValues` become **candidates**: its + * {@link ReferenceField.transform} (where IRI normalisation lives, so two + * spellings of one IRI become one candidate), then the + * {@link isAbsoluteIri} filter, then dedupe and sort – so the default is + * deterministic whatever order the CONSTRUCT returned them in; + * 3. no candidate keys on `nodeIri`, without consulting {@link KeyField.pick}; + * 4. otherwise {@link KeyField.pick} chooses (defaulting to the first + * candidate), and `undefined` keeps `nodeIri`. + * + * So a key is always either the node’s own IRI or an IRI the graph offered for + * that node – never one invented in between. A `pick` returning anything else + * throws, naming the node and the candidates: it is a bug in a pure function + * that keys both the document and every reference to it, and letting it through + * would key those two differently. + * + * @param nodeIri the node’s own IRI – the key it falls back to + * @param rawValues the key field’s values **as a reference field reads them** – + * a node’s `@id` or a bare-string value, before the field’s `transform`. A + * caller that reads them itself (a transform working on quads) must apply the + * same rule and skip a literal-valued object, or it will compute a key for a + * value the projection never saw and key the two differently. + */ +export function documentKeyOf( + searchType: RootType, + nodeIri: string, + rawValues: readonly string[], +): string { + const key = searchType.key; + if (key === undefined) { + return nodeIri; + } + // Guaranteed a declared reference field by `searchSchema`, which is what + // validates a `key` at all. + const { transform } = fieldNamed(searchType, key.field) as ReferenceField; + const candidates = [ + ...new Set( + (transform === undefined ? rawValues : rawValues.map(transform)).filter( + isAbsoluteIri, + ), + ), + ].sort(); + if (candidates.length === 0) { + return nodeIri; + } + const picked = (key.pick ?? firstCandidate)(candidates); + if (picked === undefined) { + return nodeIri; + } + if (!candidates.includes(picked)) { + throw new Error( + `The “pick” of “${searchType.name}.key” returned “${picked}” for <${nodeIri}>, which is not among its candidates (${candidates + .map((candidate) => `<${candidate}>`) + .join( + ', ', + )}); a key must be one of them, or “undefined” to keep the node’s own IRI.`, + ); + } + return picked; +} + +/** The {@link KeyField.pick} a type that declares none falls back to. */ +function firstCandidate(candidates: readonly string[]): string { + return candidates[0]; +} + /** * The **IR Alias** predicate for a field: `urn:lde:‹SearchType.name›/‹field.name›`. * The extraction CONSTRUCT emits a field’s value under this minted predicate, and diff --git a/packages/search/test/project.test.ts b/packages/search/test/project.test.ts index ff347185..387c199d 100644 --- a/packages/search/test/project.test.ts +++ b/packages/search/test/project.test.ts @@ -1646,3 +1646,392 @@ describe('projection-time values', () => { }); }); }); + +describe('document keys (a type keyed on a declared field)', () => { + const SCHEMA_ORG = 'https://schema.org/'; + const GEONAMES = 'https://sws.geonames.org/'; + const WIKIDATA = 'https://www.wikidata.org/entity/'; + + /** A deployment’s IRI normalisation: one spelling per alignment target, so two + * publishers writing the same place alike key it alike. */ + const normaliseIri = (iri: string) => + iri.replace(/^http:/, 'https:').replace(/\/$/, ''); + + const place = defineSearchType({ + name: 'Place', + class: `${SCHEMA_ORG}Place`, + labelField: 'name', + // A preference order, not a filter: GeoNames first, then any other covered + // source; nothing matched keeps the publisher’s own node. + key: { + field: '_sameAs', + pick: (candidates) => + candidates.find((iri) => iri.startsWith(GEONAMES)) ?? + candidates.find((iri) => iri.startsWith(WIKIDATA)), + }, + fields: [ + { + name: 'name', + kind: 'text', + path: `<${SCHEMA_ORG}name>`, + locales: ['und'], + output: true, + searchable: { weight: 1 }, + }, + { + // Internal: extracted and read for the key, then pruned before the writer. + name: '_sameAs', + kind: 'reference', + array: true, + path: `<${SCHEMA_ORG}sameAs>`, + transform: normaliseIri, + }, + ], + }); + + const work = defineSearchType({ + name: 'CreativeWork', + class: `${SCHEMA_ORG}CreativeWork`, + fields: [ + { + name: 'locationCreated', + kind: 'reference', + path: `<${SCHEMA_ORG}locationCreated>`, + facetable: true, + output: true, + ref: { strategy: 'lookup', target: 'Place' }, + }, + { + name: 'about', + kind: 'reference', + array: true, + path: `<${SCHEMA_ORG}about>`, + facetable: true, + labelSource: 'Place', + }, + { + // Names no target, so it never claimed to hold a collection’s ids. + name: 'unnamed', + kind: 'reference', + array: true, + path: `<${SCHEMA_ORG}mentions>`, + facetable: true, + }, + ], + }); + + const keyedSchema = searchSchema(place, work); + const placeKey = (field: string) => alias('Place', field); + const workKey = (field: string) => alias('CreativeWork', field); + + /** A framed place node, with the raw values the CONSTRUCT emitted for it. */ + const placeNode = (id: string, ...sameAs: string[]) => ({ + '@id': id, + [placeKey('name')]: [{ '@value': 'Kessel' }], + ...(sameAs.length === 0 + ? {} + : { [placeKey('_sameAs')]: sameAs.map((iri) => ({ '@id': iri })) }), + }); + + it('keys a root on the field’s value, not on the node IRI', () => { + const document = projectDocument( + placeNode('https://ex/place/1', `${GEONAMES}2751283`), + place, + keyedSchema, + ); + + expect(document.id).toBe(`${GEONAMES}2751283`); + // The key field itself is internal, so it never reaches the writer. + expect(document).not.toHaveProperty('_sameAs'); + }); + + it('keeps the node’s own IRI when the key field is empty', () => { + expect( + projectDocument(placeNode('https://ex/place/1'), place, keyedSchema).id, + ).toBe('https://ex/place/1'); + }); + + it('keeps the node’s own IRI when pick declines every candidate', () => { + // A candidate in no source `pick` prefers: the publisher keeps its node. + expect( + projectDocument( + placeNode('https://ex/place/1', 'https://ex/other/1'), + place, + keyedSchema, + ).id, + ).toBe('https://ex/place/1'); + }); + + it('applies the key field’s transform before pick, so spellings merge', () => { + // Two publishers spelling one GeoNames IRI differently – `http://` and a + // trailing slash – project to ONE document key, which is what makes the + // writer’s upsert merge them. + const first = projectDocument( + placeNode('https://a/place/1', 'http://sws.geonames.org/2751283/'), + place, + keyedSchema, + ); + const second = projectDocument( + placeNode('https://b/place/9', `${GEONAMES}2751283`), + place, + keyedSchema, + ); + + expect(first.id).toBe(`${GEONAMES}2751283`); + expect(second.id).toBe(first.id); + }); + + it('sorts and dedupes the candidates, whatever order the CONSTRUCT returned', () => { + // `pick` here takes the FIRST candidate (the default), so the answer is + // stable only because the candidates are sorted before it sees them. + const unordered = defineSearchType({ + ...place, + name: 'AnyPlace', + class: 'urn:x:AnyPlace', + key: { field: '_sameAs' }, + }); + const anySchema = searchSchema(unordered); + const candidates = [`${WIKIDATA}Q1`, `${GEONAMES}1`, `${WIKIDATA}Q1`]; + const forward = { + '@id': 'https://ex/place/1', + [alias('AnyPlace', '_sameAs')]: candidates.map((iri) => ({ '@id': iri })), + }; + const reversed = { + '@id': 'https://ex/place/1', + [alias('AnyPlace', '_sameAs')]: [...candidates] + .reverse() + .map((iri) => ({ '@id': iri })), + }; + + expect(projectDocument(forward, unordered, anySchema).id).toBe( + `${GEONAMES}1`, + ); + expect(projectDocument(reversed, unordered, anySchema).id).toBe( + `${GEONAMES}1`, + ); + }); + + it('throws when pick returns a value that is not a candidate', () => { + // A key must be an IRI the graph offered for that node – the same pure + // function keys the document AND every reference to it, so a `pick` that + // invents one would key the two differently. + const invented = defineSearchType({ + ...place, + name: 'Invented', + class: 'urn:x:Invented', + key: { field: '_sameAs', pick: () => 'https://elsewhere/1' }, + }); + + expect(() => + projectDocument( + { + '@id': 'https://ex/place/1', + [alias('Invented', '_sameAs')]: [{ '@id': `${GEONAMES}1` }], + }, + invented, + searchSchema(invented), + ), + ).toThrow( + /returned “https:\/\/elsewhere\/1” for , which is not among its candidates \(\)/, + ); + }); + + it('stores a referent’s key on a lookup and on a labelSource reference', () => { + // Both name their target, which is what a `lookup`/`labelSource` already + // means: this field holds ids of documents in that collection. + const document = projectDocument( + { + '@id': 'https://ex/work/1', + [workKey('locationCreated')]: { + '@id': 'https://ex/place/1', + [placeKey('_sameAs')]: [{ '@id': `${GEONAMES}2751283` }], + }, + [workKey('about')]: [ + { + '@id': 'https://ex/place/2', + [placeKey('_sameAs')]: [{ '@id': `${WIKIDATA}Q1` }], + }, + ], + }, + work, + keyedSchema, + ); + + expect(document.locationCreated).toBe(`${GEONAMES}2751283`); + expect(document.about).toEqual([`${WIKIDATA}Q1`]); + }); + + it('stores the referent’s own IRI when it offers no key', () => { + // An unaligned place: the reference still resolves, against the document + // that place is written under. + const document = projectDocument( + { + '@id': 'https://ex/work/1', + [workKey('locationCreated')]: { '@id': 'https://ex/place/1' }, + // A reference naming no target is never re-keyed, even into a keyed type. + [workKey('unnamed')]: [ + { + '@id': 'https://ex/place/2', + [placeKey('_sameAs')]: [{ '@id': `${GEONAMES}1` }], + }, + ], + }, + work, + keyedSchema, + ); + + expect(document.locationCreated).toBe('https://ex/place/1'); + expect(document.unnamed).toEqual(['https://ex/place/2']); + }); + + it('re-keys a referent given as a bare IRI, and drops a value that is none', () => { + // `schema:sameAs` and friends range on `schema:URL`, which a source may emit + // as a literal – so a reference value may arrive as a bare string, carrying + // no candidates of its own; a value that is no IRI at all yields nothing, + // exactly as it does for an unkeyed reference. + const document = projectDocument( + { + '@id': 'https://ex/work/1', + [workKey('about')]: ['https://ex/place/3', { '@value': 'Kessel' }], + }, + work, + keyedSchema, + ); + + expect(document.about).toEqual(['https://ex/place/3']); + }); + + it('runs the referring field’s transform on the key, not the node IRI', () => { + // A `transform` transforms what the field STORES, and for a keyed target + // that is the key – so a transform written against the referent’s node IRIs + // sees a key instead. Pinned because the two are only worth declaring + // together deliberately. + const seen: string[] = []; + const withTransform = defineSearchType({ + ...work, + name: 'TransformingWork', + class: 'urn:x:TransformingWork', + fields: [ + { + name: 'about', + kind: 'reference', + array: true, + path: `<${SCHEMA_ORG}about>`, + facetable: true, + labelSource: 'Place', + transform: (value) => { + seen.push(value); + return value; + }, + }, + ], + }); + projectDocument( + { + '@id': 'https://ex/work/1', + [alias('TransformingWork', 'about')]: [ + { + '@id': 'https://ex/place/1', + [placeKey('_sameAs')]: [{ '@id': `${GEONAMES}1` }], + }, + ], + }, + withTransform, + searchSchema(place, withTransform), + ); + + expect(seen).toEqual([`${GEONAMES}1`]); + }); + + it('leaves a reference into an unkeyed target alone', () => { + const unkeyed = defineSearchType({ + name: 'Place', + class: `${SCHEMA_ORG}Place`, + labelField: 'name', + fields: place.fields, + }); + const document = projectDocument( + { + '@id': 'https://ex/work/1', + [workKey('locationCreated')]: { + '@id': 'https://ex/place/1', + [placeKey('_sameAs')]: [{ '@id': `${GEONAMES}1` }], + }, + }, + work, + searchSchema(unkeyed, work), + ); + + expect(document.locationCreated).toBe('https://ex/place/1'); + }); + + it('leaves references as they are when projected without a schema', () => { + // No schema, no way to resolve the target – the same graceful degradation + // an inline reference makes. + const document = projectDocument( + { + '@id': 'https://ex/work/1', + [workKey('locationCreated')]: { + '@id': 'https://ex/place/1', + [placeKey('_sameAs')]: [{ '@id': `${GEONAMES}1` }], + }, + }, + work, + ); + + expect(document.locationCreated).toBe('https://ex/place/1'); + }); + + it('assigns the key before any derive runs', () => { + const withDerive = defineSearchType({ + ...place, + name: 'DerivedPlace', + class: 'urn:x:DerivedPlace', + key: { field: '_sameAs' }, + fields: [ + ...place.fields, + { + name: 'keyedOn', + kind: 'keyword', + output: true, + derive: (document) => document.id, + }, + ], + }); + + expect( + projectDocument( + { + '@id': 'https://ex/place/1', + [alias('DerivedPlace', '_sameAs')]: [{ '@id': `${GEONAMES}1` }], + }, + withDerive, + searchSchema(withDerive), + ).keyedOn, + ).toBe(`${GEONAMES}1`); + }); + + it('lets two distinct roots project to one document key', async () => { + // Two publishers’ nodes for one place. The projection emits both documents, + // under one `id`; folding them is the writer’s upsert, not the projection’s. + const quads = new Parser({ format: 'N-Triples' }).parse(` + <${placeKey('_sameAs')}> <${GEONAMES}2751283> . + <${placeKey('_sameAs')}> <${GEONAMES}2751283> . + `); + + const documents: SearchDocument[] = []; + for await (const document of projectRoots( + quads, + ['https://a/place/1', 'https://b/place/9'], + keyedSchema, + place, + )) { + documents.push(document); + } + + expect(documents.map((document) => document.id)).toEqual([ + `${GEONAMES}2751283`, + `${GEONAMES}2751283`, + ]); + }); +}); diff --git a/packages/search/test/schema.test.ts b/packages/search/test/schema.test.ts index 67166e7e..851c8879 100644 --- a/packages/search/test/schema.test.ts +++ b/packages/search/test/schema.test.ts @@ -4,6 +4,7 @@ import { assertValidSearchType, datasetField, displayFieldName, + documentKeyOf, displayFieldPattern, displayLangOf, facetableFields, @@ -22,12 +23,15 @@ import { physicalFields, referenceFields, referenceTypeNamed, + rootTypeNamed, searchableFields, searchSchema, sortableFields, unixSecondsToIso, validateSearchType, + type KeyField, type ReferenceStrategy, + type RootType, type SearchField, type SearchType, type TextField, @@ -1330,6 +1334,36 @@ describe('searchSchema validation', () => { ).toThrow(/must declare an output, searchable text field “name”/); }); + it('names a Reference Type label source for what it is, not as unknown', () => { + // The name IS declared, just not as a Root Type – so “declare a + // SearchType with that name” would send an author looking in the wrong + // place. Reachable only through a lookup: `labelSource` on a nested field + // is rejected earlier, and a Reference Type carrying a label field is + // rejected for being searchable. + expect(() => + searchSchema( + { + name: 'AddressRef', + fields: [{ name: 'street', kind: 'keyword', output: true }], + }, + { + name: 'Organization', + class: 'https://example.org/Organization', + fields: [ + { + name: 'address', + kind: 'reference', + output: true, + ref: { strategy: 'lookup', target: 'AddressRef' }, + }, + ], + }, + ), + ).toThrow( + /names label source “AddressRef”, which is a Reference Type; a label source must be a Root Type/, + ); + }); + it('rejects a Reference Type as a label source: it cannot be searchable', () => { // Why a label source is always a Root Type, and so always has a // collection to resolve from: a Reference Type carries `output` only. @@ -1510,3 +1544,219 @@ describe('date storage codec', () => { expect(bce).toBeLessThan(ce); }); }); + +describe('key fields', () => { + const SCHEMA_ORG = 'https://schema.org/'; + const GEONAMES = 'https://sws.geonames.org/'; + + const sameAsField: SearchField = { + name: '_sameAs', + kind: 'reference', + array: true, + path: `<${SCHEMA_ORG}sameAs>`, + }; + const nameField: SearchField = { + name: 'name', + kind: 'text', + path: `<${SCHEMA_ORG}name>`, + locales: ['und'], + output: true, + searchable: { weight: 1 }, + }; + /** A `Place` keyed on its alignment target, with an overridable key. */ + const keyedPlace = (key: RootType['key'] = { field: '_sameAs' }) => + ({ + name: 'Place', + class: `${SCHEMA_ORG}Place`, + labelField: 'name', + key, + fields: [nameField, sameAsField], + }) as const satisfies SearchType; + + describe('declaration', () => { + it('accepts a key naming a path-bearing, array, non-inline reference field', () => { + expect(() => searchSchema(keyedPlace())).not.toThrow(); + expect(validateSearchType(keyedPlace())).toEqual([]); + }); + + it('rejects a key naming a field the type does not declare', () => { + expect(validateSearchType(keyedPlace({ field: 'absent' }))).toEqual([ + { field: 'absent', reason: 'key-field-unknown' }, + ]); + }); + + it('rejects a key on a field that is not a reference', () => { + // Candidates are IRIs, and the projection reads them as such. + expect(validateSearchType(keyedPlace({ field: 'name' }))).toEqual([ + { field: 'name', reason: 'key-field-not-reference' }, + ]); + }); + + it('rejects a key field with no path: a key is read from the graph', () => { + expect( + validateSearchType({ + ...keyedPlace(), + fields: [ + nameField, + { + name: '_sameAs', + kind: 'reference', + array: true, + derive: () => [], + }, + ], + }), + ).toEqual([{ field: '_sameAs', reason: 'key-field-without-path' }]); + }); + + it('rejects a single-valued key field: it would drop candidates unseen', () => { + // A node may offer several; a single-valued field keeps the first, + // whatever the CONSTRUCT’s order, so `pick` would never see the rest. + expect( + validateSearchType({ + ...keyedPlace(), + fields: [nameField, { ...sameAsField, array: undefined }], + }), + ).toEqual([{ field: '_sameAs', reason: 'key-field-not-array' }]); + }); + + it('rejects an inline key field: it carries fields, not an IRI', () => { + const marker: SearchType = { + name: 'Marker', + fields: [nameField], + }; + const inlineKeyed: SearchType = { + ...keyedPlace(), + fields: [ + nameField, + { ...sameAsField, ref: { strategy: 'inline', typeName: 'Marker' } }, + ], + }; + expect(validateSearchType(inlineKeyed)).toEqual([ + { field: '_sameAs', reason: 'key-field-inline' }, + ]); + expect(() => searchSchema(inlineKeyed, marker)).toThrow( + /“_sameAs” \(key-field-inline\)/, + ); + }); + + it('rejects a pick that is not a function', () => { + // A declaration built outside TypeScript (a generator, plain JS) is what + // this guards; a typed one cannot express it. + const key = { field: '_sameAs', pick: 'first' } as unknown as KeyField; + expect(validateSearchType(keyedPlace(key))).toEqual([ + { field: '_sameAs', reason: 'key-pick-not-a-function' }, + ]); + }); + + it('rejects a key on a Reference Type, which is never keyed at all', () => { + // Nested inside its referrer rather than keyed in a collection of its own. + const nested = { + name: 'Marker', + key: { field: '_sameAs' }, + fields: [sameAsField], + } as unknown as SearchType; + expect(validateSearchType(nested)).toEqual([ + { field: '_sameAs', reason: 'key-not-allowed' }, + ]); + }); + }); + + describe('documentKeyOf', () => { + it('keys on the node’s own IRI for a type that declares no key', () => { + const unkeyed: RootType = { + name: 'Place', + class: `${SCHEMA_ORG}Place`, + fields: [sameAsField], + }; + expect( + documentKeyOf(unkeyed, 'https://ex/place/1', [`${GEONAMES}1`]), + ).toBe('https://ex/place/1'); + }); + + it('defaults to the first candidate, sorted and deduplicated', () => { + // Sorting is what makes the default deterministic whatever order the + // CONSTRUCT returned the values in. + const place = keyedPlace(); + expect( + documentKeyOf(place, 'https://ex/place/1', [ + `${GEONAMES}2`, + `${GEONAMES}1`, + `${GEONAMES}1`, + ]), + ).toBe(`${GEONAMES}1`); + }); + + it('applies the key field’s transform, then filters non-IRIs', () => { + const place = { + ...keyedPlace(), + fields: [ + nameField, + { + ...sameAsField, + transform: (iri: string) => iri.replace(/\/$/, ''), + }, + ], + } as const satisfies SearchType; + expect( + documentKeyOf(place, 'https://ex/place/1', [ + 'boerenbont', + `${GEONAMES}1/`, + ]), + ).toBe(`${GEONAMES}1`); + }); + + it('keeps the node’s IRI when nothing survives, without consulting pick', () => { + let consulted = false; + const place = keyedPlace({ + field: '_sameAs', + pick: (candidates) => { + consulted = true; + return candidates[0]; + }, + }); + + expect(documentKeyOf(place, 'https://ex/place/1', ['_:b0'])).toBe( + 'https://ex/place/1', + ); + expect(consulted).toBe(false); + }); + + it('keeps the node’s IRI when pick declines', () => { + const place = keyedPlace({ field: '_sameAs', pick: () => undefined }); + expect(documentKeyOf(place, 'https://ex/place/1', [`${GEONAMES}1`])).toBe( + 'https://ex/place/1', + ); + }); + + it('throws when pick returns something no candidate offered', () => { + const place = keyedPlace({ + field: '_sameAs', + pick: () => 'https://elsewhere/1', + }); + expect(() => + documentKeyOf(place, 'https://ex/place/1', [`${GEONAMES}1`]), + ).toThrow(/is not among its candidates/); + }); + }); +}); + +describe('rootTypeNamed', () => { + const place: SearchType = { + name: 'Place', + class: 'https://schema.org/Place', + fields: [], + }; + const marker: SearchType = { name: 'Marker', fields: [] }; + + it('resolves a Root Type by the name a declaration points at', () => { + expect(rootTypeNamed(searchSchema(place, marker), 'Place')).toBe(place); + }); + + it('resolves neither a Reference Type nor an undeclared name', () => { + // The two name indexes are disjoint: `searchSchema` rejects a name twice. + const schema = searchSchema(place, marker); + expect(rootTypeNamed(schema, 'Marker')).toBeUndefined(); + expect(rootTypeNamed(schema, 'Absent')).toBeUndefined(); + }); +}); diff --git a/packages/search/vite.config.ts b/packages/search/vite.config.ts index e64239d4..c070a10d 100644 --- a/packages/search/vite.config.ts +++ b/packages/search/vite.config.ts @@ -12,7 +12,7 @@ export default mergeConfig( thresholds: { functions: 100, lines: 100, - branches: 99.58, + branches: 99.61, statements: 100, }, },