From 3657bc4312564a19cc212ca7825cde3016535907 Mon Sep 17 00:00:00 2001 From: David de Boer Date: Mon, 17 Aug 2026 16:31:29 +0200 Subject: [PATCH 1/3] docs(adr-19): say which pairs idOnly replaces, in this package's own terms Two problems, one cause: the ADR reasoned about the deployment that motivated it rather than about the shape of the case, in a package that names neither a domain nor an engine. The factual one first. It listed five properties as pairs `idOnly` removes the reason for, but `idOnly` surfaces a bare `IRI`, and three of those hold **locators** - a content URL, a thumbnail, a landing page. Nothing selects on them, they carry no filter or facet, and typing them `IRI` would assert a selection key that does not exist. They keep the internal-reader-plus-keyword pair, which is still the only declaration yielding a plain `String` over an IRI-valued path. A reader following the ADR as written would have retyped its URLs as identity. The register, second. The narrowing caveat argued from one profile's co-typing of a specific class, which a reader of this package cannot check and does not need: the general statement is that a declaration names the target whose collection resolves its labels, which need not be the widest class the data admits. The counted claim about that deployment's declarations is gone with it - it was wrong as well as out of place. --- ...type-a-filter-by-what-its-field-keys-on.md | 69 ++++++++++--------- 1 file changed, 37 insertions(+), 32 deletions(-) diff --git a/docs/decisions/0019-type-a-filter-by-what-its-field-keys-on.md b/docs/decisions/0019-type-a-filter-by-what-its-field-keys-on.md index 2fee6e97..be5f9fc0 100644 --- a/docs/decisions/0019-type-a-filter-by-what-its-field-keys-on.md +++ b/docs/decisions/0019-type-a-filter-by-what-its-field-keys-on.md @@ -38,32 +38,24 @@ what follows is only what the shape cost, and what it was chosen over. ### The coarse strategy is exhaustive; the refined one narrows -The two strategies do not carry the same guarantee, and a consumer has to know -which it is holding. - -**Coarse is complete.** Every field that keys on identity takes an `IRI`, so -selecting the criterion fields whose `in` element type is `IRI` returns all of -them. A consumer that must not miss a reference uses this one. - -**Refined is a narrowing, not a filter of equal standing.** It keys on -`ref.typeName`, which is the target a deployment **declares** – not necessarily -the only type the data admits there. Linked Open Limburg declares -`about: { typeName: 'Term' }` because SCHEMA-AP-NDE co-types referenced persons -and organizations as `DefinedTerm`, while the profile also admits a Person or an -Organization as the referent. So a consumer asking “which fields could hold this -person IRI?” gets `creator` through `PersonFilter` and does **not** get `about`. -What it returns is correct; what it returns is not everything. - -Declaring every admissible type per field would fix that and cost more than it -buys: a field has one `ref.typeName` because it emits one reference type and -resolves labels from one source, and multiplying the declaration multiplies both. -The narrowing is worth having as a precision tool, not as a completeness claim. - -Refinement is also unavailable where the target is not itself a root type: there -is no `‹Target›Where.id` to resolve the collection through, so coarse is the only -strategy. Both limits are stated in -[the package reference](../reference/search-api-graphql#finding-which-fields-accept-an-iri), -since a consumer meets them before it meets this ADR. +The two strategies do not carry the same guarantee. + +**Coarse is complete.** Every field keying on identity takes an `IRI`, so +selecting the criterion fields whose `in` element is `IRI` returns all of them. + +**Refined returns a subset.** A reference declares **one** `ref.typeName` and one +`labelSource` – no list form, since the field emits one reference type and +resolves labels from one collection. Where the data admits several classes as +referent, the declaration names one and nothing records the rest. So asking +“which fields could hold an IRI of type X?” returns the fields that **declare** +X, not those that may also **hold** one. + +Admitting a list of targets would multiply the emitted types and label sources +for every such field, to answer what coarse already answers. Refinement is a +precision tool, not a completeness claim – and unavailable where the target is +not a root type, since there is no `‹Target›Where.id` to resolve through. Both +limits are in +[the package reference](../reference/search-api-graphql#finding-which-fields-accept-an-iri). ### `kind` is the discriminator, because `idOnly` now exists @@ -75,14 +67,27 @@ The reason those fields were `keyword` was never that they held literals. It was the **output shape**: `output` on a `reference` requires `ref`, and `ref` gave the `{ id, label }` object when what the deployment wanted was a flat list of IRIs. So a `reference` internal field laundered the IRIs and a `keyword` `derive` -surfaced them – seven such pairs in Linked Open Limburg, for `sameAs`, `license`, -`contentUrl`, `thumbnailUrl` and `landingPage`. +surfaced them – a pattern the first deployment to hit this carried over half a +dozen times. + +**`idOnly` replaces that pair only where the values are selection keys.** A +canonical vocabulary URI or a licence is one, and typing it `IRI` states a fact. +A **locator** is not – a content or thumbnail URL, a landing page, a manifest to +dereference. Nothing selects on those, they carry no filter or facet, and `IRI` +would assert a selection key that does not exist. They keep the +internal-reader-plus-`keyword` pair, still the only declaration yielding a plain +`String` over an IRI-valued path. + +So the split is per property, and the pattern that makes those pairs look alike +does not decide it – in that deployment they divide about evenly. A reader who +collapses every pair on sight retypes locators as identity, and a filter typed +`IRI` over a thumbnail URL is a promise nothing keeps. Implementing `idOnly` removes the reason instead of working around it. The marker -would have added a concept to the declaration language every deployment author -reads, to serve two fields, while leaving all seven pairs standing and `kind` a -non-discriminator. `idOnly` adds no concept – it was forward-declared in ADR 3 -and already documented as “the IRI” – and deletes fourteen declarations. +would have added a concept every deployment author reads, to serve two fields, +while leaving every pair standing and `kind` a non-discriminator. `idOnly` adds +no concept – it was forward-declared in ADR 3 and already documented as “the +IRI”. ### `IRI` validates in both directions From 31a2d985975ff50966b0aa21b7ff15a0158a843f Mon Sep 17 00:00:00 2001 From: David de Boer Date: Wed, 19 Aug 2026 21:11:34 +0200 Subject: [PATCH 2/3] fix(search): read an out-of-form year as a year, not an offset or local date MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - expand a year outside the plain four-digit form to the signed six digits ISO 8601 expects, inside isoToUnixSeconds, so both call sites are fixed at once: the projection on the way in and the query compiler’s filter bounds on the way out, which otherwise disagree once either end is patched alone - unexpanded, ‘-1100’ parsed as a bare UTC offset and landed around 1100 CE, and ‘25000’ parsed as a legacy local-time date and landed a year early on a boundary that moved with the host’s timezone – neither yielding NaN, so nothing threw and the value sorted and filtered as another date entirely - deep time is conformant data: SCHEMA-AP-NDE blesses years beyond four digits for schema:dateCreated - document how a date is stored and which year forms the codec accepts --- docs/reference/search.md | 12 +++++ .../test/query-compiler.test.ts | 24 ++++++++++ packages/search/src/schema.ts | 25 +++++++++- packages/search/test/schema.test.ts | 47 +++++++++++++++++++ packages/search/vite.config.ts | 2 +- 5 files changed, 108 insertions(+), 2 deletions(-) diff --git a/docs/reference/search.md b/docs/reference/search.md index b39d5cca..6bcac6d7 100644 --- a/docs/reference/search.md +++ b/docs/reference/search.md @@ -289,6 +289,18 @@ directly. | `date` | `range` (inclusive) | yes | yes | ISO 8601 string (surface) | | `boolean` | `is` | yes | – | boolean (absent = false) | +A **`date`** is stored as Unix seconds – a sortable, range-filterable integer – +and is ISO 8601 at every edge: the projection converts a value on the way in, +the query compiler converts a filter bound on the way out, and the surface emits +a string. A year outside the plain four-digit form is accepted however it is written – +expanded (`-001100-01-01T00:00:00.000Z`) or not (`-1100`, `-250000`, `25000`): +the codec pads it to the signed six digits `Date` needs before parsing, so the +same value indexes and filters alike. Deep time is real data – SCHEMA-AP-NDE +blesses years beyond four digits for `schema:dateCreated` – and an unpadded year +would otherwise parse as something else entirely (a BCE year as a UTC offset, +landing in the wrong era; a five-digit CE year as a legacy local-time date, +landing a year early and differently per host timezone), without ever failing. + **`array` decides a field’s shape**, whatever the graph carries: a declared `array` field stores a list, and a single-valued one stores the first value – for every kind alike, so the projection, the engine collection definition diff --git a/packages/search-typesense/test/query-compiler.test.ts b/packages/search-typesense/test/query-compiler.test.ts index 22f5512f..d8d7951c 100644 --- a/packages/search-typesense/test/query-compiler.test.ts +++ b/packages/search-typesense/test/query-compiler.test.ts @@ -413,6 +413,30 @@ describe('buildSearchParams', () => { ).toBe(`datePosted:<=${max}`); }); + it('reads an unpadded BCE bound as a year, so the range is not inverted', () => { + const min = Date.parse('-001100-01-01T00:00:00.000Z') / 1000; + const max = Date.parse('-000800-01-01T00:00:00.000Z') / 1000; + expect( + buildSearchParams( + { + ...base, + where: [ + { + or: [ + { + field: 'datePosted', + range: { min: '-1100', max: '-0800' }, + }, + ], + }, + ], + }, + schema, + ).filter_by, + ).toBe(`datePosted:[${min}..${max}]`); + expect(min).toBeLessThan(max); + }); + it('compiles orderBy: RELEVANCE → _text_match and a localized field → its sort key', () => { expect( buildSearchParams( diff --git a/packages/search/src/schema.ts b/packages/search/src/schema.ts index 78aafeea..637129b3 100644 --- a/packages/search/src/schema.ts +++ b/packages/search/src/schema.ts @@ -1420,10 +1420,33 @@ const ABSOLUTE_IRI = /^[A-Za-z][A-Za-z0-9+.-]*:\S*$/; * unparseable value. */ export function isoToUnixSeconds(iso: string): number | undefined { - const millis = new Date(iso).getTime(); + const millis = new Date(expandYear(iso)).getTime(); return Number.isNaN(millis) ? undefined : Math.trunc(millis / 1000); } +/** + * Pad a year outside the plain four-digit form to the signed six digits ISO + * 8601 expects, so `Date` reads it as a year at all. Deep time is real data, + * not a corner case: SCHEMA-AP-NDE blesses years beyond four digits for + * `schema:dateCreated`. Left unexpanded, neither era parses as written and + * neither fails loudly – `-1100` is read as a bare UTC offset and lands + * around 1100 **CE** (shifted by the host’s offset), `25000` is read as a + * legacy local-time date and lands a year early, on a boundary that moves + * with the host’s timezone. Either way nothing throws, no field is left + * absent, and the value sorts and filters as if it were another date + * entirely. A plain four-digit year and an already-expanded one pass through + * unchanged, so the form the API itself emits round-trips. + */ +function expandYear(iso: string): string { + // The year, then the rest: a non-digit stops the run, so a longer digit + // string (epoch millis, say) is left alone rather than cut into a year. + const year = /^([+-]?)(\d{1,6})(\D.*|)$/.exec(iso); + if (year === null || (year[1] === '' && year[2].length === 4)) { + return iso; + } + return `${year[1] || '+'}${year[2].padStart(6, '0')}${year[3]}`; +} + /** The inverse of {@link isoToUnixSeconds}: stored Unix seconds → ISO 8601. */ export function unixSecondsToIso(seconds: number): string { return new Date(seconds * 1000).toISOString(); diff --git a/packages/search/test/schema.test.ts b/packages/search/test/schema.test.ts index d6c78f13..072cc579 100644 --- a/packages/search/test/schema.test.ts +++ b/packages/search/test/schema.test.ts @@ -1340,4 +1340,51 @@ describe('date storage codec', () => { it('returns undefined for an unparseable date', () => { expect(isoToUnixSeconds('not-a-date')).toBeUndefined(); }); + + it('reads an unpadded BCE year as a year, not as a UTC offset', () => { + const seconds = isoToUnixSeconds('-1100'); + expect(unixSecondsToIso(seconds ?? 0)).toBe('-001100-01-01T00:00:00.000Z'); + expect(seconds).toBe(isoToUnixSeconds('-001100-01-01T00:00:00.000Z')); + }); + + it('reads a CE year beyond four digits, independent of the host timezone', () => { + expect(unixSecondsToIso(isoToUnixSeconds('25000') ?? 0)).toBe( + '+025000-01-01T00:00:00.000Z', + ); + expect(unixSecondsToIso(isoToUnixSeconds('25000-06-01') ?? 0)).toBe( + '+025000-06-01T00:00:00.000Z', + ); + }); + + it('leaves a plain four-digit year and an expanded one alone', () => { + expect(unixSecondsToIso(isoToUnixSeconds('1999') ?? 0)).toBe( + '1999-01-01T00:00:00.000Z', + ); + expect( + unixSecondsToIso(isoToUnixSeconds('+025000-01-01T00:00:00.000Z') ?? 0), + ).toBe('+025000-01-01T00:00:00.000Z'); + }); + + it('reads a deep-time BCE year beyond four digits', () => { + expect(unixSecondsToIso(isoToUnixSeconds('-250000') ?? 0)).toBe( + '-250000-01-01T00:00:00.000Z', + ); + expect(unixSecondsToIso(isoToUnixSeconds('-38000') ?? 0)).toBe( + '-038000-01-01T00:00:00.000Z', + ); + }); + + it('expands the year of a fuller BCE date, leaving the rest alone', () => { + expect(unixSecondsToIso(isoToUnixSeconds('-1100-05-03') ?? 0)).toBe( + '-001100-05-03T00:00:00.000Z', + ); + }); + + it('orders BCE before CE, and earlier BCE before later', () => { + const deepTime = isoToUnixSeconds('-250000') ?? 0; + const bce = isoToUnixSeconds('-1100') ?? 0; + const ce = isoToUnixSeconds('1100') ?? 0; + expect(deepTime).toBeLessThan(bce); + expect(bce).toBeLessThan(ce); + }); }); diff --git a/packages/search/vite.config.ts b/packages/search/vite.config.ts index 955a1a9d..92f11072 100644 --- a/packages/search/vite.config.ts +++ b/packages/search/vite.config.ts @@ -12,7 +12,7 @@ export default mergeConfig( thresholds: { functions: 100, lines: 100, - branches: 99.52, + branches: 99.53, statements: 100, }, }, From 96422ef39f666bb4af3d3daa90bfbc0d4aa6d5ea Mon Sep 17 00:00:00 2001 From: David de Boer Date: Thu, 20 Aug 2026 11:45:45 +0200 Subject: [PATCH 3/3] fix(search): collapse whitespace around a date, and drop seconds no Date can hold MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Trim a date literal before expanding its year: XSD collapses whitespace around one, so a padded form is legal and arrives verbatim. Left in place it defeated the expansion in both directions – a leading space stopped the match and read a BCE year as a UTC offset, a trailing one survived into the legacy parser and landed a year early, per host timezone. - Drop a numeric date derive outside the range Date represents, as an unparseable string already is: stored, it read back as an error at the surface instead of leaving the field absent. - Say in the docs which window deep time is supported over. --- docs/reference/search.md | 2 ++ packages/search/src/project.ts | 19 ++++++++++++++++--- packages/search/src/schema.ts | 7 ++++++- packages/search/test/project.test.ts | 9 +++++++++ packages/search/test/schema.test.ts | 17 +++++++++++++++++ packages/search/vite.config.ts | 2 +- 6 files changed, 51 insertions(+), 5 deletions(-) diff --git a/docs/reference/search.md b/docs/reference/search.md index 72d99e2a..daa9c0e7 100644 --- a/docs/reference/search.md +++ b/docs/reference/search.md @@ -300,6 +300,8 @@ blesses years beyond four digits for `schema:dateCreated` – and an unpadded ye would otherwise parse as something else entirely (a BCE year as a UTC offset, landing in the wrong era; a five-digit CE year as a legacy local-time date, landing a year early and differently per host timezone), without ever failing. +The window is the one `Date` can represent, ±271,821 years around 1970; a year +outside it leaves the field absent, as any unparseable value does. **`array` decides a field’s shape**, whatever the graph carries: a declared `array` field stores a list, and a single-valued one stores the first value – diff --git a/packages/search/src/project.ts b/packages/search/src/project.ts index 552e0268..97552a4c 100644 --- a/packages/search/src/project.ts +++ b/packages/search/src/project.ts @@ -230,11 +230,17 @@ function applyField( ? derivedIris(derived) : // A derived `date` goes through the same storage codec as a read one: // the field is stored as Unix seconds, so an ISO string returned here - // would otherwise land as a string in an int64 field. A derive that - // already computed seconds returns a number and passes through. + // would otherwise land as a string in an int64 field. field.kind === 'date' && typeof derived === 'string' ? isoToUnixSeconds(derived) - : derived; + : // A derive that computed seconds itself returns a number, which the + // codec never saw and so never range-checked. Seconds outside what + // `Date` can represent – reachable now that deep time is – would + // store fine and then throw on every read at the surface, where an + // unparseable string merely leaves the field absent. Drop it too. + field.kind === 'date' && typeof derived === 'number' + ? storableSeconds(derived) + : derived; // NaN is not a value any kind stores, and a derive is the only route that // can produce one (`setNumber` guards the read route). It serializes as // `null`, which an engine rejects for a numeric field – so drop it, leaving @@ -588,6 +594,13 @@ function derivedIris(value: unknown): unknown { return typeof value === 'string' && isAbsoluteIri(value) ? value : undefined; } +/** Stored Unix seconds a `Date` can represent, or `undefined` – the same + * answer {@link isoToUnixSeconds} gives a string it cannot parse, so the two + * routes into a `date` field agree on what is storable. */ +function storableSeconds(seconds: number): number | undefined { + return Number.isNaN(new Date(seconds * 1000).getTime()) ? undefined : seconds; +} + function toInteger(literal: string | undefined): number | undefined { return literal === undefined ? undefined : Math.trunc(Number(literal)); } diff --git a/packages/search/src/schema.ts b/packages/search/src/schema.ts index 6fcef924..bdfa755b 100644 --- a/packages/search/src/schema.ts +++ b/packages/search/src/schema.ts @@ -1454,7 +1454,12 @@ const ABSOLUTE_IRI = /^[A-Za-z][A-Za-z0-9+.-]*:\S*$/; * unparseable value. */ export function isoToUnixSeconds(iso: string): number | undefined { - const millis = new Date(expandYear(iso)).getTime(); + // Trim first: XSD collapses whitespace around a date literal before parsing, + // so a padded lexical form is legal and reaches here verbatim. Left in place + // it defeats the year expansion below in both directions – a leading space + // stops the match, a trailing one survives it into the legacy parser – and + // each lands the value in a different year than it reads. + const millis = new Date(expandYear(iso.trim())).getTime(); return Number.isNaN(millis) ? undefined : Math.trunc(millis / 1000); } diff --git a/packages/search/test/project.test.ts b/packages/search/test/project.test.ts index 11f6ded1..1ef72e0f 100644 --- a/packages/search/test/project.test.ts +++ b/packages/search/test/project.test.ts @@ -557,6 +557,14 @@ describe('projectDocument', () => { kind: 'date', derive: () => Date.parse('not-a-date') / 1000, }, + // Seconds past what `Date` can represent are dropped too: stored, + // they read back as an error at the surface rather than as a value, + // where an unparseable string simply leaves the field absent. + { + name: 'temporal', + kind: 'date', + derive: () => -8_640_000_000_001, + }, ], }, ); @@ -564,6 +572,7 @@ describe('projectDocument', () => { expect(document.modified).toBe(1_704_067_200); expect(document).not.toHaveProperty('created'); expect(document).not.toHaveProperty('available'); + expect(document).not.toHaveProperty('temporal'); }); it('holds a reference to absolute IRIs on every route a value can arrive by', () => { diff --git a/packages/search/test/schema.test.ts b/packages/search/test/schema.test.ts index 96b5034f..67166e7e 100644 --- a/packages/search/test/schema.test.ts +++ b/packages/search/test/schema.test.ts @@ -1485,6 +1485,23 @@ describe('date storage codec', () => { ); }); + it('reads a whitespace-padded year as the year it spells', () => { + // XSD collapses whitespace around a date literal, so a padded form is legal + // and reaches the codec verbatim. + for (const padded of [' -1100', '\t-1100', '-1100 ', ' -1100 ']) { + expect(isoToUnixSeconds(padded)).toBe(isoToUnixSeconds('-1100')); + } + expect(isoToUnixSeconds(' 25000 ')).toBe(isoToUnixSeconds('25000')); + expect(isoToUnixSeconds(' 1999 ')).toBe(isoToUnixSeconds('1999')); + }); + + it('returns undefined for a year beyond what Date can represent', () => { + // ±271,821 years is the edge of the Date range; past it there is no value + // to store, so the field is left absent as an unparseable string is. + expect(isoToUnixSeconds('-500000')).toBeUndefined(); + expect(isoToUnixSeconds('-271821')).toBeUndefined(); + }); + it('orders BCE before CE, and earlier BCE before later', () => { const deepTime = isoToUnixSeconds('-250000') ?? 0; const bce = isoToUnixSeconds('-1100') ?? 0; diff --git a/packages/search/vite.config.ts b/packages/search/vite.config.ts index 661f9376..e3f60542 100644 --- a/packages/search/vite.config.ts +++ b/packages/search/vite.config.ts @@ -12,7 +12,7 @@ export default mergeConfig( thresholds: { functions: 100, lines: 100, - branches: 99.56, + branches: 99.57, statements: 100, }, },