Skip to content

fix(search): read an out-of-form year as a year, not an offset or local date - #747

Merged
ddeboer merged 5 commits into
mainfrom
worktree-fix-725-bce-years
Aug 20, 2026
Merged

fix(search): read an out-of-form year as a year, not an offset or local date#747
ddeboer merged 5 commits into
mainfrom
worktree-fix-725-bce-years

Conversation

@ddeboer

@ddeboer ddeboer commented Aug 19, 2026

Copy link
Copy Markdown
Member

isoToUnixSeconds passed its argument straight to Date, which reads a year outside the plain four-digit form as something other than a year – and never as NaN, so nothing threw, no field was left absent, and the value sorted and filtered as another date entirely:

  • -1100 is read as a bare UTC offset and lands around 1100 CE – eleven centuries and a sign out.
  • 25000 is read as a legacy local-time date and lands a year early (+024999-12-31T23:00:00Z on a UTC+1 host), on a boundary that moves with the host’s timezone – so the same source value indexes differently in a UTC container than on a developer’s machine.

The codec now pads the year to the signed six digits ISO 8601 expects before parsing, for both eras. Fixing it inside the codec fixes both call sites at once – the projection on the way in and @lde/search-typesense’s query compiler on the way out – which is what matters here: patching only one end turns a hidden, self-consistent error into a BCE range filter that matches nothing. A plain four-digit year and an already-expanded one pass through unchanged, so the form the API itself emits still round-trips, and a longer digit run (epoch millis, say) is left alone rather than cut into a year.

Deep time is conformant data, not a corner case: SCHEMA-AP-NDE blesses years beyond four digits for schema:dateCreated (-38000), and says nothing about era.

Two follow-ups from review close the gaps around that fix:

  • A whitespace-padded year defeated the expansion, in both directions. XSD collapses whitespace around a date literal, so a padded form is legal and reaches the codec verbatim – nothing applies the facet on the way in, and the GraphQL DateRange bound is a plain string. A leading space stopped the anchored match (" -1100" → 1100 CE again, the very bug this fixes); a trailing one survived it into the legacy parser ("-1100 " → a year early, host-timezone dependent). The codec trims first.
  • A numeric date derive could store seconds no Date can represent. Only a string goes through the codec, so a derive computing its own seconds was never range-checked; stored, it made every later read at the surface throw where an unparseable string simply leaves the field absent. The projection now drops it, next to the existing NaN guard.

The representable window is Date’s own, ±271,821 years around 1970, and the docs now say so rather than promising deep time without a limit.

  • packages/search/src/schema.ts – expand the year in isoToUnixSeconds, trimming first
  • packages/search/src/project.ts – drop a numeric date derive outside the representable range
  • packages/search/test/schema.test.ts – unpadded BCE, deep time, a five-digit CE year (asserted on the absolute instant, so a timezone-dependent parse fails the test), a fuller BCE date, the pass-through forms, BCE-before-CE ordering, whitespace padding, and the range edge
  • packages/search/test/project.test.ts – an out-of-range numeric derive leaves the field absent
  • packages/search-typesense/test/query-compiler.test.ts – an unpadded BCE range bound compiles to a non-inverted range
  • packages/search/vite.config.ts – the branch-coverage ratchet
  • docs/reference/search.md – how a date is stored, which year forms the codec accepts, and over which window
  • docs/decisions/0019-type-a-filter-by-what-its-field-keys-on.md – say which pairs idOnly replaces, in this package’s own terms

Fix #725

…terms

Two problems, one cause: the ADR reasoned about the deployment that
motivated it rather than about the shape of the case, in a package that
names neither a domain nor an engine.

The factual one first. It listed five properties as pairs `idOnly`
removes the reason for, but `idOnly` surfaces a bare `IRI`, and three of
those hold **locators** - a content URL, a thumbnail, a landing page.
Nothing selects on them, they carry no filter or facet, and typing them
`IRI` would assert a selection key that does not exist. They keep the
internal-reader-plus-keyword pair, which is still the only declaration
yielding a plain `String` over an IRI-valued path. A reader following the
ADR as written would have retyped its URLs as identity.

The register, second. The narrowing caveat argued from one profile's
co-typing of a specific class, which a reader of this package cannot
check and does not need: the general statement is that a declaration
names the target whose collection resolves its labels, which need not be
the widest class the data admits. The counted claim about that
deployment's declarations is gone with it - it was wrong as well as out
of place.
@ddeboer
ddeboer force-pushed the worktree-fix-725-bce-years branch from 1b32439 to 9c804c9 Compare August 19, 2026 18:53
@ddeboer ddeboer changed the title fix(search): read a BCE year as a year, not as a UTC offset fix(search): read an out-of-form year as a year, not an offset or local date Aug 19, 2026
@ddeboer ddeboer closed this Aug 19, 2026
@ddeboer ddeboer reopened this Aug 19, 2026
…al date

- expand a year outside the plain four-digit form to the signed six digits
  ISO 8601 expects, inside isoToUnixSeconds, so both call sites are fixed at
  once: the projection on the way in and the query compiler’s filter bounds
  on the way out, which otherwise disagree once either end is patched alone
- unexpanded, ‘-1100’ parsed as a bare UTC offset and landed around 1100 CE,
  and ‘25000’ parsed as a legacy local-time date and landed a year early on a
  boundary that moved with the host’s timezone – neither yielding NaN, so
  nothing threw and the value sorted and filtered as another date entirely
- deep time is conformant data: SCHEMA-AP-NDE blesses years beyond four
  digits for schema:dateCreated
- document how a date is stored and which year forms the codec accepts
@ddeboer
ddeboer force-pushed the worktree-fix-725-bce-years branch 2 times, most recently from 02f9ce9 to 31a2d98 Compare August 19, 2026 19:27
…years

# Conflicts:
#	packages/search/vite.config.ts
…ate can hold

- Trim a date literal before expanding its year: XSD collapses whitespace
  around one, so a padded form is legal and arrives verbatim. Left in place
  it defeated the expansion in both directions – a leading space stopped the
  match and read a BCE year as a UTC offset, a trailing one survived into the
  legacy parser and landed a year early, per host timezone.
- Drop a numeric date derive outside the range Date represents, as an
  unparseable string already is: stored, it read back as an error at the
  surface instead of leaving the field absent.
- Say in the docs which window deep time is supported over.
…years

# Conflicts:
#	docs/decisions/0019-type-a-filter-by-what-its-field-keys-on.md
@ddeboer
ddeboer merged commit 93bc0be into main Aug 20, 2026
4 checks passed
@ddeboer
ddeboer deleted the worktree-fix-725-bce-years branch August 20, 2026 11:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

isoToUnixSeconds mis-parses BCE years as UTC offsets

1 participant