Add SchemaBinary fingerprint positional mode

[tim-smart]

Summary

Adds the opt-in fingerprint / positional wire mode for SchemaBinary, selected by envelope flag bit 0. The default evolution-friendly mode is unchanged and stays the default.

• SchemaBinary.toCodec(schema, { fingerprint: true }) and SchemaBinary.parser(schema, { fingerprint: true }) select the new mode
• every frame carries a fixed 8-byte 64-bit FNV-1a hash of the compiled wire layout (offset basis 14695981039346656037, prime 1099511628211, little-endian); a reader whose layout hashes differently rejects the frame rather than guessing
• structs are written positionally: no field ids, a leading presence bitmap for optional fields, no length prefix on fixed-size or self-delimiting leaves, and extra keys behind their own count
• union members are addressed by a canonical varint index (variants by ascending sentinel tag, then remaining members by ascending kind) instead of the kind byte plus 32-bit tag, so declaration order never reaches the wire
• tuples, arrays, and the native leaves keep their default-mode layout

The hash is a Merkle walk over the compiled layout with relative back edges for cycles. It mixes field ids, optionality, wire kinds, variant tags, and array shape; never names, checks, or annotations. An acyclic sub-layout hashes the same whether it is a shared compiled node or written out twice, and a given cycle hashes the same wherever it is entered.

What the hash is, and is not

It hashes the compiled layout graph, not the infinite wire shape that graph denotes. Those coincide for acyclic layouts. They do not once a cycle is involved, because the same unfolding has many finite cyclic representations and de Bruijn style back references encode each of them differently.

Concretely, a self-recursive Tree, the same schema behind one extra non-recursive alias, and a two-node mutual recursion of the same shape all produce byte-identical frames, and all three hash differently and reject each other. Closing that gap needs bisimulation minimisation over the layout graph, which is a lot of machinery for a case that already fails closed, so this PR documents the semantics instead: peers must ship the same schema definition, not merely the same wire shape. The limitation is pinned by a test rather than left to be discovered in production.

Thanks to the review for catching that the original wording promised canonicalisation the implementation does not deliver.

No stream-level fingerprint negotiation: the fingerprint is per frame, so concatenation still works and every frame stands alone.

Results

Sizes are exact; rates are medians of three runs on Node 26.7.0, Linux x64, measured against the optimized default mode in the same tree.

Case	Raw bytes	Encode	Decode	Stream single	Stream batch	Stream fragmented
small record	-39.7%	+3.2%	+2.2%	+4.9%	+7.0%	+14.7%
nested payload	-35.2%	+7.0%	+7.5%	+11.4%	+6.7%	+6.3%
collections	-1.0%	-0.5%	-0.5%	+1.4%	-0.3%	-0.5%
index signatures / 128	+0.3%	+0.8%	-1.7%	+0.8%	+0.5%	+1.0%
index signatures / 512	+0.1%	+1.2%	-1.7%	+2.7%	-1.4%	-0.9%
large repeated records	-29.6%	+13.2%	+9.0%	+10.2%	+10.9%	+9.7%
per-frame repeated record	-23.7%			+8.1%	+5.3%	+2.2%

Raw and compressed sizes for the 200-frame per-frame stream, which is the workload the mode is aimed at:

Format	Raw bytes	gzip -6	zstd
SchemaBinary	27840	3109	3174
SchemaBinary fingerprint	21240	2876	2733
Msgpack	51520	2956	2888

The size result splits by shape. Struct-heavy payloads drop the 5-byte field id and, for fixed-size leaves, the length byte, which is where the 24% to 40% raw savings come from. Index-signature records carry almost no named fields, so they pay the 8-byte fingerprint and save nothing; those two cases get marginally larger. Compression narrows the gap without erasing it: -7.5% gzip and -13.9% zstd on the per-frame stream against -23.7% raw.

The decode gain is reported as a measurement against the optimized parser. The pre-optimization estimate that field varints were 24% of decode time predates the unrolled reader and is not reused as a forecast.

Coverage

Golden vectors for the fingerprint envelope and every positional shape (struct with and without an optional present, union positions, extra-key map, tuple, array, inline leaves), plus:

• fingerprint stability: checks, annotations, decoded-side transformations, property declaration order, union member order, and repeating an acyclic sub-schema instead of sharing one all leave the hash alone
• fingerprint semantics: three factorings of one recursive wire shape produce identical payloads, decode each other in default mode, and hash three different ways in fingerprint mode
• fingerprint sensitivity: rename, added field, optionality, isInt, fieldId, leaf type, tuple arity, Array versus Tuple, Date versus DateTimeUtc, and union membership all change it
• fail-closed paths: mismatched fingerprint, either mode reading the other's frames, truncated fingerprint, truncated payload, leftover bytes, a union position outside the member table, duplicate extra keys, and a required field a newer writer left unreadable
• parser: concatenated frames, one-byte feeds, a split inside the fingerprint, a default-mode frame in a fingerprint stream, maxFrameSize, and truncation at end
• the default mode is asserted byte-identical with no options, {}, and { fingerprint: false }

Verification

• nix develop -c pnpm vitest run --project effect test/unstable/encoding (116 passed)
• nix develop -c pnpm vitest run --project effect (8709 passed, 47 skipped)
• nix develop -c pnpm lint
• nix develop -c pnpm check
• nix develop -c pnpm --dir packages/effect exec node benchmark/schema/SchemaBinary.ts

Stacked on the merged #7369 at 0d9caf9380. Keep this PR open for the EFF-783 simplification pass, which continues on this branch.

Closes EFF-780

🤖 Generated with Claude Code

[changeset-bot]

🦋 Changeset detected

Latest commit: 5bb27c0

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 30 packages

Name	Type
effect	Major
@effect/opentelemetry	Major
@effect/vitest	Major
@effect/ai-anthropic	Major
@effect/ai-openai-compat	Major
@effect/ai-openai	Major
@effect/ai-openrouter	Major
@effect/atom-react	Major
@effect/atom-solid	Major
@effect/atom-vue	Major
@effect/platform-browser	Major
@effect/platform-bun	Major
@effect/platform-deno	Major
@effect/platform-node-shared	Major
@effect/platform-node	Major
@effect/sql-clickhouse	Major
@effect/sql-d1	Major
@effect/sql-libsql	Major
@effect/sql-mssql	Major
@effect/sql-mysql2	Major
@effect/sql-pg	Major
@effect/sql-pglite	Major
@effect/sql-sqlite-bun	Major
@effect/sql-sqlite-do	Major
@effect/sql-sqlite-node	Major
@effect/sql-sqlite-react-native	Major
@effect/sql-sqlite-wasm	Major
@effect/doctest	Major
@effect/openapi-generator	Major
@effect/docgen	Major

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR