Skip to content

feat(cs): ASAPv1 wire payload for Count Sketch - #79

Draft
GordonYuanyc wants to merge 1 commit into
mainfrom
feat/countsketch-payload
Draft

feat(cs): ASAPv1 wire payload for Count Sketch#79
GordonYuanyc wants to merge 1 commit into
mainfrom
feat/countsketch-payload

Conversation

@GordonYuanyc

Copy link
Copy Markdown
Collaborator

What

Self-describing ASAPv1 wire serialization for Count Sketch → kind_id 0x04 0x00, following the Count-Min pattern from #65 and the KLL pattern from #75.

serialize_to_bytes / deserialize_from_bytes now emit/read the ASAPv1 envelope (replacing the old plain-rmp form), authored in a per-sketch wire submodule (src/sketches/countsketch/wire.rs).

Design decision (docs/asapv1_wire_format.md §3.4 + Q-CS)

Count Sketch mirrors Count-Min's metadata and [counts] payload with exactly one key removed: no counter_type.

CountSketchCounter requires Neg + From<i32>, so counters must be signed and negatable — which leaves i64 as the only wire-eligible type (i32 widens, i128 is not a wire type, f64 doesn't qualify at all). The kind_id already implies it, and a metadata key with a single legal value describes nothing. The variance from Q-CMS is deliberate: the two sketches are not forced symmetric where their type domains differ.

Everything else is identical to §3.2 — hash-spec group derived from the hasher's HashProfile, then rows / cols / mode; payload is the same 1-element positional array [counts], packed row-major. Cells are signed (the sketch adds ±weight), so a decoder must not assume monotonicity.

API narrowing

The wire methods are implemented only for the wire-eligible config (Vector2D<i64> storage, fast/regular mode, a profiled hasher), so the old generic-over-all-storage rmp helpers are gone and MessagePackCodec is narrowed to match. Exotic counters (i32/i128) and non-Vector2D storage must be converted first — same cut CMS made in #65. Decode is a hard cut with no legacy-rmp fallback, consistent with every other sketch.

Fail-closed decode

kind_id check; metadata equality check (hash spec + mode); non-zero dimensions (a crafted cols == 0 would otherwise panic in Vector2D::from_fn via ilog2); and a counts-length check that runs before the allocation, so crafted dimensions cannot drive a huge reserve.

Tests

Nine new tests in the wire module: round-trip; negative-counter round-trip; custom-hasher profile self-describing + cross-rejection; mode in metadata + cross-mode rejection; foreign kind_id (CMS bytes) rejection; zero dimension; dimension/length mismatch; unknown metadata key; and one asserting that adding counter_type back is rejected.

Golden byte-vectors cs_i64_regular_2x4 / cs_i64_fast_2x4 hold the same matrix and differ only by mode, so the pair also pins that mode reaches the bytes. Their values sweep the msgpack integer widths in both directions — negative fixint / int8 / int16 / int32 alongside positive fixint / uint8 / uint32 — since Count Sketch cells are signed:

9198 00 7f cc80 ce00010000 ff d0df d18000 d280000000
     0  127  u8     u32    -1  i8    i16      i32

Full suite green (491 + 9 golden); clippy --all-targets clean; fmt --check clean; cargo doc clean under CI's -D warnings.

Not in this PR

  • sketchlib-go must mirror 0x04 0x00 and copy the two cs_*.hex fixtures. Deliberately out of scope here (same standing item as KLL's 0x06 0x00/0x06 0x01 from feat(kll): ASAPv1 wire payload for KLL and KLLDynamic #75).
  • 0x0a 0x00 Count-Sketch-with-heap (CSHeap) is a separate kind_id and untouched.

🤖 Generated with Claude Code

Self-describing ASAPv1 serialization for Count Sketch, kind_id `0x04 0x00`,
following the Count-Min pattern from #65. `serialize_to_bytes` /
`deserialize_from_bytes` now emit/read the ASAPv1 envelope (replacing the
old plain-rmp form) and live in a per-sketch `wire` submodule.

Metadata mirrors Count-Min's — the hash-spec group derived from the
hasher's `HashProfile`, then `rows` / `cols` / `mode` — with **one key
removed: no `counter_type`** (Q-CS). Count Sketch counters must be signed
and negatable (`CountSketchCounter` requires `Neg` + `From<i32>`), so
`i64` is the only wire-eligible type and the kind_id already implies it;
a key with a single legal value describes nothing. Payload is the same
1-element positional array `[counts]`, packed row-major, signed cells.

The wire methods are implemented only for the wire-eligible config
(`Vector2D<i64>` storage, fast/regular mode, a profiled hasher), so the
old generic-over-all-storage rmp helpers are gone — exotic counters
(i32/i128) and non-`Vector2D` storage must be converted first, matching
what CMS did. `MessagePackCodec` is narrowed to match.

Fail-closed decode: kind_id check, metadata equality check (hash spec +
mode), non-zero dimensions (crafted `cols == 0` would panic in
`Vector2D::from_fn` via `ilog2`), and a counts-length check that runs
before the allocation so crafted dimensions cannot drive a huge reserve.

Golden byte-vectors `cs_i64_regular_2x4` / `cs_i64_fast_2x4` hold the
same matrix and differ only by `mode`, so the pair also pins that mode
reaches the bytes. Their values sweep the msgpack integer widths in both
directions (negative fixint / int8 / int16 / int32 alongside the positive
widths), since Count Sketch cells are signed.

Docs: new §3.4, registry row flipped to implemented, Q-CS recorded.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant