Add exact hash provider

[krakhit]

Motivation

Follow-on to sgl-project#31057 ("semantic KV cache reuse via a pluggable fuzzy-match
radix backend"). SemanticEmbeddingProvider (the existing provider) finds
donor KV by semantic similarity — matches merely-similar content and is
explicitly not lossless. This PR adds a second FuzzyMatchProvider
implementation, ExactHashProvider, for the narrower but common case where
content is byte-identical to the current prompt's unmatched tail but
sits at a different offset than where it was originally computed — e.g.
repeated tool schemas, retrieved documents, or system-prompt boilerplate
reappearing later in a different position. It has no external dependency
(no embedding model, unlike SemanticEmbedding's semblend requirement)
and is lossless: a token-ID equality check on every hash hit is mandatory,
so a fingerprint collision never gets served as a false match.

The content-defined chunking parameters (Gear-hash rolling window, boundary
bits, min/max chunk clamp, sink-token carve-out) follow the methodology
described in Irminsul: MLA-Native Position-Independent Caching for Agentic
LLM Serving (Ma, Eitzinger, Köstler; arXiv:2605.05696).
That paper's own mechanism (RoPE delta-rotation over MLA's decomposed
k_nope/k_rope) is specific to MLA architectures and is not used
here; ExactHashProvider reuses only its public, architecture-agnostic
CDC/chunking layer, and works against any KV pool realizer.py supports
(currently MHA-style pools), independent of MLA.

Modifications

• chunker.py (new): content-defined chunking over token-ID sequences.
  Gear-hash rolling boundaries (64-token window, 7 boundary bits, chunks
  clamped to [32, 512] tokens, xxHash64 fingerprints with a blake2b
  fallback), so identical content chunks identically regardless of what
  precedes it — the property fixed-boundary chunking structurally lacks.
  The first SINK_TOKENS=32 positions are never used as a chunk-
  registration start, since attention-sink positions absorb a
  disproportionate, content-independent share of attention and aren't a
  trustworthy content signal.
• exact_hash_provider.py (new): ExactHashProvider(FuzzyMatchProvider).
  Registers donor chunks keyed by (extra_key, fingerprint), and on a
  prefix miss looks up the first chunk of the unmatched tail. Every
  candidate is confirmed by token-ID equality before being returned as a
  match — the fingerprint alone is never trusted. Current limitation:
  single-chunk, non-segmented matches only (FuzzyMatchResult.segments=None);
  multi-chunk / N:M segment matching is a natural follow-up, not required
  for the mechanism to be correct.
• fuzzy_match_provider.py, config.py, server_args.py: wire
  "ExactHash" in as a selectable --fuzzy-match-provider value alongside
  "SemanticEmbedding".
• realizer.py: generalized _resolve_rotary_emb to scan every decoder
  layer (not just layer 0's self_attn) for a rotary_emb, since hybrid
  architectures mix layer types across model.model.layers and don't
  uniformly expose it in the same place. Also unwraps HybridLinearKVPool
  (full-attention + linear-attention hybrids) to its full_kv_pool before
  running correction, since the linear-attention layers hold irreversible
  recurrent state, not addressable per-token K.
• fuzzy_radix_cache.py: rejects hybrid-SSM models explicitly at
  backend construction (is_hybrid_ssm guard) rather than failing
  confusingly later.
• environ.py: adds SGLANG_TEST_FUZZY_FORCE_HASH_COLLISION, a
  test-only hook that forces every chunk fingerprint to collide, used to
  test the equality-check fallback without waiting on an astronomically
  unlikely real collision.
• README.md (mem_cache/fuzzy_match/): documents ExactHash as a
  second provider option, its no-external-dependency launch command, and
  its lossless-by-construction guarantee vs. SemanticEmbedding's lossy one.
• New tests: test/registered/unit/mem_cache/fuzzy_match/test_chunker.py,
  test_exact_hash_provider.py,
  test/registered/fuzzy_match/test_exact_hash_shifted_offset_kl.py,
  test_exact_hash_e2e_safety.py.

Accuracy Tests

Run on a single H100, model Qwen/Qwen2.5-7B-Instruct-AWQ (the same
checkpoint sgl-project#31057 was itself tested against) for the E2E tests.

Suite	Result
test_chunker.py (4 cases)	✅ 4 passed
test_exact_hash_provider.py (4 cases)	✅ 4 passed
test_fuzzy_match_providers.py + test_fuzzy_radix_cache.py — sgl-project#31057's own existing unit tests, run to confirm no regression from the realizer.py/fuzzy_radix_cache.py changes (26 cases)	✅ 26 passed
test_exact_hash_shifted_offset_kl.py — E2E, shifted-offset reuse vs. full recompute	✅ 1 passed — avg KL divergence 0.00414 (threshold 0.02) across 8 samples
test_exact_hash_e2e_safety.py — E2E, forced hash-collision disambiguation + cross-tenant isolation	✅ 2 passed

37/37 test cases pass, including a full regression run of PR sgl-project#31057's
own pre-existing test suite.

Reproduce:

# from the repo root, on this branch
export PYTHONPATH="$(pwd)/python"

# Unit tests (no GPU needed)
python3 -m pytest \
  test/registered/unit/mem_cache/fuzzy_match/test_chunker.py \
  test/registered/unit/mem_cache/fuzzy_match/test_exact_hash_provider.py \
  -v

# PR #31057's own existing tests (regression check, no GPU needed)
python3 -m pytest \
  test/registered/unit/mem_cache/test_fuzzy_match_providers.py \
  test/registered/unit/mem_cache/test_fuzzy_radix_cache.py \
  -v

# E2E tests (needs 1 GPU; launches a real sglang server via the `sglang`
# CLI entry point, so its install location must be on $PATH)
python3 -m pytest test/registered/fuzzy_match/test_exact_hash_shifted_offset_kl.py -v
python3 -m pytest test/registered/fuzzy_match/test_exact_hash_e2e_safety.py -v

Benchmarking and Profiling

Ran a single-H100 TTFT comparison, Qwen/Qwen2.5-7B-Instruct-AWQ: register
a 4000-token donor behind a SINK_TOKENS-length prefix, then query it
behind a different, longer, exact-matched prefix (5 trials each, 1 warm-up
trial discarded), timing max_new_tokens=1 as a TTFT proxy.

	mean latency
plain radix cache (baseline)	0.1912s
fuzzy_match + ExactHash	0.1943s

No measurable speedup at this scale (0.98x — within noise). The
server's own instrumentation explains why: ExactHashProvider only matches
the first CDC chunk of the unmatched tail (see Modifications above —
FuzzyMatchResult.segments=None, no multi-chunk matching yet). A chunk is
capped at MAX_CHUNK_TOKENS=512, averaging ~128 tokens. So regardless of
how large the donor content is, only one chunk's worth gets reused per
query — in this run, 65–259 tokens out of every ~4300-token request
(2–6%), confirmed directly from the prefill batch log
(#cached-token/#fuzzy-token counts):

fuzzy=168 total=4299   fuzzy=259 total=4299   fuzzy=65 total=4299
fuzzy=213 total=4299   fuzzy=66  total=4299   fuzzy=77 total=4299

That small a fraction of a request's prefill work doesn't clear the fixed
per-request overhead (tokenization, scheduling, HTTP round-trip) on an
H100, where raw prefill compute for a few hundred tokens is sub-
millisecond. This isn't a correctness issue — the KL-divergence results
above confirm the reused span is numerically valid — it's a real, current
ceiling on this PR's latency upside: today it's capped at ~512 tokens of
avoided prefill per request no matter the donor size. Multi-chunk / N:M
segment matching (the follow-up already flagged in
exact_hash_provider.py's docstring) is what would need to land before
this mechanism shows up in end-to-end throughput/TTFT numbers.

Checklist

• Format your code according to Format code with pre-commit — black, isort (black profile), ruff --select=F401,F821,UP037, and codespell all pass on every changed file (full pre-commit wasn't installed in this environment; ran the underlying tools directly, pinned to the versions in .pre-commit-config.yaml).
• Add unit tests according to Run and add unit tests — see Accuracy Tests above.
• Update documentation according to Write documentations — mem_cache/fuzzy_match/README.md updated.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed — see Accuracy Tests and Benchmarking and Profiling above. Speed result is an honest no-measurable-speedup-yet finding, with root cause diagnosed (current single-chunk-only matching limitation), not a fabricated win.
• Follow the SGLang code style guidance.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 67c2e5c0-c080-4160-9b2b-242268f2f51b

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}