Avner/ssd correctness - #11
Conversation
If launch_tgl_server itself raised, tgl_server was unbound and the finally block died with UnboundLocalError, hiding the primary error from the pytest report. Initialize to None and guard the kill path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… verify flags; fix temp>0 landmines and uninitialized activations New CPU-only unit tests (production code under test, no re-implementations): - test_fork_token_selection: spec-token exclusion at depths 0..K-1 (not at the bonus position), miss rows' all-fanout-at-k0 layout, depth-major ordering consistency with the cache-key k layout, row equivariance. - test_hit_cache_unbound: populate/lookup key structure, first-match-wins, miss-row single-source consistency, glue-input == returned-tokens contract, empty-cache fast path determinism (B6 NaN canary). - test_build_tree_batch_layout: uniform fast path == varlen fallback per seq (heterogeneous extend_counts, mixed hit/miss, eagle+phoenix), fused token/hidden placement, positions/slots/cu_seqlens/context_lens, tree-seed fan repeat counts, phoenix recovery expansion to (B*MQ_LEN, dim). - test_verify_flags_matrix: verify() contract across temperature x communicate_* flags, row-wise mixed-batch semantics, jit ratio behavior. - test_dump_roundtrip_b_gt1: batch-aware trace_reader vs the real dump writers (late joins, heterogeneous accepts, flat speculations, sentinels). Fixes: - verify(): raise explicit ValueError instead of (a) silently degrading to greedy acceptance at temp>0 when cache_hits is missing (losslessness was silently broken) and (b) TypeError-crashing when logits_q is missing on ratio rows. - hit_cache: zero-fill out_activations (was torch.empty) so the empty-cache fast path can't leak uninitialized memory into glue-decode conditioning. - tests/hf/trace_reader.py: new batch-aware dump reader (pairing, [B,K] reshape, per-seq splitting keyed by cache_keys[:,0]/hash_rid, prefill matching, prefix chaining, accept-length reconstruction incl. the engine-comparable final-round tail). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ftRunner replay harness Investigation results (eagle-fast/jit rank failures at lookahead=4): - Replaying the dumped request stream through a fresh DraftRunner reproduces every dumped response bit-exactly, in BOTH cudagraph and eager modes (tests/draft_runner/_harness.py) — the draft side is deterministic and graph==eager. - Dumped extend/prefill activations match HF-computed target activations to ~1-3% relative norm at every round — the target side ships correct conditioning. - A faithful multi-round-chain HF mirror (tests/hf/mirror_replay.py) explains the engine's 'bad' rounds (e.g. its worst round ranks [1,1,4,8] under true chain conditioning) while diverging at other rounds — the recurrence chain is mildly chaotic, so ANY reconstruction disagrees somewhere. Test changes accordingly: - accept-length: compute completion_tokens / num_rounds (structurally identical to the engine's spec_accept_length) instead of averaging between-round prefix diffs, which drops the final round and ran ~0.05 high; cross-check the dump chain against the emitted completion (final tail). - ranks: strict per-token threshold only for chain-free rounds (force-jit, jit-reconstructed misses, phoenix); statistical bound (>=90% tokens, >=80% rounds within rank 4) for eagle cache-hit rounds. - full_ssd_simulation now gets the engine's actual fanout (was silently simulating fan_out=5 against a fanout-3 engine). - HF-reference devices env-overridable (SSD_TEST_TARGET_DEVICE/_DRAFT_DEVICE). - fix teardown masking launch errors; note SpeculationResponse.receive classmethod is shadowed by the instance method (use prepare()+receive()). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…Runner suite - tests/hf/test_ssd_vs_hf_reference_batch.py: N=4 concurrent greedy requests with client rids against one server (--max-running-requests 4, --disable-radix-cache); per-sequence HF completion gaps, speculation reconstruction (chain-aware thresholds), exact accept-length vs per-request meta_info, and a batching-honesty gate (>=60% of rounds at full batch). SSD_TRACE_REUSE=<dir> skips the server phase to iterate on reconstruction. - helpers.launch_tgl_server: max_running_requests + extra_args knobs. - scripted DraftRunner suite: real-text prompts + real 8B conditioning acts (random-token/random-act inputs leave fork margins at coin-flip levels), margin-aware hit scripting (boundary ties downgrade to scripted misses so hit/miss stays exactly assertable), stale-cache probe on batch regrow. - mirror_replay: expose per-k fork margins. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ed hits - tests/e2e/_helpers.py: share model snapshot constants with tests/hf/helpers (the stale /scratch paths made the whole e2e suite silently skip). - scripted suite: hits only at the chain-free k=0 fork position (k>=1 fork sets depend on the engine's private recurrence chain and are not predictable externally); k>=1 exercised via scripted misses; added a B1-altprompts param that runs the B2 prompt set at B=1 to separate batch effects from content-margin effects. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…lock tables run_glue_decode_cudagraph pads ghost rows (batch size below the graph bucket) by replicating the last real sequence's block-table row — with a full-width assignment into the static buffer. TGL sizes wire block tables dynamically (seq_lens.max()+tree blocks, typically ~3 wide vs the 32-wide static buffer), so the first round that needed bucket padding (e.g. B=3 -> bucket 4) crashed the draft process with a broadcast error. Impossible to hit at max_num_seqs=1, which is why every single-request run passed. Slice the width like the real-row copy above (the other replay paths already do). Also fix the scripted-driver suite: send a real prefill per sequence group (the glue decode otherwise attends unwritten prompt KV) and use max_model_len/block_size-wide block tables to match the graph buffers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…le cache rows
Miss rows used to be filled from tree-cache row 0 ('any consistent tokens are
fine'). Consistent, yes — but it leaked state: responses depended on session
history (whatever happened to be in row 0), the next round's fork sets
inherited the leak through the trunk-token exclusion in
get_forked_recovery_tokens_from_logits, and at B>1 row 0 belongs to a
DIFFERENT sequence, leaking one request's branch content to another. Runs were
irreproducible (caught by the scripted suite's rerun-determinism and
row-permutation-equivariance tests). Zeros (token 0 + sentinel logits + zero
activations, exactly like the empty-cache round) are equally consistent,
deterministic, and match the documented fast-backup semantics.
Batch test upgrades:
- assert shipped extend/recovery activations match HF target activations
per-slot (rel-diff < 0.15; misalignment bugs score ~1.0, bf16 noise ~0.03) —
the chaos-immune conditioning check; validated the activation-slice fix
end-to-end at B=4.
- B>1-calibrated chain-round bounds (composition changes feed extra bf16
perturbations into the recurrence chain; the strict guarantees at B>1 come
from completion/accept-length/activation exactness + the scripted suite's
bit-exact tests).
- NOTE: length-only prompt matching is ambiguous (Kyoto and Nairobi chat
prompts both tokenize to 47 tokens) — a wrong pairing masquerades as
smoothly-growing activation corruption. Always pair by exact tokens/rid.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… only when absent Pure zeros on cache misses (previous commit) killed the deliberate fallback heuristic hiding inside the old stale-row behavior: the sequence's previous k=0 top-fork branch is often still a good speculation after a near-miss, and dropping it cost ~0.3-0.4 average accept length in fast mode (eagle 2.03 -> 1.73, standalone 3.20 -> 2.78 on the reference run). Restore the heuristic but index the SAME sequence's first cache entry instead of global row 0: - B=1 behavior is bit-identical to the original engine (row 0 == own row), - B>1 no longer leaks one request's branch content into another's response, - fresh sequences (no cache entries) get deterministic zeros, so scripted rerun-determinism/equivariance hold without cross-session leakage. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…cture, status) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… port per server The loosened B>1 chain bounds were calibrated against runs where the greedy temperature bug made the draft SAMPLE its speculations; with that fixed, B=4 chain fractions sit at 0.94-1.0 token / 0.77-0.95 round — restore near-B=1 bounds (0.9/0.7). Chain-free rounds at B>1 get calibrated slack for isolated batched-kernel near-tie flips (worst<=64 AND >=97% tokens <=4; systematic conditioning bugs blow both). Also derive --speculative-async-port from the HTTP port: the fixed 29600 default collides with other async-spec servers on a shared node (observed EADDRINUSE against another user's job). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…orrected Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…h, zeros only when absent Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…/failing on shared nodes The default torch.distributed group for the multi-GPU target / same-node async draft always bound tcp://localhost:1223. Any foreign process holding that port breaks engine startup — a torch-store-speaking squatter (e.g. another user's job) absorbs the rendezvous and the engine waits forever; this is what made every async e2e subprocess time out on a shared node while the same config runs fine on a quiet one. LLMEngine now picks a free port before spawning workers/draft (children inherit it via the pickled config; 1223 remains only the fallback for direct ModelRunner construction), and the engine passes with port 1223 deliberately squatted. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…-redirected output, process-group reaping Every async e2e test 'timed out' at 600s even though generation succeeded (RUNNER_RESULT printed): the runner never called llm.exit(), so its exit depended on multiprocessing's atexit joining the non-daemon draft child, whose own exit depends on NCCL teardown ordering; and run_llm_subprocess captured output via PIPES, so subprocess.run only returned on pipe EOF — which lingering grandchildren (draft/worker processes) hold open. Fixes: - _runner.py: explicit llm.exit(hard=False) then os._exit(0) after printing the result (no reliance on interpreter-shutdown ordering). - run_llm_subprocess: stdout/stderr to temp files (wait() returns when the RUNNER exits, not when every descendant closes a pipe), stdin devnull, start_new_session + killpg on timeout AND after completion — no more orphaned engine families squatting on GPUs after failed runs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…pe EOF) resolved Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…campaign evidence e2e suite on the fixed engine/runner: 3 passed, 1 xfailed in 5:56 (was 3 timeout-failures in 43:24). The seq-1 divergence reproduces exactly as recorded; its xfail reason now states the established facts (benign per-row bf16 numerics between process topologies; final tokens identical; draft bit-deterministic, row-permutation-equivariant, HF-correct conditioning) instead of the old KV-rollback hypothesis. Strict, so any change in either direction flags. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Broly Security ScanNote Baseline snapshot is missing for this repo. Broly is running in PR-only fallback mode until the first scheduled baseline completes. This does not block the PR. Note Summary 1 actionable finding(s) in this PR
All actionable items are in the table below.
Fix SuggestionsUnsafe deserialization of untrusted .pt files via `torch.load(..., weights_only=False)`. — tests/hf/trace_reader.py:120Set
--- a/tests/hf/trace_reader.py
+++ b/tests/hf/trace_reader.py
@@ -117,7 +117,12 @@
"""Each prefill dump split into per-seq records: prompt token list + eagle acts slice."""
out = []
for f in sorted(Path(trace_dir).glob("prefill_request_*.pt")):
- d = torch.load(f, weights_only=False)
+ d = torch.load(f, weights_only=True)
+FIX_RANGE: 120
+FIX_NOTES:
+- Changes `torch.load(f, weights_only=False)` to `torch.load(f, weights_only=True)` on line 120 of `load_prefills`.
+- `weights_only=True` restricts deserialization to tensor and primitive types, blocking arbitrary pickle code execution from malicious `.pt` files.
+- Limitation: `load_trace` and any other call sites in `trace_reader.py` that use `torch.load(..., weights_only=False)` remain vulnerable and must be fixed separately.
num_tokens = d["num_tokens"].tolist()
ids = d["input_ids"].tolist()
acts = d.get("eagle_acts")🟢 Dismissed false positives (1) — not shown above
Dismiss false positivesTick a box to dismiss the finding; untick it to bring the finding back. That is the same as replying
Note Re-scan this PR anytime with
|
|
|
||
| def hash_rid(rid: str) -> int: | ||
| """Replicates tgl python/sglang/private/speculative/spec_worker.py:hash_to_int64.""" | ||
| return int.from_bytes(hashlib.md5(rid.encode()).digest()[:8], "little", signed=True) |
| """Each prefill dump split into per-seq records: prompt token list + eagle acts slice.""" | ||
| out = [] | ||
| for f in sorted(Path(trace_dir).glob("prefill_request_*.pt")): | ||
| d = torch.load(f, weights_only=False) |
No description provided.