Skip to content

Avner/ssd correctness - #11

Open
avnermay wants to merge 16 commits into
avner/sglang-fa4-phnxfrom
avner/ssd-correctness
Open

Avner/ssd correctness#11
avnermay wants to merge 16 commits into
avner/sglang-fa4-phnxfrom
avner/ssd-correctness

Conversation

@avnermay

@avnermay avnermay commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

No description provided.

avnermay and others added 16 commits July 4, 2026 22:41
If launch_tgl_server itself raised, tgl_server was unbound and the finally
block died with UnboundLocalError, hiding the primary error from the pytest
report. Initialize to None and guard the kill path.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… verify flags; fix temp>0 landmines and uninitialized activations

New CPU-only unit tests (production code under test, no re-implementations):
- test_fork_token_selection: spec-token exclusion at depths 0..K-1 (not at the
  bonus position), miss rows' all-fanout-at-k0 layout, depth-major ordering
  consistency with the cache-key k layout, row equivariance.
- test_hit_cache_unbound: populate/lookup key structure, first-match-wins,
  miss-row single-source consistency, glue-input == returned-tokens contract,
  empty-cache fast path determinism (B6 NaN canary).
- test_build_tree_batch_layout: uniform fast path == varlen fallback per seq
  (heterogeneous extend_counts, mixed hit/miss, eagle+phoenix), fused
  token/hidden placement, positions/slots/cu_seqlens/context_lens, tree-seed
  fan repeat counts, phoenix recovery expansion to (B*MQ_LEN, dim).
- test_verify_flags_matrix: verify() contract across temperature x
  communicate_* flags, row-wise mixed-batch semantics, jit ratio behavior.
- test_dump_roundtrip_b_gt1: batch-aware trace_reader vs the real dump
  writers (late joins, heterogeneous accepts, flat speculations, sentinels).

Fixes:
- verify(): raise explicit ValueError instead of (a) silently degrading to
  greedy acceptance at temp>0 when cache_hits is missing (losslessness was
  silently broken) and (b) TypeError-crashing when logits_q is missing on
  ratio rows.
- hit_cache: zero-fill out_activations (was torch.empty) so the empty-cache
  fast path can't leak uninitialized memory into glue-decode conditioning.
- tests/hf/trace_reader.py: new batch-aware dump reader (pairing, [B,K]
  reshape, per-seq splitting keyed by cache_keys[:,0]/hash_rid, prefill
  matching, prefix chaining, accept-length reconstruction incl. the
  engine-comparable final-round tail).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ftRunner replay harness

Investigation results (eagle-fast/jit rank failures at lookahead=4):
- Replaying the dumped request stream through a fresh DraftRunner reproduces
  every dumped response bit-exactly, in BOTH cudagraph and eager modes
  (tests/draft_runner/_harness.py) — the draft side is deterministic and
  graph==eager.
- Dumped extend/prefill activations match HF-computed target activations to
  ~1-3% relative norm at every round — the target side ships correct
  conditioning.
- A faithful multi-round-chain HF mirror (tests/hf/mirror_replay.py) explains
  the engine's 'bad' rounds (e.g. its worst round ranks [1,1,4,8] under true
  chain conditioning) while diverging at other rounds — the recurrence chain
  is mildly chaotic, so ANY reconstruction disagrees somewhere.

Test changes accordingly:
- accept-length: compute completion_tokens / num_rounds (structurally
  identical to the engine's spec_accept_length) instead of averaging
  between-round prefix diffs, which drops the final round and ran ~0.05 high;
  cross-check the dump chain against the emitted completion (final tail).
- ranks: strict per-token threshold only for chain-free rounds (force-jit,
  jit-reconstructed misses, phoenix); statistical bound (>=90% tokens, >=80%
  rounds within rank 4) for eagle cache-hit rounds.
- full_ssd_simulation now gets the engine's actual fanout (was silently
  simulating fan_out=5 against a fanout-3 engine).
- HF-reference devices env-overridable (SSD_TEST_TARGET_DEVICE/_DRAFT_DEVICE).
- fix teardown masking launch errors; note SpeculationResponse.receive
  classmethod is shadowed by the instance method (use prepare()+receive()).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…Runner suite

- tests/hf/test_ssd_vs_hf_reference_batch.py: N=4 concurrent greedy requests
  with client rids against one server (--max-running-requests 4,
  --disable-radix-cache); per-sequence HF completion gaps, speculation
  reconstruction (chain-aware thresholds), exact accept-length vs per-request
  meta_info, and a batching-honesty gate (>=60% of rounds at full batch).
  SSD_TRACE_REUSE=<dir> skips the server phase to iterate on reconstruction.
- helpers.launch_tgl_server: max_running_requests + extra_args knobs.
- scripted DraftRunner suite: real-text prompts + real 8B conditioning acts
  (random-token/random-act inputs leave fork margins at coin-flip levels),
  margin-aware hit scripting (boundary ties downgrade to scripted misses so
  hit/miss stays exactly assertable), stale-cache probe on batch regrow.
- mirror_replay: expose per-k fork margins.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ed hits

- tests/e2e/_helpers.py: share model snapshot constants with tests/hf/helpers
  (the stale /scratch paths made the whole e2e suite silently skip).
- scripted suite: hits only at the chain-free k=0 fork position (k>=1 fork
  sets depend on the engine's private recurrence chain and are not
  predictable externally); k>=1 exercised via scripted misses; added a
  B1-altprompts param that runs the B2 prompt set at B=1 to separate batch
  effects from content-margin effects.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…lock tables

run_glue_decode_cudagraph pads ghost rows (batch size below the graph bucket)
by replicating the last real sequence's block-table row — with a full-width
assignment into the static buffer. TGL sizes wire block tables dynamically
(seq_lens.max()+tree blocks, typically ~3 wide vs the 32-wide static buffer),
so the first round that needed bucket padding (e.g. B=3 -> bucket 4) crashed
the draft process with a broadcast error. Impossible to hit at max_num_seqs=1,
which is why every single-request run passed. Slice the width like the
real-row copy above (the other replay paths already do).

Also fix the scripted-driver suite: send a real prefill per sequence group
(the glue decode otherwise attends unwritten prompt KV) and use
max_model_len/block_size-wide block tables to match the graph buffers.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…le cache rows

Miss rows used to be filled from tree-cache row 0 ('any consistent tokens are
fine'). Consistent, yes — but it leaked state: responses depended on session
history (whatever happened to be in row 0), the next round's fork sets
inherited the leak through the trunk-token exclusion in
get_forked_recovery_tokens_from_logits, and at B>1 row 0 belongs to a
DIFFERENT sequence, leaking one request's branch content to another. Runs were
irreproducible (caught by the scripted suite's rerun-determinism and
row-permutation-equivariance tests). Zeros (token 0 + sentinel logits + zero
activations, exactly like the empty-cache round) are equally consistent,
deterministic, and match the documented fast-backup semantics.

Batch test upgrades:
- assert shipped extend/recovery activations match HF target activations
  per-slot (rel-diff < 0.15; misalignment bugs score ~1.0, bf16 noise ~0.03) —
  the chaos-immune conditioning check; validated the activation-slice fix
  end-to-end at B=4.
- B>1-calibrated chain-round bounds (composition changes feed extra bf16
  perturbations into the recurrence chain; the strict guarantees at B>1 come
  from completion/accept-length/activation exactness + the scripted suite's
  bit-exact tests).
- NOTE: length-only prompt matching is ambiguous (Kyoto and Nairobi chat
  prompts both tokenize to 47 tokens) — a wrong pairing masquerades as
  smoothly-growing activation corruption. Always pair by exact tokens/rid.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… only when absent

Pure zeros on cache misses (previous commit) killed the deliberate fallback
heuristic hiding inside the old stale-row behavior: the sequence's previous
k=0 top-fork branch is often still a good speculation after a near-miss, and
dropping it cost ~0.3-0.4 average accept length in fast mode (eagle 2.03 ->
1.73, standalone 3.20 -> 2.78 on the reference run). Restore the heuristic
but index the SAME sequence's first cache entry instead of global row 0:

- B=1 behavior is bit-identical to the original engine (row 0 == own row),
- B>1 no longer leaks one request's branch content into another's response,
- fresh sequences (no cache entries) get deterministic zeros, so scripted
  rerun-determinism/equivariance hold without cross-session leakage.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…cture, status)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… port per server

The loosened B>1 chain bounds were calibrated against runs where the greedy
temperature bug made the draft SAMPLE its speculations; with that fixed, B=4
chain fractions sit at 0.94-1.0 token / 0.77-0.95 round — restore near-B=1
bounds (0.9/0.7). Chain-free rounds at B>1 get calibrated slack for isolated
batched-kernel near-tie flips (worst<=64 AND >=97% tokens <=4; systematic
conditioning bugs blow both). Also derive --speculative-async-port from the
HTTP port: the fixed 29600 default collides with other async-spec servers on
a shared node (observed EADDRINUSE against another user's job).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…orrected

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…h, zeros only when absent

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…/failing on shared nodes

The default torch.distributed group for the multi-GPU target / same-node
async draft always bound tcp://localhost:1223. Any foreign process holding
that port breaks engine startup — a torch-store-speaking squatter (e.g.
another user's job) absorbs the rendezvous and the engine waits forever;
this is what made every async e2e subprocess time out on a shared node while
the same config runs fine on a quiet one. LLMEngine now picks a free port
before spawning workers/draft (children inherit it via the pickled config;
1223 remains only the fallback for direct ModelRunner construction), and the
engine passes with port 1223 deliberately squatted.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…-redirected output, process-group reaping

Every async e2e test 'timed out' at 600s even though generation succeeded
(RUNNER_RESULT printed): the runner never called llm.exit(), so its exit
depended on multiprocessing's atexit joining the non-daemon draft child,
whose own exit depends on NCCL teardown ordering; and run_llm_subprocess
captured output via PIPES, so subprocess.run only returned on pipe EOF —
which lingering grandchildren (draft/worker processes) hold open. Fixes:

- _runner.py: explicit llm.exit(hard=False) then os._exit(0) after printing
  the result (no reliance on interpreter-shutdown ordering).
- run_llm_subprocess: stdout/stderr to temp files (wait() returns when the
  RUNNER exits, not when every descendant closes a pipe), stdin devnull,
  start_new_session + killpg on timeout AND after completion — no more
  orphaned engine families squatting on GPUs after failed runs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…pe EOF) resolved

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…campaign evidence

e2e suite on the fixed engine/runner: 3 passed, 1 xfailed in 5:56 (was 3
timeout-failures in 43:24). The seq-1 divergence reproduces exactly as
recorded; its xfail reason now states the established facts (benign per-row
bf16 numerics between process topologies; final tokens identical; draft
bit-deterministic, row-permutation-equivariant, HF-correct conditioning)
instead of the old KV-rollback hypothesis. Strict, so any change in either
direction flags.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@broly-code-security-scanner

Copy link
Copy Markdown

Broly Security Scan

Note

Baseline snapshot is missing for this repo. Broly is running in PR-only fallback mode until the first scheduled baseline completes. This does not block the PR.

Note

Summary

1 actionable finding(s) in this PR
2 total in scan · 1 dismissed false positive

  • 🟠 1 high

All actionable items are in the table below.

Severity Scanner Issue Location Dismiss Verdict
🟠 HIGH SAST Unsafe deserialization of untrusted .pt files via
torch.load(..., weights_only=False).
tests/hf/trace_reader.py:120 d1 🔺 TRUE_POSITIVE · Confidence: HIGH

Fix Suggestions

Unsafe deserialization of untrusted .pt files via `torch.load(..., weights_only=False)`. — tests/hf/trace_reader.py:120

Set weights_only=True in the torch.load call to restrict deserialization to tensor data only, preventing arbitrary code execution from untrusted .pt files.

  • Changes torch.load(f, weights_only=False) to torch.load(f, weights_only=True) on line 120 of load_prefills.
  • weights_only=True restricts deserialization to tensor and primitive types, blocking arbitrary pickle code execution from malicious .pt files.
  • Limitation: load_trace and any other call sites in trace_reader.py that use torch.load(..., weights_only=False) remain vulnerable and must be fixed separately.
--- a/tests/hf/trace_reader.py
+++ b/tests/hf/trace_reader.py
@@ -117,7 +117,12 @@
     """Each prefill dump split into per-seq records: prompt token list + eagle acts slice."""
     out = []
     for f in sorted(Path(trace_dir).glob("prefill_request_*.pt")):
-        d = torch.load(f, weights_only=False)
+          d = torch.load(f, weights_only=True)
+FIX_RANGE: 120
+FIX_NOTES:
+- Changes `torch.load(f, weights_only=False)` to `torch.load(f, weights_only=True)` on line 120 of `load_prefills`.
+- `weights_only=True` restricts deserialization to tensor and primitive types, blocking arbitrary pickle code execution from malicious `.pt` files.
+- Limitation: `load_trace` and any other call sites in `trace_reader.py` that use `torch.load(..., weights_only=False)` remain vulnerable and must be fixed separately.
         num_tokens = d["num_tokens"].tolist()
         ids = d["input_ids"].tolist()
         acts = d.get("eagle_acts")

🟢 Dismissed false positives (1) — not shown above
  • 🟠 HIGH · Weak hash (MD5) · tests/hf/trace_reader.py:35 — hash_ridis a test utility that replicates the productionhash_to_int64` function for mapping dump rows to requests by request ID, not a security control — no secrets, passwords, or authentication material are hashed.

Dismiss false positives

Tick a box to dismiss the finding; untick it to bring the finding back. That is the same as replying /broly dismiss d1 and /broly undismiss d1. To record why it is a false positive, reply with /broly dismiss d1: your reason instead — Broly reuses those reasons to triage similar findings across the org.

  • d1 · 🟠 HIGH     · tests/hf/trace_reader.py:120 · Unsafe deserialization of untrusted .pt files via `torch.load(..., weights_on...

Note

Re-scan this PR anytime with /broly scan — useful after /broly undismiss, or to refresh findings without a new push.

Broly — SAST (zai-org/GLM-5.2) · Secrets · SCA · IaC · GH Actions · Base Images · Supply Chain Threats · Exploit Chains · Adversarial Verification

We're continuously improving Broly's accuracy and finding quality — your feedback is valuable. False positives, missed findings, bugs, and feature requests all welcome.

Ask in #security-engineering   Powered by Together AI

Comment thread tests/hf/trace_reader.py

def hash_rid(rid: str) -> int:
"""Replicates tgl python/sglang/private/speculative/spec_worker.py:hash_to_int64."""
return int.from_bytes(hashlib.md5(rid.encode()).digest()[:8], "little", signed=True)
Comment thread tests/hf/trace_reader.py
"""Each prefill dump split into per-seq records: prompt token list + eagle acts slice."""
out = []
for f in sorted(Path(trace_dir).glob("prefill_request_*.pt")):
d = torch.load(f, weights_only=False)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants