[3/3] perf(dsv4): add hardware-aware adaptive verification

[calvarado2004]

Outcome

This PR turns the exact DSpark path from PR 2 into a hardware-aware implementation: it profiles the real TP target-verification cost curve, chooses the prefix width that maximizes expected emitted tokens per millisecond, safely replays the matching CUDA graph, and performs probabilistic acceptance on GPU.

On structured code generation, the complete stack improves steady single-request generation from approximately 18 tok/s target-only to 27.7–29.7 tok/s at temperature 1.0 (1.54–1.65×). Open-ended Spanish reasoning/chat reaches 17.7–18.6 tok/s at lower acceptance; that workload-sensitive result is reported deliberately rather than hidden.

Stack and review boundary

This is PR 3 of 3 and depends on PR 1: the TP foundation followed by PR 2: exact DSpark:

Commit	Purpose
13d4560	TP-safe target graphs for every legal verification width and paper-style adaptive scheduling
38d7b2f	GPU-resident probabilistic verification, transfer reduction, and measured cost profiling

Delta from PR 2: 33 files, 2,251 additions, 1,095 deletions. The deletion count includes replacement of the first fixed-width rollback/loop scaffolding with graph-stable per-row state journals and the production scheduler path.

Design references

• DSpark §3.2 and §5.2: confidence-scheduled verification, cost-aware capacity, and stale asynchronous confidence for deployment
• FreeToken: exact bandwidth-adaptive CPU/GPU MoE execution used by every target graph
• vLLM DSV4 reference branch: checkpoint and adaptive-state behavior used as the implementation cross-check

This PR prices the algorithm from the actual FreeToken engine. It does not substitute a confidence threshold or a guessed linear cost model for the paper's hardware-aware capacity decision.

Adaptive objective

For proposal confidence values c₁ … cγ, the probability that proposal position k survives is the cumulative prefix product:

Sₖ = ∏ᵢ₌₁ᵏ cᵢ

Verifying width k is expected to emit the target bonus plus the surviving proposal prefix:

E(k) = 1 + Σᵢ₌₁ᵏ Sᵢ

The engine measures draft cost D and the target CUDA-graph cost V(k) for every legal width. It selects:

k* = argmaxₖ E(k) / (D + V(k)),  k ∈ [0, γ]

Important details:

• Confidence is cumulative, not independently sorted raw confidence. A low early value reduces the usefulness of every later proposal.
• Width zero is legal: the target verifies the anchor and still emits one target token.
• Ties keep the smaller prefix, matching first-argmax behavior.
• The measured verify curve is made monotonic before use so profiling noise cannot make a larger physical graph appear artificially cheaper.
• The drafter has already generated the complete checkpoint block, so D is common to all candidate widths.

Measuring the real engine

• Capture target graphs for every single-request physical span: anchor-only through anchor plus five proposals (1..6 rows).
• Time graph replay with CUDA events and use the median replay cost for each span.
• Reset the MoE cache once before profiling a curve, not before every replay; repeated cold-cache resets would measure an artificial workload.
• Measure draft cost from five real serving steps and use their median. No synthetic input or guessed fixed cost enters the selector.
• Broadcast the rank-0 cost curve and draft cost so every TP rank chooses the same collective graph.

A representative hot target-verification curve on the validation system was:

Draft proposals verified	Physical target rows	Target verify time
0	1	73.7 ms
1	2	96.6 ms
2	3	116.2 ms
3	4	133.8 ms
4	5	150.0 ms
5	6	159.4 ms

The measured draft pass was approximately 23.6 ms. These values explain why a fixed “always verify five” policy is not universally optimal and why the selector must account for hardware cost cliffs.

Stale-confidence scheduling

The implementation uses two pinned host buffers and a dedicated CUDA copy stream:

1. Current-step confidence is copied asynchronously into one buffer.
2. Capacity selection reads the previous ready buffer.
3. Buffers swap after the dependency event is satisfied.
4. A new request UID resets stale confidence to all ones so one request cannot inherit another request's admission policy.

This preserves the causal/non-blocking deployment shape described by DSpark instead of synchronizing the current confidence tensor onto the CPU in the same critical path.

The full five-position draft is still produced before admission. Only target inputs and metadata are compacted to anchor + k; allocation retains the gamma-wide frontier until acceptance, so releasing a rejected or unverified tail remains correct.

TP-safe variable-width CUDA graphs

Variable physical widths exposed a TP4-only correctness bug in the Lightning Indexer path. Captured graphs could read row indices from a stale six-row buffer while replaying a shorter graph, producing an out-of-bounds access and CUDA Xid 13.

This PR:

• gives every captured target width stable, width-specific input and metadata views;
• snapshots Lightning Indexer rows for the selected physical span;
• captures anchor-only as a first-class graph rather than emulating it with padded proposals;
• shares a maximum-size compressor carry journal across all prefix graphs, resetting only the active row count;
• keeps target features, recurrent carry, KV ownership, and output mapping aligned after compaction;
• recomputes reply mappings from the post-acceptance frontier;
• cleans up speculative allocations safely on client disconnect.

Every TP rank selects and replays the same span, preserving collective ordering.

Exact GPU-resident probabilistic verification

The earlier exact implementation copied full-vocabulary target probabilities to CPU for p/q acceptance. This PR keeps the distribution-correct algorithm on GPU:

• gather p(x) and q(x) for proposed tokens;
• generate acceptance uniforms;
• find the first prefix rejection;
• compute normalized max(p - q, 0) for the rejected row;
• select the target bonus distribution after full acceptance;
• draw the recovered or bonus token;
• transfer only two integers together: accepted length and emitted target token.

Probabilistic requests no longer compute or copy an unused target argmax. Greedy requests reduce logits to argmax on GPU before crossing PCIe. Temperature 1.0 skips a mathematically redundant divide-by-one kernel while producing bit-identical probabilities to direct FP32 softmax.

These changes remove transfers and launches; they do not approximate, truncate, or otherwise alter the p/q rejection rule.

Suggested review order

Area	Files	What to verify
Capacity math	deepseek_v4/dspark.py	Cumulative survival, expected-token objective, first-argmax tie behavior
Cost collection	engine/graph.py, engine/engine.py	Per-width graph timing, median draft samples, TP broadcast
Graph correctness	graph runner, DSV4 attention/compressor/indexer files	Stable buffers, legal spans, carry storage, TP row addressing
Batch compaction	engine.py, scheduler.py	Compact target views while preserving full allocation frontier
Exact sampler	deepseek_v4/dspark.py, generation/OpenAI paths	Device p/q, first rejection, residual/bonus draw, two-int transfer
Failure cleanup	scheduler/API and disconnect tests	Post-acceptance mappings and in-flight abort behavior

Validation

The suite covers:

• expensive-profile-cliff and flat-cost selector behavior;
• cumulative survival rather than raw confidence;
• tie behavior and malformed cost curves;
• all legal widths, including width zero;
• stale-confidence request reset and draft warmup;
• shared carry-journal reset and overflow protection;
• compaction of target views without moving the allocation frontier;
• TP graph selection and Lightning Indexer row ownership;
• greedy reduction before PCIe transfer;
• sampled requests without target argmax;
• device-resident full acceptance, rejection recovery, zero width, and one host-result transfer;
• temperature-1 probability equivalence;
• proposal writeback, target feature ownership, and client-disconnect cleanup.

Final result on the validation server:

192 passed in 36.14s

End-to-end measurements

Model:        DeepSeek-V4-Flash-0731
Hardware:     4 × RTX A4000 16 GiB
Interconnect: PCIe-only, split across two NUMA nodes
Serving:      TP=4, one active request, hybrid CPU/GPU routed MoE
Sampling:     temperature 1.0 probabilistic decoding
Baseline:     target-only decode, approximately 18 tok/s

Workload	Proposal acceptance	Steady generation	Relative to ~18 tok/s baseline
Structured code generation	65–70%	27.7–29.7 tok/s	1.54–1.65×
Open-ended Spanish reasoning/chat	~59%	17.7–18.6 tok/s	approximately baseline

These are steady decode measurements from serving logs, not prefill throughput or a first interval diluted by startup/queueing. Both workloads are included because the DSpark paper predicts workload-dependent prefix survival: structured code is more predictable and amortizes target verification better than open-ended multilingual reasoning.

The current bottleneck at wider prefixes remains target verification through the hybrid MoE path. The measured curve makes that cost visible to the scheduler; it does not claim the remaining CPU/GPU expert service is solved.

Scope and follow-ups

• Adaptive compaction is deliberately enabled only for one running request. Multi-request independent widths require the paper's flattened variable-length marker-tensor kernels; padded fixed-width execution is not presented as equivalent.
• gamma remains the checkpoint-trained value of five.
• The PR does not force greedy sampling to raise acceptance numbers.
• Experimental changes to the generic NCCL backend are intentionally excluded from this series; they need broader hardware-aware validation before upstreaming.
• Reaching substantially beyond the measured 1.6× requires reducing target/hybrid-MoE verification time, not merely increasing the confidence threshold.

[gdevenyi]

Reviewed this stack against my open PRs. There's real overlap in the CPU-MoE / host-memory layer: one filename collision worth settling before either lands, and one calibration point that I think bears directly on the adaptive width model here.

1. python/freetoken/utils/numa.py collides — and the two halves are complementary

This stack adds utils/numa.py (topology + how TP ranks spread across nodes; placement() is load-bearing for _tp_core_share). I have a branch not yet up as a PR that adds a file at the same path solving the other half: where the expert banks' pages actually land, via mbind(MPOL_PREFERRED).

Your docstring names the exact failure mine exists for:

If the loader's threads are unbound they scatter across every socket, and the CPU expert pool, pinned to one socket's cores, then reads much of its data across the interconnect.

The part worth flagging: confining the workers without also placing the banks can make things less predictable, not more. On a 2× Xeon Gold 6526Y I measured the CPU MoE GEMV at a steady ~69 GB/s unconfined, and 56–123 GB/s run-to-run once the workers were pinned but the banks were left to first-touch. Binding ranks to nodes — which this stack does — is what puts a machine into that regime, so the placement half is what makes the binding pay reliably rather than turning it into a coin flip.

Since you're already importing placement() from that path, it'd be worth agreeing on the module's API before both land. Your topology/rank-spread half is the one that has to exist first, so I'm happy to rebase mine on top as an additive place_banks() rather than shipping a competing file. The file arrives in #69, so that's where the collision actually bites.

2. The hybrid-vs-offload verdict is calibrated at bs=1; DSpark changes the regime

ft bench bw measures the CPU MoE kernel at bs=1 and picks hybrid-vs-offload from that ratio. But DSpark verification runs the target over width + 1 tokens, so the MoE step this stack executes is bs≈2–6 — never bs=1.

That matters because the CPU MoE arm gets cheaper per route as the block grows: experts that several tokens in the block route to are read from DRAM once instead of once per token. So the ratio deciding hybrid vs offload is currently measured in a regime DSpark never runs in.

#81 adds --batch to ft bench bw for exactly this. On a bandwidth-bound consumer box (9950X, ~59 GB/s host read ceiling), for ds_fp4 — DSV4's expert format — with #46/#47/#49 in:

bs	reuse	speedup
4	1.08x	1.06x
8	1.19x	1.16x
64	3.16x	2.96x

At DSpark's width that's a modest ~1.05–1.1x, and on a 2-socket Xeon with more bandwidth per core it will be smaller still — the traffic saved is hardware-independent, but how much time it buys is not. Not a headline. But it's free, and it moves the right way.

3. It shifts the cost curve this PR profiles

The more interesting consequence. choose_adaptive_draft_width maximizes expected tokens/ms against a profiled verification cost curve. Expert dedup makes the MoE component of that curve sublinear in width, so the width that maximizes the objective moves up. Because you profile at runtime instead of hard-coding the curve, this stack should find the better operating point on its own once #46/#47/#49 land — nothing to change here, but the achievable ceiling rises.

Merge-order note

#36 and this stack both touch moe/cpu_executor.py, but in disjoint regions (mine: _make_table / _resolve_banks; yours: threading/affinity and _tp_core_share). Should merge cleanly in either order.

All my numbers above are from a single-socket consumer box, so treat them as indicative for your 2× Xeon / 4× A4000 target rather than predictive.

[gdevenyi]

Reviewed and benchmarked as part of the #70 → #69 → #71 stack on 2× RTX 6000 Ada, DeepSeek-V4-Flash-0731, offloaded experts, TP=2. Merged cleanly; full suite 1606 passed with no failures attributable to this PR.

I could not isolate this PR's contribution, and I want to be straight about why rather than imply I validated it: with the whole stack applied, speculation is a net ~8% slowdown on this box (35.9 vs 39.3 tok/s at 42% acceptance, details in my comment on #69). Adaptive verification tunes something that is losing here for a reason upstream of it — with offloaded experts, more tokens per step means more expert traffic per step, and expert traffic is the bottleneck. So the adaptive machinery had nothing to recover.

That is a statement about the regime, not about the code. On a resident-expert or latency-bound machine this is presumably where the acceptance-rate gains get realised, and I have no measurement contradicting that.

One thing that would help a reviewer in my position: the PR is +6332/−204 across 53 files, stacked on a +5138 PR, and the end-to-end effect on my hardware is negative for reasons neither PR controls. A short "expected regime" note — resident vs offloaded experts, what acceptance rate is needed to break even, what hardware it was tuned on — would let people decide whether to enable it without running the full stack first. Right now the only way to find out is to build all three and measure.

Not blocking, and no defects found.

[calvarado2004]

Thank you for testing the complete stack and for separating correctness from performance. Your result is consistent with ours rather than contradictory: exact DSpark can lose when the target is dominated by offloaded-expert traffic and prefix survival is low.

The selector does not assume linear verification cost. It measures V(k) for every captured width and maximizes E(k) / (D + V(k)), so the sublinear verification curve introduced by expert dedup should automatically move the chosen width without a selector change. I agree that the calibration path must exercise the same geometry as the deployed path, though.

There is a concrete mismatch in the current calibration that reinforces your point: ft bench bw measures the full DSV4 expert geometry at bs=1, while our TP=4 runtime CPU shard is I=512 and DSpark verifies 2–6 rows. A cached bandwidth-derived q* from that benchmark is therefore only an approximation for the actual speculative workload. #81's batch sweep is the right direction; the deployment profile should ultimately be batch-, shard-, and format-aware.

I also prototyped grouped DSFP4 route processing locally while investigating this. It preserved exact output bits in the CPU/GPU/graph-replay tests, but the production-shaped H=4096, I=512, top-k=6, rows=6 cold-working-set case (34 valid routes, 19 unique experts) improved only 1.03x, and a fixed hot loop regressed about 5%. That agrees with your observation that the gain at DSpark widths is modest and workload/cache dependent. Since #81 and its prerequisite stack already own this work, I do not want to land a competing dedup implementation here; composing the adaptive profiler with that stack is cleaner.

On the expected regime: there is no universal acceptance threshold because break-even depends on D, the measured V(k), and the baseline one-token cost. On the 4x A4000 hybrid system, representative full-width measurements were about 23–27 ms of draft plus 147–166 ms for a width-6 target pass. Against the ~18 tok/s baseline (55.6 ms/token), that cycle needs roughly 3.1–3.5 emitted tokens to break even. Under an iid shorthand for five proposals, that corresponds roughly to proposal survival in the low-to-mid 70% range. The adaptive policy can lower that threshold by choosing a narrower prefix.

The workload measurements show the same boundary clearly:

• structured code at roughly 80–82% proposal acceptance: about 28–30 tok/s versus the 18 tok/s baseline;
• an open-ended poem/reasoning workload at roughly 37% acceptance: about 11–14 tok/s, below baseline;
• your 42% acceptance result, 35.91 versus 39.25 tok/s on the pure-offload TP=2 system, falls in the same losing regime.

So I agree the documentation should say this explicitly: the current win is for a hybrid/resident-enough target path with high prefix survival; pure offload plus low acceptance can be a net loss. Hardware tuning should be enabled and calibrated on the deployment geometry, not treated as a portable constant.

The NUMA work is complementary as well. The current intent is to bind each rank before expert-bank allocation so loader first-touch follows rank affinity, but explicit mbind(MPOL_PREFERRED) makes that placement reliable and observable. Keeping utils.numa as the topology/placement API and adding place_banks() on top is a good composition path.

Thanks again for the full-stack review and for #81. These measurements give us a much sharper boundary for what this PR does and does not accelerate.

[calvarado2004]

This was the baseline without DSpark (but with NUMA awareness already in place, as I have 4 cards I need that).

This is with DSpark working. I'm still improving with the community contributions and feedback, I believe the performance still can be way better than this.

[calvarado2004]

Follow-up correctness fix: we missed one multi-token output-path issue in the original stack. Sorry about that — the final raw-API coherence gate caught it.

When a speculative verify accepted more than one token, the scheduler placed multiple DetokenizeMsgs for the same request into one BatchTokenizerMsg. The original detokenizer appended all of those IDs before advancing that request’s incremental offsets, so it could emit cumulative prefixes (for example, pres followed by presidencia) instead of exact deltas. This was an output transport/detokenization bug; it was not a change to DSpark rejection sampling or the target distribution.

Commit 986823e fixes the contract by carrying the ordered accepted block as token_ids in one message and advancing it sequentially, matching vLLM’s EngineCoreOutput.new_token_ids behavior. Completion-token accounting and offline generation now consume the full block as well.

Validation on the TP=4 hybrid A4000 workstation:

• 56/56 focused detokenization, wire, NUMA, and placement tests passed.
• The exact non-streaming Spanish reasoning gate completed coherently for 991 output tokens with no repeated prefixes.
• Streaming delivery also concatenated cleanly.
• The protected coding baseline remained ~28–29 tok/s with coherent HTML output.

The PR head is now 986823e991ca12355a523caec08b1113e9d8c234, so this fix is included in the [3/3] branch.

[calvarado2004]

Promoted the validated community HC-normalization improvement into this PR.

• Cherry-picked Gabriel Devenyi's fused inv_rms implementation from perf(dsv4): compute the hyper-connection pre-norm in one kernel #101 with original authorship preserved (4082dd8).
• Added the DSpark/production follow-up in df6c340: use the fused path in the drafter, retain bf16 HC combines, and enforce configured prefill chunking so long OpenWebUI histories do not recreate the large fp32 activation/OOM path.
• The focused community composition preserved our established TP4 quality/performance gates: coherent 1,200-token coding output at 28.34–29.54 tok/s, coherent temperature-1 reasoning at 16.52–17.89 tok/s, and no repeated-token regression. The fused-kernel microbenchmark reported in perf(dsv4): compute the hyper-connection pre-norm in one kernel #101 is 6.4x faster at 4K/8K rows and 1.5x at 64 rows.

Thank you Gabriel for the measured optimization and numerical-error analysis. We are deliberately building on the community result rather than duplicating it. The original recovery branch remains protected; this enhanced state is also preserved at calvarado2004:prod/dsv4-dspark-enhanced (df6c340).

[calvarado2004]

Exact promoted-head validation (df6c340) is complete on the 4x RTX A4000 TP4 workstation:

• 76 focused DSV4/DSpark/cache/detokenization tests passed.
• Startup completed without OOM at memory-ratio=0.80, max-prefill-length=1024; 1.22 GiB remained free after CUDA-graph capture.
• The standard 1,200-token HTML coding gate held 27.76–29.22 tok/s after warmup with 82–85% speculative acceptance.
• A separately captured output was coherent, correctly structured HTML/CSS with no repeated-prefix/token regression.

This confirms the PR head itself—not only the larger experiment branch—preserves the established coding baseline and quality gate.

[calvarado2004]

Follow-up after composing the community feedback on the 4x RTX A4000 / dual-NUMA TP4 workstation.

What we adopted:

• Gabriel Devenyi's fused HC inverse-RMS work from perf(dsv4): compute the hyper-connection pre-norm in one kernel #101 is now in this PR with original authorship preserved, plus the DSpark drafter/long-prefill safety follow-up.
• The hybrid q* fetch budget now prioritizes missing experts used by the most rows in the current speculative block, with recency as the tie-breaker.
• We tested direct standard NCCL all-reduce instead of forcing symmetric staging/copy on this PCIe topology. This matches vLLM's default VLLM_USE_NCCL_SYMM_MEM=0; its optional symmetric thresholds are H100/GB200 measurements, not A4000 guidance.

Measured promoted PR head df6c340:

• V(1..6): 73.26 / 95.68 / 127.58 / 151.75 / 157.29 / 173.62 ms
• 1.22 GiB free after graphs
• coherent 1,200-token coding at 27.76-29.22 tok/s, 82-85% acceptance

Isolated milestone 2ce0070 (milestone/dsv4-dspark-33tps-routes):

• V(1..6): 62.16 / 82.73 / 110.03 / 123.74 / 135.82 / 155.66 ms
• 1.26 GiB free after graphs
• 1,200-token coding reached 33.07 tok/s at 83% acceptance and 31.29 tok/s at 84%
• 3,000-token coding held 82-83% acceptance, averaged about 29.6 tok/s over 15 steady checkpoints, and finished at 31.71 tok/s
• streaming and a required tool-call round trip passed without repeated-prefix regression

The re-profile is more important than the peak number. The original trace had symmetric NCCL all-reduce at 878.0 ms / 92.1% CUDA time across 87 calls. The improved trace has standard NCCL at 27.2 ms / 15.8%. The new leader is fast_index_copy_multi at 65.9 ms / 38.3%, so Gabriel's batch-geometry calibration concern is now the measured next bottleneck: speculative q* must balance PCIe expert fetch against CPU execution at rows=2..6, not inherit a bs=1 constant.

Final pre-ready accounting reports 146.62 GiB CPU host experts plus 55.24 GiB GPU footprint, or counted placement CPU 72.6% / GPU 27.4%, with ranks 0/1 on NUMA0 and 2/3 on NUMA1. This is ownership/footprint accounting, not proof of page residency; I still agree explicit mbind/place_banks is stronger than first touch and should compose additively.

Decision: keep exact probabilistic verification plus device-resident adaptive width selection as the integration spine, while composing measured community improvements underneath it. The selector automatically consumed the changed non-linear V(k) curve; no portable constant or static-width replacement was needed.

This is not a universal DSpark claim. Open-ended reasoning remains acceptance-limited, and the reported pure-offload TP2 result at 42% acceptance is a valid losing regime. Our favorable regime is hybrid/resident-enough execution with structured-code survival above 80%.

Also, the earlier repeated-prefix issue was our own multi-token detokenization contract bug, fixed in 986823e; it was not caused by the community changes.

The milestone remains separate while q* is re-calibrated and stability gates continue; we are not replacing the protected/PROD branch on the strength of a peak checkpoint alone.

[gdevenyi]

Thanks for picking up the fused inv_rms and crediting it. Passing the bf16 source to hc_pre_combine instead of xf is the right follow-on: it removes the second consumer of that fp32 copy, which I had left in place because I only touched the reduction.

I merged #69, #70 and #71 at their current heads into a deployment branch with the other open PRs and ran it on 2× RTX 6000 Ada (sm_89), DeepSeek-V4-Flash. Two results worth passing back.

--speculative-dspark crashes during CUDA graph capture

Reproducible at both TP=1 and TP=2, so it is not a tensor-parallel interaction. Without the flag the same branch serves normally.

Default flags (moe_prefill_overlap on):

torch.AcceleratorError: CUDA error: dependency created on uncaptured work in another stream
  cudaErrorStreamCaptureIsolation

  engine/graph.py:387  _capture_graphs
  layers/moe.py:387    _prefill_routed
  layers/moe.py:421    _wait_prefill_overlap
  moe/offload_cache.py:613  prefetch_prefill_layer
  torch/cuda/streams.py:63  wait_event

The MoE prefill-overlap prefetch stream waits on an event while the graph is being captured.

With --disable-moe-prefill-overlap the failure changes rather than clearing:

torch.AcceleratorError: CUDA error: out of memory
  cudaErrorMemoryAllocation
  engine/graph.py:387  _capture_graphs

so there is a second issue: the drafter's own VRAM demand is not accounted for in the capture budget at --memory-ratio 0.94. I have not yet swept the ratio down to find where it starts capturing.

Happy to run any diagnostic you want on this hardware.

The prefill chunk limit lifted the memory-ratio ceiling

On the previous branch --memory-ratio 0.97 OOMed on a ~6600-token prefill, which capped that deployment at 0.94. With this branch, 0.97 completes prompts of 2k, 4k, 6.6k, 9k and 12k tokens, and aggregate throughput at 8 concurrent streams goes 115.5 → 119.5 tok/s. I attribute that to the chunk limit in df6c340 together with the two fp32 activations the fused norm no longer materialises.

[calvarado2004]

PR3 head has now been promoted from df6c340 to the confirmed-gains baseline at 2ce0070.

Newly included in this PR:

• ad423c6: direct in-place standard NCCL all-reduce;
• 507f172 + 9f4595d: final NUMA-aware CPU/GPU distribution report and self-contained topology fix;
• 2ce0070: spend the existing paper-derived q* fetch budget on the missing experts repeated across the most speculative rows.

Promotion gates on the TP=4 RTX A4000 workstation:

• long coding remained coherent and repeat-free at ~29.6 tok/s sustained, with 33.07 tok/s observed peak;
• explicit reasoning_effort=high coding averaged 30.36 tok/s after warm-up (28.31–32.10 range) with ~86.9% isolated speculative acceptance;
• bounded high-effort reasoning improved quality: Bayes completed correctly at 1/6, and knights/knaves completed with a clean symbolic proof;
• required tool selection, JSON arguments, tool-result follow-up, streaming, and non-streaming paths passed;
• the same 87 TP collective calls fell from 878.0 ms / 92.1% of the old CUDA profile to 27.2 ms / 15.8% with the direct NCCL path;
• the startup report shows the counted model placement as 72.6% CPU / 27.4% GPU, with ranks/worker pools mapped across both NUMA nodes.

The request-level reasoning_effort setting is not itself a code commit; these measurements validate that the promoted code preserves ~30 tok/s coding while materially improving structured reasoning when callers select high.

A newer experiment implementing the FreeToken paper's “at least one cache fill” rule remains isolated on the experiment branch and is intentionally not included here until its A/B proves a net gain.

[calvarado2004]

I feel that my setup should be able to reach 60 tok/s on coding tasks or so. I won't stop until proven wrong on that idea haha @gdevenyi reasoning could be lower, around. 35 tok/s but that would be amazing for a 75% on CPU model.

[calvarado2004]

I added NUMA informative logs just before the model is ready to serve, to inform how the model is split on the machine:

[calvarado2004]

Thank you — both results are extremely useful, and the independent 115.5 -> 119.5 tok/s result is especially valuable validation of the fused norm + bf16 combine + chunk-limit follow-on.

I traced the capture stack and found a likely common cause for the two failure modes.

At the widest DSpark verify span, T * top_k crosses the DSV4 prompt-density threshold and intentionally selects the fast whole-layer prefill-overlap path. The eager warm-up leaves _prefill_buffer_has_release_event set. begin_prefill() already fences the copy stream behind the new compute stream, but capture then redundantly waits on those old eager release events. That produces the uncaptured-work dependency in your stack. With overlap disabled, the same wide verify captures synchronous whole-layer materialization instead, which plausibly explains the OOM without requiring a separate drafter-weight accounting bug.

The current fix is here:

• branch: https://github.com/calvarado2004/FreeToken/tree/fix/dspark-graph-capture-v2
• head: 95e3233

It does two things:

1. after the new begin-prefill fence, clear release-event-presence flags from the preceding eager pass; releases recorded inside the current prefill/capture set them again, so intra-prefill buffer reuse remains ordered;
2. preserve the fast whole-layer path when overlap is enabled, but use hybrid/on-demand experts for wide DSpark verify when overlap is explicitly disabled, avoiding full-layer capture materialization.

Lenovo validation on 4x RTX A4000 Ampere (sm_86):

• 205 focused DSV4/hybrid/distribution tests pass;
• default-overlap DSpark graph capture succeeds;
• post-graph free VRAM remains 1.26 GiB;
• verify curve: 58.77 / 84.89 / 111.05 / 123.12 / 139.84 / 158.24 ms for rows 1..6;
• 2,000-token coding check: coherent/repeat-free, 30.26 tok/s steady mean, 32.41 peak, ~89% isolated acceptance.

Could you please try the branch on RTX 6000 Ada in this matrix?

• default overlap, TP=1 and TP=2, first at your known 0.94;
• default overlap at 0.97 if 0.94 captures;
• --disable-moe-prefill-overlap at 0.94 to exercise the on-demand fallback.

If default overlap still throws capture isolation, the next useful detail is the first event/stream line after begin_prefill. If capture succeeds but 0.97 OOMs, then we have clean evidence of an independent graph-headroom budget issue and can size that separately rather than conflate it with full-layer capture.

Thanks again for testing this on hardware we do not have locally.

[calvarado2004]

Promoted the validated request-local low-acceptance fallback into PR3 at 16ce51a.

What changed

This is an opt-in circuit breaker around the existing exact DSpark path; it does not classify prompts or replace the paper's measured-width selector.

• accumulate actual accepted/drafted proposals per request;
• after at least 32 proposals, a window below 60% acceptance uses ordinary target decoding for 64 steps;
• probe DSpark again after the cooldown and resume it only when the workload supports it;
• reset all state at the request boundary;
• keep the feature path current during target-only steps, so a later probe receives current target hidden features/context KV rather than stale state.

The feature is disabled by default (--dspark-fallback-acceptance=0). The Lenovo experiment used 0.60 / 32 / 64. Target-only decoding and the existing rejection sampler both preserve the target distribution; the policy only avoids paying for repeatedly unproductive drafts.

Exact PR-head validation

The pushed PR head was fetched into an isolated Lenovo Git worktree and passed:

202 passed in 56.73s

Suite: tests/moe/test_hybrid_fetch.py tests/dsv4.

Hardware remains the 4x RTX A4000 TP4 hybrid workstation, with the counted placement reported by the runtime as 72.6% CPU / 27.4% GPU.

Structured-code control

The standard 600-token pet-store landing-page request did not enter fallback:

• 600 coherent HTML/CSS tokens in 21.11 s;
• 28.43 tok/s end-to-end;
• steady scheduler windows 30.55 and 27.33 tok/s;
• no repeated-token/prefix regression.

So the established approximately 30 tok/s coding baseline is preserved.

Low-acceptance creative-prose control

Identical temperature-1 prompt: tell me a story about horses and humans.

Run	Completion	Wall	Rate
Prior fallback-disabled run	about 1,536 tokens	about 98 s	about 15.7 tok/s
Request-local fallback	1,513 tokens	88.07 s	17.18 tok/s

The fallback run repeatedly measured local windows in the 30.3%-59.4% range. Target-only phases sustained 17.86-19.59 tok/s. The response stopped naturally, was coherent end to end, and contained no repetition. This is about 10% lower wall time on this matched sample; it is evidence for this hardware/workload, not a universal claim.

Long high-reasoning quality/stability gate

Prompt: the 1/2/7/10-minute bridge-and-torch problem, temperature 1.0, reasoning_effort=high, requiring both the minimum and an optimality proof.

The intermediate 8K test did not pass the output gate:

• 8,000 completion tokens in 441.95 s (18.10 tok/s);
• finish_reason=length, no final-answer channel;
• reasoning consistently identified the correct 17-minute solution;
• only 12 duplicate 12-word spans across 29,169 reasoning characters, so this was long proof search rather than a token-repetition loop.

With a 20K ceiling, the identical test did complete naturally:

• 16,941 completion tokens in 972.86 s (17.41 tok/s end-to-end);
• finish_reason=stop;
• 60,905 reasoning characters followed by a concise 1,400-character final answer;
• correct schedule 2 + 1 + 10 + 2 + 2 = 17 and a valid lower-bound argument;
• no repetition loop (57 duplicate 12-word spans over the entire chain; maximum count 3);
• no OOM, state corruption, or scheduler failure across the sustained DSpark/target-only transitions.

This passes the requested long-reasoning correctness/stability gate, while also showing that DeepSeek-V4-Flash high effort may legitimately require a very large reasoning budget for an explicit optimality proof. The fallback improves low-acceptance execution; it does not attempt to change the model's reasoning termination policy.

Scope

The result sharpens the expected regime already discussed above:

• high-survival structured code stays on full DSpark and retains its gain;
• persistently low-survival creative/reasoning requests recover toward target-only performance instead of remaining below baseline;
• the thresholds remain opt-in and hardware-calibrated rather than portable constants.