Skip to content

fix(core): stop the paged budget test underflowing past the v2 reserve - #1130

Merged
inureyes merged 3 commits into
mainfrom
fix/issue-1091-paged-budget-underflow
Aug 13, 2026
Merged

fix(core): stop the paged budget test underflowing past the v2 reserve#1130
inureyes merged 3 commits into
mainfrom
fix/issue-1091-paged-budget-underflow

Conversation

@inureyes

@inureyes inureyes commented Aug 13, 2026

Copy link
Copy Markdown
Member

Summary

execution::memory_estimate::tests::resolve_block_budget_explicit_bytes_floors_to_block_count failed deterministically in 0.00s on GB10 / CUDA sm_121 / Linux aarch64 at assertion failed: shortfall < 100. The test computed its own expected value with (per_block * 100 - workspace) / per_block, an unsigned subtraction of a device-derived reserve from a fixed budget. device_target_ctas() returns 512 on every non-Metal host, putting the paged decode v2 workspace reserve at 16.25 MiB for this geometry against a 12.5 MiB budget, so the u64 subtraction wrapped and shortfall became 2^47 - 31.

Where the underflow lives: test-only, confirmed

The issue's evidence said test-only and the acceptance criteria required confirming or refuting that rather than assuming it. Both halves of the confirmation are here.

paged_v2_workspace_reserve_bytes has exactly four call sites in the tree:

Site Subtracts from a budget? Arithmetic
memory_estimate.rs (resolve_paged_block_budget, production) yes saturating_sub, already correct
memory_estimate.rs (the failing test) yes -, wraps
the_v2_workspace_reserve_is_small_and_scales_with_the_head_dim, 2 sites no measurement only

One subtraction in non-test code, and it already saturated. The claim does not rest on the grep: the new resolve_block_budget_below_the_workspace_reserve_is_zero_blocks calls the production function at budgets of 0, 1, workspace / 2, workspace - 1 and workspace and requires Some(0) for each. A wrapping - would return a block count near 2^47 for the last three (a budget near u64::MAX over a 128 KiB block, not usize::MAX, which on this host the try_from never reaches). The test is green, so the implementation is measured clean, not read clean.

What changed

  • src/execution/memory_estimate.rs
    • resolve_paged_block_budget: a comment recording why the subtraction must saturate, with the numbers that make a sub-reserve budget reachable on any non-Metal host, so the saturating_sub is not "simplified" back to - later.
    • resolve_block_budget_explicit_bytes_floors_to_block_count: the wrapping - becomes saturating_sub; the arithmetic sanity check shortfall < 100 is replaced by the exact-reserve contract stated twice, so neither statement is vacuous on any host. Capped: 100 - shortfall == min(ceil(workspace / per_block), 100), an identity below the crossover and pinned at 100 above it. Uncapped: asking for per_block * (100 + ceil(workspace / per_block)) bytes resolves to exactly 100 blocks, which exercises the exact-reserve arithmetic on large-CTA devices rather than leaving it covered only on small Apple parts. All reserve-derived request and expectation arithmetic in the test is now saturating, since test-fast inherits release and would otherwise wrap silently into a wrong expectation rather than panicking.
    • New resolve_block_budget_below_the_workspace_reserve_is_zero_blocks, pinning the boundary from both sides: workspace + per_block - 1 yields Some(0), workspace + per_block yields Some(1).
  • src/server/model_worker.rs: the zero-block arm of resolve_worker_paged_block_budget emitted one message for both directives, and it blamed model size and available memory. That is the right diagnosis for --kv-cache-budget auto and the wrong one for an explicit byte budget, where on a non-Metal host anything under roughly 16 MiB lands there regardless of model size. The arm now matches on the directive: Auto keeps its original wording, and Bytes reports the requested budget, the reserve, and the smallest budget that would mint one block, through the module's format_bytes. The issue asked whether the Some(0) path is reachable and handled by callers rather than assumed; it is reachable, and the caller already warns and leaves the pool unbounded rather than installing a wedging zero budget. Only message text changed.
  • CHANGELOG.md: one ### Fixed entry under ## [Unreleased], stating explicitly that no budget resolves to a different block count and that auto keeps its message.

The MLXCEL_PAGED_DECODE_V2_TARGET_CTAS override was rejected. device_target_ctas() memoizes through a OnceLock, so pinning it inside a test is order-dependent under a shared test binary, which would trade a deterministic failure for an order-dependent one.

Review round

A review pass found that the first version of the warning gave the reserve-naming message to Auto as well. It is accurate there (budget < reserve + per_block holds either way) but misleading, since Auto is the shipped default and reaches zero blocks when the model leaves no room for KV at all, where the real shortfall is tens of gigabytes and the reserve is a red herring. Fixed by splitting on the directive. The same pass corrected the magnitude of the wrapped block count from usize::MAX to roughly 2^47, in the code comments, the tests, this body, and the committed report.

Test plan

  • cargo test --profile test-fast --features cuda --lib -- --exact execution::memory_estimate::tests::resolve_block_budget_explicit_bytes_floors_to_block_count on GB10, the case that failed before
  • cargo test --profile test-fast --features cuda --lib execution::memory_estimate on GB10: 36 passed, 0 failed
  • cargo clippy --profile test-fast --features cuda --lib --tests -- -D warnings
  • cargo fmt --all -- --check

Not covered: the Apple small-CTA branch was not run on hardware. It is covered by construction rather than by execution, since every assertion is now either an identity over the measured workspace or a request computed from it. --test-threads=1 was not used; the full lib suite aborts under parallel execution on this CUDA host for reasons unrelated to this change, so the module filter above is the equivalent evidence. The changed warning text was read, not observed in a live server log.

Closes #1091

`resolve_block_budget_explicit_bytes_floors_to_block_count` failed deterministically on GB10 / CUDA sm_121 / Linux aarch64 at `assertion failed: shortfall < 100`. The test computed its own expected value with `(per_block * 100 - workspace) / per_block`, an unsigned subtraction of a device-derived reserve from a fixed budget. `device_target_ctas()` returns 512 on every non-Metal host, which puts the paged decode v2 workspace reserve at 16.25 MiB for this geometry against a 12.5 MiB budget, so the `u64` subtraction wrapped and `shortfall` became 2^47 - 31.

The underflow is test-only, and that is confirmed rather than assumed. `paged_v2_workspace_reserve_bytes` has four call sites: the implementation in `resolve_paged_block_budget`, which already used `saturating_sub`, the failing test, and two measurement-only sites that never subtract. The new `resolve_block_budget_below_the_workspace_reserve_is_zero_blocks` calls the production function at budgets of 0, 1, workspace/2, workspace-1 and workspace and requires `Some(0)` for each, so the implementation is measured clean rather than read clean; a wrapping `-` would return `Some(usize::MAX)` for the last three.

The repaired test mirrors the implementation's `saturating_sub` and states the contract twice so neither statement is vacuous on any host. Capped: the reserve costs `min(ceil(workspace / per_block), 100)` of the 100 blocks requested, which is an identity below the crossover and pins both sides at 100 above it. Uncapped: asking for `per_block * (100 + ceil(workspace / per_block))` bytes resolves to exactly 100 blocks, which exercises the exact-reserve arithmetic on large-CTA devices instead of leaving it covered only on small Apple parts. The environment override was rejected because `device_target_ctas()` memoizes through a `OnceLock`, so pinning it inside a test is order-dependent under a shared test binary.

The issue also asked whether the resulting `Some(0)` is reachable and handled by callers rather than assumed. It is: `resolve_worker_paged_block_budget` warns and leaves the pool unbounded instead of installing a wedging zero budget. Its warning blamed model size and available memory, which is wrong for an explicit byte budget, since any `--kv-cache-budget` under roughly 16.25 MiB lands there on a non-Metal host regardless of model size. The message now names the reserve in bytes and the one-block-on-top-of-it threshold. No budget resolves to a different block count than before.

Validated on GB10: the previously failing test passes, `execution::memory_estimate` is 36 passed and 0 failed, `cargo clippy --profile test-fast --features cuda --lib --tests -- -D warnings` is clean, and `cargo fmt --all -- --check` is clean.

Closes #1091
@inureyes inureyes added type:bug Bug fixes, error corrections, or issue resolutions priority:medium Medium priority area:core mlxcel-core: MLX FFI, primitives, KV cache, layers status:review Under review labels Aug 13, 2026
Review found the previous commit gave the reserve-naming warning to both `--kv-cache-budget` directives. It is accurate for `auto` (a budget under `reserve + per_block` is what both paths hit) but it is the wrong diagnosis, and `auto` is the shipped default on both binaries. `auto` reaches zero blocks when `auto_kv_budget_bytes` finds the model leaves no room for KV at all, where the real shortfall is tens of gigabytes; pointing at a 16 MiB workspace reserve there sends the operator after `MLXCEL_PAGED_DECODE_V2_TARGET_CTAS`, which is the wrong knob. The deleted "model too large for a meaningful paged budget" text was the correct diagnosis for exactly that case.

The zero-block arm now matches on the directive. `Auto` keeps its original wording verbatim. `Bytes` reports the requested budget, the workspace reserve, and the smallest budget that would mint one block, all rendered through the module's own `format_bytes`, so the operator gets a threshold they can act on rather than a raw byte count they would have to combine with the per-block cost themselves. `PagedBudgetDirective` is `Copy` and the directive is still in scope, so the split costs two lines. The `paged_block_bytes` fallback is unreachable, since `resolve_paged_block_budget` returns `None` rather than `Some(0)` when the per-block cost is underivable, but it is kept total so a later change to that contract degrades the message instead of panicking.

Also from the same review round:

The wrapped-block-count magnitude was wrong in the comments. A wrapping `-` produces a budget near `u64::MAX`, which divides down to roughly 2^47 blocks for a 128 KiB block, not `usize::MAX`: the `unwrap_or(usize::MAX)` only fires where `usize::try_from` can fail, i.e. a 32-bit target. Issue #1091 computed the right magnitude itself. Corrected in the implementation comment, the new test's doc comment, and the committed report. The guard is unaffected, since the test asserts `Some(0)` and any non-zero value fails it, but this comment exists to stop a future simplification back to `-`, so the number in it should be right.

The reserve-derived arithmetic in `resolve_block_budget_explicit_bytes_floors_to_block_count` and the new regression test now saturates throughout. The `test-fast` profile inherits `release`, so overflow checks are off and a plain `*` or `+` on a device with an extreme CTA target would wrap silently into a wrong expectation rather than panicking. That is the same class of defect this test exists to pin, and the PR's own thesis is that the expectation may not assume the reserve is small.

The capped assertion's failure message read "a 17040384-byte reserve should cost 131 of the 100 blocks requested" on GB10, the machine this was written for. Reworded to state the cap explicitly.

Validated on GB10: `execution::memory_estimate` is 36 passed and 0 failed, `cargo clippy --profile test-fast --features cuda --lib --tests -- -D warnings` is clean, and `cargo fmt --all -- --check` is clean.

Refs #1091
@inureyes inureyes added status:done Completed and removed status:review Under review labels Aug 13, 2026
@inureyes
inureyes merged commit 8a5f061 into main Aug 13, 2026
8 checks passed
@inureyes
inureyes deleted the fix/issue-1091-paged-budget-underflow branch August 13, 2026 18:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:core mlxcel-core: MLX FFI, primitives, KV cache, layers priority:medium Medium priority status:done Completed type:bug Bug fixes, error corrections, or issue resolutions

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: Paged block budget test underflows when the paged-v2 workspace reserve exceeds the byte budget

1 participant