fix(core): stop the paged budget test underflowing past the v2 reserve - #1130
Merged
Conversation
`resolve_block_budget_explicit_bytes_floors_to_block_count` failed deterministically on GB10 / CUDA sm_121 / Linux aarch64 at `assertion failed: shortfall < 100`. The test computed its own expected value with `(per_block * 100 - workspace) / per_block`, an unsigned subtraction of a device-derived reserve from a fixed budget. `device_target_ctas()` returns 512 on every non-Metal host, which puts the paged decode v2 workspace reserve at 16.25 MiB for this geometry against a 12.5 MiB budget, so the `u64` subtraction wrapped and `shortfall` became 2^47 - 31. The underflow is test-only, and that is confirmed rather than assumed. `paged_v2_workspace_reserve_bytes` has four call sites: the implementation in `resolve_paged_block_budget`, which already used `saturating_sub`, the failing test, and two measurement-only sites that never subtract. The new `resolve_block_budget_below_the_workspace_reserve_is_zero_blocks` calls the production function at budgets of 0, 1, workspace/2, workspace-1 and workspace and requires `Some(0)` for each, so the implementation is measured clean rather than read clean; a wrapping `-` would return `Some(usize::MAX)` for the last three. The repaired test mirrors the implementation's `saturating_sub` and states the contract twice so neither statement is vacuous on any host. Capped: the reserve costs `min(ceil(workspace / per_block), 100)` of the 100 blocks requested, which is an identity below the crossover and pins both sides at 100 above it. Uncapped: asking for `per_block * (100 + ceil(workspace / per_block))` bytes resolves to exactly 100 blocks, which exercises the exact-reserve arithmetic on large-CTA devices instead of leaving it covered only on small Apple parts. The environment override was rejected because `device_target_ctas()` memoizes through a `OnceLock`, so pinning it inside a test is order-dependent under a shared test binary. The issue also asked whether the resulting `Some(0)` is reachable and handled by callers rather than assumed. It is: `resolve_worker_paged_block_budget` warns and leaves the pool unbounded instead of installing a wedging zero budget. Its warning blamed model size and available memory, which is wrong for an explicit byte budget, since any `--kv-cache-budget` under roughly 16.25 MiB lands there on a non-Metal host regardless of model size. The message now names the reserve in bytes and the one-block-on-top-of-it threshold. No budget resolves to a different block count than before. Validated on GB10: the previously failing test passes, `execution::memory_estimate` is 36 passed and 0 failed, `cargo clippy --profile test-fast --features cuda --lib --tests -- -D warnings` is clean, and `cargo fmt --all -- --check` is clean. Closes #1091
6 tasks
Review found the previous commit gave the reserve-naming warning to both `--kv-cache-budget` directives. It is accurate for `auto` (a budget under `reserve + per_block` is what both paths hit) but it is the wrong diagnosis, and `auto` is the shipped default on both binaries. `auto` reaches zero blocks when `auto_kv_budget_bytes` finds the model leaves no room for KV at all, where the real shortfall is tens of gigabytes; pointing at a 16 MiB workspace reserve there sends the operator after `MLXCEL_PAGED_DECODE_V2_TARGET_CTAS`, which is the wrong knob. The deleted "model too large for a meaningful paged budget" text was the correct diagnosis for exactly that case. The zero-block arm now matches on the directive. `Auto` keeps its original wording verbatim. `Bytes` reports the requested budget, the workspace reserve, and the smallest budget that would mint one block, all rendered through the module's own `format_bytes`, so the operator gets a threshold they can act on rather than a raw byte count they would have to combine with the per-block cost themselves. `PagedBudgetDirective` is `Copy` and the directive is still in scope, so the split costs two lines. The `paged_block_bytes` fallback is unreachable, since `resolve_paged_block_budget` returns `None` rather than `Some(0)` when the per-block cost is underivable, but it is kept total so a later change to that contract degrades the message instead of panicking. Also from the same review round: The wrapped-block-count magnitude was wrong in the comments. A wrapping `-` produces a budget near `u64::MAX`, which divides down to roughly 2^47 blocks for a 128 KiB block, not `usize::MAX`: the `unwrap_or(usize::MAX)` only fires where `usize::try_from` can fail, i.e. a 32-bit target. Issue #1091 computed the right magnitude itself. Corrected in the implementation comment, the new test's doc comment, and the committed report. The guard is unaffected, since the test asserts `Some(0)` and any non-zero value fails it, but this comment exists to stop a future simplification back to `-`, so the number in it should be right. The reserve-derived arithmetic in `resolve_block_budget_explicit_bytes_floors_to_block_count` and the new regression test now saturates throughout. The `test-fast` profile inherits `release`, so overflow checks are off and a plain `*` or `+` on a device with an extreme CTA target would wrap silently into a wrong expectation rather than panicking. That is the same class of defect this test exists to pin, and the PR's own thesis is that the expectation may not assume the reserve is small. The capped assertion's failure message read "a 17040384-byte reserve should cost 131 of the 100 blocks requested" on GB10, the machine this was written for. Reworded to state the cap explicitly. Validated on GB10: `execution::memory_estimate` is 36 passed and 0 failed, `cargo clippy --profile test-fast --features cuda --lib --tests -- -D warnings` is clean, and `cargo fmt --all -- --check` is clean. Refs #1091
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
execution::memory_estimate::tests::resolve_block_budget_explicit_bytes_floors_to_block_countfailed deterministically in 0.00s on GB10 / CUDA sm_121 / Linux aarch64 atassertion failed: shortfall < 100. The test computed its own expected value with(per_block * 100 - workspace) / per_block, an unsigned subtraction of a device-derived reserve from a fixed budget.device_target_ctas()returns 512 on every non-Metal host, putting the paged decode v2 workspace reserve at 16.25 MiB for this geometry against a 12.5 MiB budget, so theu64subtraction wrapped andshortfallbecame2^47 - 31.Where the underflow lives: test-only, confirmed
The issue's evidence said test-only and the acceptance criteria required confirming or refuting that rather than assuming it. Both halves of the confirmation are here.
paged_v2_workspace_reserve_byteshas exactly four call sites in the tree:memory_estimate.rs(resolve_paged_block_budget, production)saturating_sub, already correctmemory_estimate.rs(the failing test)-, wrapsthe_v2_workspace_reserve_is_small_and_scales_with_the_head_dim, 2 sitesOne subtraction in non-test code, and it already saturated. The claim does not rest on the grep: the new
resolve_block_budget_below_the_workspace_reserve_is_zero_blockscalls the production function at budgets of0,1,workspace / 2,workspace - 1andworkspaceand requiresSome(0)for each. A wrapping-would return a block count near2^47for the last three (a budget nearu64::MAXover a 128 KiB block, notusize::MAX, which on this host thetry_fromnever reaches). The test is green, so the implementation is measured clean, not read clean.What changed
src/execution/memory_estimate.rsresolve_paged_block_budget: a comment recording why the subtraction must saturate, with the numbers that make a sub-reserve budget reachable on any non-Metal host, so thesaturating_subis not "simplified" back to-later.resolve_block_budget_explicit_bytes_floors_to_block_count: the wrapping-becomessaturating_sub; the arithmetic sanity checkshortfall < 100is replaced by the exact-reserve contract stated twice, so neither statement is vacuous on any host. Capped:100 - shortfall == min(ceil(workspace / per_block), 100), an identity below the crossover and pinned at 100 above it. Uncapped: asking forper_block * (100 + ceil(workspace / per_block))bytes resolves to exactly 100 blocks, which exercises the exact-reserve arithmetic on large-CTA devices rather than leaving it covered only on small Apple parts. All reserve-derived request and expectation arithmetic in the test is now saturating, sincetest-fastinheritsreleaseand would otherwise wrap silently into a wrong expectation rather than panicking.resolve_block_budget_below_the_workspace_reserve_is_zero_blocks, pinning the boundary from both sides:workspace + per_block - 1yieldsSome(0),workspace + per_blockyieldsSome(1).src/server/model_worker.rs: the zero-block arm ofresolve_worker_paged_block_budgetemitted one message for both directives, and it blamed model size and available memory. That is the right diagnosis for--kv-cache-budget autoand the wrong one for an explicit byte budget, where on a non-Metal host anything under roughly 16 MiB lands there regardless of model size. The arm now matches on the directive:Autokeeps its original wording, andBytesreports the requested budget, the reserve, and the smallest budget that would mint one block, through the module'sformat_bytes. The issue asked whether theSome(0)path is reachable and handled by callers rather than assumed; it is reachable, and the caller already warns and leaves the pool unbounded rather than installing a wedging zero budget. Only message text changed.CHANGELOG.md: one### Fixedentry under## [Unreleased], stating explicitly that no budget resolves to a different block count and thatautokeeps its message.The
MLXCEL_PAGED_DECODE_V2_TARGET_CTASoverride was rejected.device_target_ctas()memoizes through aOnceLock, so pinning it inside a test is order-dependent under a shared test binary, which would trade a deterministic failure for an order-dependent one.Review round
A review pass found that the first version of the warning gave the reserve-naming message to
Autoas well. It is accurate there (budget < reserve + per_blockholds either way) but misleading, sinceAutois the shipped default and reaches zero blocks when the model leaves no room for KV at all, where the real shortfall is tens of gigabytes and the reserve is a red herring. Fixed by splitting on the directive. The same pass corrected the magnitude of the wrapped block count fromusize::MAXto roughly2^47, in the code comments, the tests, this body, and the committed report.Test plan
cargo test --profile test-fast --features cuda --lib -- --exact execution::memory_estimate::tests::resolve_block_budget_explicit_bytes_floors_to_block_counton GB10, the case that failed beforecargo test --profile test-fast --features cuda --lib execution::memory_estimateon GB10: 36 passed, 0 failedcargo clippy --profile test-fast --features cuda --lib --tests -- -D warningscargo fmt --all -- --checkNot covered: the Apple small-CTA branch was not run on hardware. It is covered by construction rather than by execution, since every assertion is now either an identity over the measured
workspaceor a request computed from it.--test-threads=1was not used; the full lib suite aborts under parallel execution on this CUDA host for reasons unrelated to this change, so the module filter above is the equivalent evidence. The changed warning text was read, not observed in a live server log.Closes #1091