[None][infra] Unwaive 5 disagg gen only cases - #17442
Conversation
WalkthroughThe change removes three skipped performance sanity test entries for Qwen3.5 397B FP4 Blackwell configurations from the integration waiver list. ChangesPerformance waiver cleanup
Estimated code review effort: 1 (Trivial) | ~2 minutes Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Removes five perf-sanity waivers from tests/integration/test_lists/waives.txt across two bugs. nvbugs/6535767 -- fixed by PR NVIDIA#17163 ("Drain in-flight requests before clearing the KV cache reuse state", merged 2026-08-06 as b7a9d6f), which resolved the "V2 Mamba state pool has only X slots" startup failure but did not clean up the waivers it obsoleted: - test_e2e[aggr_upload-qwen3_5_397b_fp4_blackwell-qwen3_5_397b_fp4_dep8_8k1k] - test_e2e[aggr_upload-qwen3_5_397b_fp4_blackwell-qwen3_5_397b_fp4_dep8_mtp3_8k1k] - test_e2e[aggr_upload-qwen3_5_397b_fp4_blackwell-qwen3_5_397b_fp4_tp4_8k1k] nvbugs/6490049 -- verified individually and unwaived case by case, the same pattern as PR NVIDIA#16920 ("Unwaive GB300 Kimi disagg e2e", merged 2026-08-06): - test_e2e[disagg_upload-gen_only-gb200_deepseek-v32-fp4_8k1k_con4096_ctx1_dep4_gen1_dep32_eplb256_mtp0_ccb-NIXL] - test_e2e[disagg_upload-gen_only-gb200_gpt-oss-120b-fp4_8k1k_con4_ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-NIXL] All five cases remain listed in their respective test-db lists (l0_b200_multi_gpus_perf_sanity.yml, l0_gb200_multi_nodes_perf_sanity_ctx1_node1_gpu4_gen1_node8_gpu32.yml, l0_gb200_multi_nodes_perf_sanity_ctx1_node1_gpu1_gen1_node1_gpu4.yml), so removing the waivers re-enables them rather than leaving them unrun. The remaining nvbugs/6490049 waiver (gb300 kimi-k25-thinking gen_only) is intentionally left in place. Signed-off-by: chenfeiz0326 <203214996+chenfeiz0326@users.noreply.github.com>
c1b6f18 to
0e8d1b9
Compare
|
/bot run --disable-fail-fast --stage-list "DGX_B200-8_GPUs-PyTorch-PerfSanity-Post-Merge-2,DGX_B200-8_GPUs-PyTorch-PerfSanity-Post-Merge-4,GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-5,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-3" |
|
PR_Github #64842 [ run ] triggered by Bot. Commit: |
|
PR_Github #64842 [ run ] completed with state
|
|
/bot skip --comment "Only unwaive perf tests, no need to run the whole CI pipeline" |
|
PR_Github #64927 [ skip ] triggered by Bot. Commit: |
|
PR_Github #64927 [ skip ] completed with state |
Description
Removes five perf-sanity waivers from
tests/integration/test_lists/waives.txt,spanning two bugs.
https://nvbugs/6535767 — fix merged
The
ValueError: The V2 Mamba state pool has only X slots but needs at least N ...startup failure was fixed by #17163 ("Drain in-flight requests before clearing
the KV cache reuse state"), merged to
mainon 2026-08-06 asb7a9d6f5d7.That PR did not clean up the waivers it obsoleted, which were added in
post-merge 2869 (#17041):
perf/test_perf_sanity.py::test_e2e[aggr_upload-qwen3_5_397b_fp4_blackwell-qwen3_5_397b_fp4_dep8_8k1k]perf/test_perf_sanity.py::test_e2e[aggr_upload-qwen3_5_397b_fp4_blackwell-qwen3_5_397b_fp4_dep8_mtp3_8k1k]perf/test_perf_sanity.py::test_e2e[aggr_upload-qwen3_5_397b_fp4_blackwell-qwen3_5_397b_fp4_tp4_8k1k]https://nvbugs/6490049 — verified case by case
This bug tracks an intermittent gen_only harness metric-extraction failure
(
missing 'prev_device_step_time'/ missingAverage Per Iter Device Step Time)across 13 cases, and is still open. Its waivers are being retired individually
as each case is verified — the same pattern as #16920 ("Unwaive GB300 Kimi
disagg e2e", merged 2026-08-06), which unwaived one case while the bug
remained open. This PR retires two more:
perf/test_perf_sanity.py::test_e2e[disagg_upload-gen_only-gb200_deepseek-v32-fp4_8k1k_con4096_ctx1_dep4_gen1_dep32_eplb256_mtp0_ccb-NIXL]perf/test_perf_sanity.py::test_e2e[disagg_upload-gen_only-gb200_gpt-oss-120b-fp4_8k1k_con4_ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-NIXL]The remaining 6490049 waiver (
gb300_kimi-k25-thinking-fp4_8k1k_con1024_...)is intentionally left in place.
Test Coverage
Waiver removal only — no source changes. All five cases remain listed in their
test-db lists, so removing the waivers re-enables them rather than leaving them
unrun:
qwen3_5_397b_fp4l0_b200_multi_gpus_perf_sanity.ymldeepseek-v32-fp4_8k1k_con4096l0_gb200_multi_nodes_perf_sanity_ctx1_node1_gpu4_gen1_node8_gpu32.ymlgpt-oss-120b-fp4_8k1k_con4l0_gb200_multi_nodes_perf_sanity_ctx1_node1_gpu1_gen1_node1_gpu4.ymlPR Checklist
🤖 Generated with Claude Code
Dev Engineer Review
qwen3_5_397b_fp4perf-sanity waivers fromtests/integration/test_lists/waives.txt.6535767from PR#17163.6535767remain unchanged.QA Engineer Review
test-db/orqa/files were modified.Verdict: needs follow-up