Skip to content

[None][infra] Unwaive 5 disagg gen only cases - #17442

Merged
chenfeiz0326 merged 1 commit into
NVIDIA:mainfrom
chenfeiz0326:user-chenfeiz/unwaive-6535767-perf-sanity
Aug 10, 2026
Merged

[None][infra] Unwaive 5 disagg gen only cases#17442
chenfeiz0326 merged 1 commit into
NVIDIA:mainfrom
chenfeiz0326:user-chenfeiz/unwaive-6535767-perf-sanity

Conversation

@chenfeiz0326

@chenfeiz0326 chenfeiz0326 commented Aug 9, 2026

Copy link
Copy Markdown
Collaborator

Description

Removes five perf-sanity waivers from tests/integration/test_lists/waives.txt,
spanning two bugs.

https://nvbugs/6535767 — fix merged

The ValueError: The V2 Mamba state pool has only X slots but needs at least N ...
startup failure was fixed by #17163 ("Drain in-flight requests before clearing
the KV cache reuse state"
), merged to main on 2026-08-06 as b7a9d6f5d7.
That PR did not clean up the waivers it obsoleted, which were added in
post-merge 2869 (#17041):

  • perf/test_perf_sanity.py::test_e2e[aggr_upload-qwen3_5_397b_fp4_blackwell-qwen3_5_397b_fp4_dep8_8k1k]
  • perf/test_perf_sanity.py::test_e2e[aggr_upload-qwen3_5_397b_fp4_blackwell-qwen3_5_397b_fp4_dep8_mtp3_8k1k]
  • perf/test_perf_sanity.py::test_e2e[aggr_upload-qwen3_5_397b_fp4_blackwell-qwen3_5_397b_fp4_tp4_8k1k]

https://nvbugs/6490049 — verified case by case

This bug tracks an intermittent gen_only harness metric-extraction failure
(missing 'prev_device_step_time' / missing Average Per Iter Device Step Time)
across 13 cases, and is still open. Its waivers are being retired individually
as each case is verified — the same pattern as #16920 ("Unwaive GB300 Kimi
disagg e2e"
, merged 2026-08-06), which unwaived one case while the bug
remained open. This PR retires two more:

  • perf/test_perf_sanity.py::test_e2e[disagg_upload-gen_only-gb200_deepseek-v32-fp4_8k1k_con4096_ctx1_dep4_gen1_dep32_eplb256_mtp0_ccb-NIXL]
  • perf/test_perf_sanity.py::test_e2e[disagg_upload-gen_only-gb200_gpt-oss-120b-fp4_8k1k_con4_ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-NIXL]

The remaining 6490049 waiver (gb300_kimi-k25-thinking-fp4_8k1k_con1024_...)
is intentionally left in place.

Test Coverage

Waiver removal only — no source changes. All five cases remain listed in their
test-db lists, so removing the waivers re-enables them rather than leaving them
unrun:

Case Test-db list
qwen3_5_397b_fp4 l0_b200_multi_gpus_perf_sanity.yml
deepseek-v32-fp4_8k1k_con4096 l0_gb200_multi_nodes_perf_sanity_ctx1_node1_gpu4_gen1_node8_gpu32.yml
gpt-oss-120b-fp4_8k1k_con4 l0_gb200_multi_nodes_perf_sanity_ctx1_node1_gpu1_gen1_node1_gpu4.yml

PR Checklist

  • I have read the Contribution Guidelines
  • Commit is DCO signed-off
  • Each waiver removal references the fix or verification that justifies it

🤖 Generated with Claude Code

Dev Engineer Review

  • Removed three qwen3_5_397b_fp4 perf-sanity waivers from tests/integration/test_lists/waives.txt.
  • The removals match the fix for nvbug 6535767 from PR #17163.
  • Five other waivers for nvbug 6535767 remain unchanged.
  • The waiver file change is correctly scoped and contains no public API changes.

QA Engineer Review

  • No test-db/ or qa/ files were modified.
  • Removed three B200 multi-GPU perf-sanity waiver entries.
  • The three cases remain in the test database for post-merge re-enablement.
  • CBTS coverage data is unavailable.

Verdict: needs follow-up

@coderabbitai

coderabbitai Bot commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Walkthrough

The change removes three skipped performance sanity test entries for Qwen3.5 397B FP4 Blackwell configurations from the integration waiver list.

Changes

Performance waiver cleanup

Layer / File(s) Summary
Remove Qwen3.5 performance waivers
tests/integration/test_lists/waives.txt
Removes skip entries for dependency, MTP, and tensor-parallel configurations.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Possibly related PRs

Suggested labels: ci: full pre-merge approved

Suggested reviewers: qijune, xinhe-nv

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title describes two real waiver removals but omits the three Qwen3.5 waivers identified as the primary change.
Description check ✅ Passed The description includes the required sections and clearly explains the five waiver removals, justification, test coverage, and checklist.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

Removes five perf-sanity waivers from
tests/integration/test_lists/waives.txt across two bugs.

nvbugs/6535767 -- fixed by PR NVIDIA#17163 ("Drain in-flight requests before
clearing the KV cache reuse state", merged 2026-08-06 as b7a9d6f),
which resolved the "V2 Mamba state pool has only X slots" startup
failure but did not clean up the waivers it obsoleted:

- test_e2e[aggr_upload-qwen3_5_397b_fp4_blackwell-qwen3_5_397b_fp4_dep8_8k1k]
- test_e2e[aggr_upload-qwen3_5_397b_fp4_blackwell-qwen3_5_397b_fp4_dep8_mtp3_8k1k]
- test_e2e[aggr_upload-qwen3_5_397b_fp4_blackwell-qwen3_5_397b_fp4_tp4_8k1k]

nvbugs/6490049 -- verified individually and unwaived case by case, the
same pattern as PR NVIDIA#16920 ("Unwaive GB300 Kimi disagg e2e", merged
2026-08-06):

- test_e2e[disagg_upload-gen_only-gb200_deepseek-v32-fp4_8k1k_con4096_ctx1_dep4_gen1_dep32_eplb256_mtp0_ccb-NIXL]
- test_e2e[disagg_upload-gen_only-gb200_gpt-oss-120b-fp4_8k1k_con4_ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-NIXL]

All five cases remain listed in their respective test-db lists
(l0_b200_multi_gpus_perf_sanity.yml,
l0_gb200_multi_nodes_perf_sanity_ctx1_node1_gpu4_gen1_node8_gpu32.yml,
l0_gb200_multi_nodes_perf_sanity_ctx1_node1_gpu1_gen1_node1_gpu4.yml),
so removing the waivers re-enables them rather than leaving them unrun.

The remaining nvbugs/6490049 waiver (gb300 kimi-k25-thinking gen_only)
is intentionally left in place.

Signed-off-by: chenfeiz0326 <203214996+chenfeiz0326@users.noreply.github.com>
@chenfeiz0326
chenfeiz0326 force-pushed the user-chenfeiz/unwaive-6535767-perf-sanity branch from c1b6f18 to 0e8d1b9 Compare August 9, 2026 06:42
@chenfeiz0326 chenfeiz0326 changed the title [None][infra] Unwaive 3 qwen3_5_397b_fp4 perf sanity cases for nvbug 6535767 [None][infra] Unwaive 5 disagg gen only cases Aug 9, 2026
Comment thread tests/integration/test_lists/waives.txt
@chenfeiz0326

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast --stage-list "DGX_B200-8_GPUs-PyTorch-PerfSanity-Post-Merge-2,DGX_B200-8_GPUs-PyTorch-PerfSanity-Post-Merge-4,GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-5,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-3"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #64842 [ run ] triggered by Bot. Commit: 0e8d1b9 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #64842 [ run ] completed with state FAILURE. Commit: 0e8d1b9
/LLM/main/L0_MergeRequest_PR pipeline #52683 (Partly Tested) completed with status: 'UNSTABLE'

CI Report

⚠️ Multi-GPU Label Required:
Multi-GPU tests require the ci: full pre-merge approved label on this PR. Ask a member of NVIDIA/trt-llm-ci-approvers to add the label, then re-trigger CI with the same bot command (no rebase needed).

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

@chenfeiz0326
chenfeiz0326 enabled auto-merge (squash) August 10, 2026 02:45
@chenfeiz0326

Copy link
Copy Markdown
Collaborator Author

/bot skip --comment "Only unwaive perf tests, no need to run the whole CI pipeline"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #64927 [ skip ] triggered by Bot. Commit: 0e8d1b9 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #64927 [ skip ] completed with state SUCCESS. Commit: 0e8d1b9
Skipping testing for commit 0e8d1b9

Link to invocation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants