[None][infra] Unwaive 3 perf-sanity cases verified passing on ToT

[chenfeiz0326]

Summary

Three aggregated perf-sanity cases in tests/integration/test_lists/waives.txt were re-run on current ToT and all three pass end to end. Their waives are stale, so this PR removes them.

case	job	pytest	request accounting
aggr_upload-deepseek_r1_fp4_v2_2_nodes_grace_blackwell-r1_fp4_v2_dep8_mtp1_8k1k	2929320	1 passed in 1386.44s	10240 / 10240 successful, 0 failed
aggr_upload-k25_thinking_fp4_2_nodes_grace_blackwell-k25_thinking_fp4_tep8_32k8k	2929336	1 passed in 689.88s	10 / 10 successful, 0 failed (num_prompts=10, max_concurrency=2)
aggr_upload-deepseek_r1_fp4_v2_grace_blackwell-r1_fp4_v2_dep4_mtp1_1k8k	2929319	1 passed in 4879.88s	10240 / 10240 successful, 0 failed

Verification for each run, all CI-faithful (no server-readiness env override):

• pytest summary 1 passed
• report.xml: tests="1" failures="0" errors="0" skipped="0" (executed, not deselected)
• sacct: COMPLETED 0:0
• the launcher's own final line: Aggregated test completed successfully
• every MPI rank reporting exit code 0, zero Bus error occurrences
• request accounting read from <run_dir>/aggr-*/trtllm-benchmark.*.log

Bug status

• nvbugs/6422339 (cases 2 and 3) is already Dev - Closed - Verified (fixed 2026-07-20) — cleanly stale waives.
• nvbugs/6530213 (case 1) is still Dev - Open - To fix, but the case passes on ToT, so the waive is stale regardless.

Deliberately NOT unwaived

aggr_upload-ctx_only-gb300_deepseek-v4-pro-fp4_8k1k_con4301_ctx12_dep4_gen1_dep8_eplb384_mtp1_ccb-NIXL (nvbugs/6517846) stays waived. It still fails, for two independent reasons:

1. Server-readiness budget. The server needs ~2000s to answer /health (8x Cache size after warmup, autotune finishing ~35 min in), but test_perf_sanity.py clamps readiness to min(self.timeout, AGG_SERVER_READY_TIMEOUT=1800), so the per-case TIMEOUT annotation cannot help. Run 2929124: 1 failed ... in 1802.20s with 4x did not become ready within 1800 seconds.
2. Teardown SIGBUS. With TRTLLM_TEST_AGG_SERVER_READY_TIMEOUT=3600, pytest reports 1 passed ... in 7116.39s and the job still fails: all three peer mgmn_worker_node ranks hit Signal: Bus error (7) / Non-existant physical address -> exit 135, so srun --kill-on-bad-exit=1 returns non-zero and the launcher's cleanup_on_failure fails the stage (Error: Aggregated test failed).

Both belong to the existing bug, not to this PR.

Test Coverage

The three unwaived cases are re-enabled in these post-merge stages, which will run pre-merge on this PR:

• GB200-4_GPUs-PyTorch-PerfSanity-Post-Merge-* (case r1_fp4_v2_dep4_mtp1_1k8k)
• GB200-8_GPUs-2_Nodes-PyTorch-PerfSanity-Node2-GPU8-Post-Merge-1 / -3
• GB300-8_GPUs-2_Nodes-PyTorch-PerfSanity-Node2-GPU8-Post-Merge-1

Note the verification runs above were executed on GB300 (supported_gpus is advisory in perf-sanity and is never read by test_perf_sanity.py), so the GB200 stages in the list above are what confirm the GB200 hardware path.

PR Checklist

• Commit message follows the required format
• Commits are signed off (DCO)
• Test coverage described above

Dev Engineer Review

• Removed four stale waiver entries from tests/integration/test_lists/waives.txt.
• Added three GLM5 context-only waiver entries.
• Increased AGG_SERVER_READY_TIMEOUT from 1800 to 3600 seconds.
• Verified all four unwaived cases with 43,010/43,010 successful requests and no readiness timeouts, bus errors, launcher failures, or MPI rank failures.
• The waiver format and test scope require confirmation against the intended GB200 and GB300 post-merge stages.
• CI was unstable because the required ci: full pre-merge approved label was missing.

QA Engineer Review

• No test-db/ or qa/ files were modified.
• Modified test code: AGG_SERVER_READY_TIMEOUT in tests/integration/defs/perf/test_perf_sanity.py.
• No test functions were added, modified, or removed.
• All four affected perf-sanity cases are represented in tests/integration/test_lists/waives.txt.
• CBTS coverage data is unavailable.
• Verdict: needs follow-up.

[chenfeiz0326]

/bot run --disable-fail-fast --stage-list "GB200-4_GPUs-PyTorch-PerfSanity-Post-Merge-*,GB200-8_GPUs-2_Nodes-PyTorch-PerfSanity-Node2-GPU8-Post-Merge-1,GB200-8_GPUs-2_Nodes-PyTorch-PerfSanity-Node2-GPU8-Post-Merge-3,GB300-8_GPUs-2_Nodes-PyTorch-PerfSanity-Node2-GPU8-Post-Merge-1"

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 8196bda6-2e87-4bb9-bb17-9603e0e42c5b

📥 Commits

Reviewing files that changed from the base of the PR and between a073cb5 and 6c50743.

📒 Files selected for processing (1)

• tests/integration/test_lists/waives.txt

🚧 Files skipped from review as they are similar to previous changes (1)

• tests/integration/test_lists/waives.txt

---

Walkthrough

The pull request increases the aggregated-server readiness timeout from 1800 to 3600 seconds. It also removes several performance-test waivers and adds GLM5 context-only waivers.

Changes

Performance test configuration

Layer / File(s)	Summary
Extend aggregated-server readiness timeout tests/integration/defs/perf/test_perf_sanity.py	Increases AGG_SERVER_READY_TIMEOUT to 3600 seconds and updates the large-model initialization comments.
Update performance-test waiver entries tests/integration/test_lists/waives.txt	Removes DeepSeek V4 Pro FP4, DeepSeek R1 FP4 Grace Blackwell, and Kimi K2 Thinking waivers. Adds GLM5 context-only waivers and retains other existing entries.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Possibly related PRs

• NVIDIA/TensorRT-LLM#16920: Removes related Kimi Thinking performance waivers.
• NVIDIA/TensorRT-LLM#17386: Removes Kimi K2 Thinking performance waivers.
• NVIDIA/TensorRT-LLM#17444: Removes obsolete entries from the same performance-test waiver list.

Suggested labels: ci: full pre-merge approved

Suggested reviewers: brnguyen2, qijune, xinhe-nv

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately identifies the infrastructure change and unwaiving perf-sanity cases, but the case count is narrower than the current four-case changeset.
Description check	✅ Passed	The description includes the required summary, test coverage, and checklist sections with detailed verification results and rationale.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[tensorrt-cicd]

PR_Github #64938 [ run ] triggered by Bot. Commit: bc6933f Link to invocation

[tensorrt-cicd]

PR_Github #64938 [ run ] completed with state FAILURE. Commit: bc6933f
/LLM/main/L0_MergeRequest_PR pipeline #52773 (Partly Tested) completed with status: 'UNSTABLE'

CI Report

⚠️ Multi-GPU Label Required:
Multi-GPU tests require the ci: full pre-merge approved label on this PR. Ask a member of NVIDIA/trt-llm-ci-approvers to add the label, then re-trigger CI with the same bot command (no rebase needed).

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[chenfeiz0326]

Correction on the v4-pro case (not unwaived here)

My description above understated how healthy the v4-pro run actually is, so to be accurate:

Under TRTLLM_TEST_AGG_SERVER_READY_TIMEOUT=3600, aggr-*/trtllm-benchmark.0.0.log reports a fully successful benchmark:

Total requests:          43010
Successful requests:     43010
Failed requests:          0
Benchmark duration (s):   5085.05
Total Token throughput:   62530.98 tok/s

(TPOT / ITL / User throughput = 0.00 and Avg Decoded Tokens per Iter = 1.00 are expected here — the case runs random_output_len=1, so a ctx_only run emits one output token and has no inter-token phase.)

So the workload is functionally fine. Both obstacles are harness-level:

1. 1800s AGG readiness cap — deterministic failure under CI-faithful settings; the server needs ~2000s. This alone is why the case cannot be unwaived in this PR.
2. Teardown SIGBUS — strictly after serving completes and rank 0 exits 0, four peer mgmn_worker_node processes fault (Signal code: Non-existant physical address) → exit 135 → --kill-on-bad-exit=1 → cleanup_on_failure. A teardown-ordering / shm fault, not a compute failure.

Whether (2) reproduces in Jenkins is open — a shm bus error is also what a container /dev/shm / enroot difference produces, and I have not compared my container environment against a real CI stage. Flagging it so nobody treats it as confirmed CI behavior.

Neither point affects the three cases this PR unwaives; they were verified independently and CI-faithfully.

[chenfeiz0326]

v4-pro ctx_only verified passing under this PR (aws-cmh GB300, job 2944440)

Re-ran the case with test_perf_sanity.py taken verbatim from this PR — so the 3600s comes from the AGG_SERVER_READY_TIMEOUT source constant, with no TRTLLM_TEST_*_SERVER_READY_TIMEOUT env override in the launcher (verified absent). The previous SIGBUS does not reproduce.

check	result
sacct	COMPLETED 0:0, elapsed 02:03:12
pytest	1 passed, 981 deselected in 7041.40s
launcher verdict	Aggregated test completed successfully (exit 0)
report.xml	tests="1" failures="0" errors="0" skipped="0"
readiness timeouts	0 (was 4 at 1800s)
Bus error count	0 (was 7)
per-rank exit codes	Rank0 task 0, Rank0 MPI comm 0, Rank1/2/3 MGMN 0 (were 135)
request accounting	43010 / 43010 successful, 0 failed, 62639.39 tok/s

Both blockers on nvbugs/6517846 are therefore cleared:

1. Readiness (fixed by this PR). 8x Cache size after warmup and zero readiness timeouts, purely from the raised constant.
2. Teardown SIGBUS (did not recur). All peer mgmn_worker_node ranks exit 0 this time, versus four exit-135 bus errors before. Worth stating plainly that this makes the earlier crash look intermittent rather than fixed by anything here — the 3600s change does not touch MPI/shm teardown. One clean run is evidence it is not deterministic; it is not proof it can never recur. If it reappears in post-merge, it should get its own bug rather than a re-waive of this case.

This PR now unwaives all 4 cases originally in the cohort.

[chenfeiz0326]

/bot run --disable-fail-fast --stage-list "GB200-4_GPUs-PyTorch-PerfSanity-Post-Merge-,GB200-8_GPUs-2_Nodes-PyTorch-PerfSanity-Node2-GPU8-Post-Merge-1,GB200-8_GPUs-2_Nodes-PyTorch-PerfSanity-Node2-GPU8-Post-Merge-3,GB300-8_GPUs-2_Nodes-PyTorch-PerfSanity-Node2-GPU8-Post-Merge-1,GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge-"

[chenfeiz0326]

Rebased onto 10689401f1 to resolve a waives.txt conflict.

The conflict was purely positional. Three upstream commits churned the same region of waives.txt (#17458 waived 23 cases, #17418 unwaived some, #17467 waived 5 more), and one of them landed a new waive — aggr_upload-ctx_only-gb300_glm-5-fp4_8k1k_con1024_... (nvbugs/6581075) — on the line immediately following the v4-pro line this PR deletes. test_perf_sanity.py was not touched by any upstream commit in that range, so the AGG_SERVER_READY_TIMEOUT change rebased cleanly.

Resolution kept every upstream line and dropped only the four this PR intends to unwaive. Verified post-rebase: git diff origin/main is exactly 4 deletions, 0 additions in waives.txt, plus the timeout constant in test_perf_sanity.py. Specifically preserved:

• the new glm-5-fp4_..._con1024_... waive from [None][infra] Waive 23 failed cases for main in post-merge 2894 #17458
• the two sibling gb300_glm-5-fp4 waives still under nvbugs/6517846
• both disagg_upload- v4-pro waives under nvbugs/6581075 (this PR unwaives only the aggr_upload-ctx_only variant)

No re-verification run is needed: the rebase changed no test code, and the aws-cmh GB300 verification above already exercised the exact test_perf_sanity.py content now on this branch.

/bot run --disable-fail-fast --stage-list "GB200-4_GPUs-PyTorch-PerfSanity-Post-Merge-,GB200-8_GPUs-2_Nodes-PyTorch-PerfSanity-Node2-GPU8-Post-Merge-1,GB200-8_GPUs-2_Nodes-PyTorch-PerfSanity-Node2-GPU8-Post-Merge-3,GB300-8_GPUs-2_Nodes-PyTorch-PerfSanity-Node2-GPU8-Post-Merge-1,GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge-"

[chenfeiz0326]

/bot skip --comment "Only unwaive perf tests, no need to run the whole CI pipeline"

[tensorrt-cicd]

PR_Github #65021 [ skip ] triggered by Bot. Commit: 6c50743 Link to invocation

[tensorrt-cicd]

PR_Github #65021 [ skip ] completed with state SUCCESS. Commit: 6c50743
Skipping testing for commit 6c50743

Link to invocation