[None][infra] Waive 23 failed cases for main in post-merge 2894 by trtllm-agent · Pull Request #17458 · NVIDIA/TensorRT-LLM

trtllm-agent · 2026-08-10T03:05:43Z

Auto-generated Waive PR

Created by: TensorRT LLM CI Report (requested by @mzweilz)
Target branch: main
Bug(s): 6572800, 6581047, 6581048, 6581049, 6581063, 6581065, 6581066, 6581067, 6581071, 6581073, 6581075

Waive entries added

accuracy/test_llm_api_pytorch.py::TestDeepSeekV4ProDSpark::test_gsm8k_dep8_megamoe_deepgemm SKIP (https://nvbugs/6581063)
accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_bfloat16_4gpus[tp4-attn_backend=TRTLLM-torch_compile=True] SKIP (https://nvbugs/6581073)
accuracy/test_llm_api_pytorch.py::TestNemotronV3Super::test_nvfp4_8gpus_mtp SKIP (https://nvbugs/6581065)
accuracy/test_llm_api_pytorch_multimodal.py::TestNanoV3Omni::test_auto_dtype[fp8_mmmu_encoder_cuda_graph] SKIP (https://nvbugs/6581049)
examples/visual_gen/test_visual_gen_wan.py::test_fastwan_lpips_against_golden SKIP (https://nvbugs/6572800)
perf/test_perf_sanity.py::test_e2e[aggr_upload-ctx_only-gb300_glm-5-fp4_8k1k_con1024_ctx1_dep2_gen1_dep8_eplb256_mtp1_ccb-NIXL] SKIP (https://nvbugs/6581075)
perf/test_perf_sanity.py::test_e2e[disagg_upload-e2e-gb300_deepseek-v4-pro-fp4_8k1k_con4301_ctx12_dep4_gen1_dep8_eplb384_mtp1_ccb-NIXL] SKIP (https://nvbugs/6581075)
perf/test_perf_sanity.py::test_e2e[disagg_upload-e2e-gb300_deepseek-v4-pro-fp4_8k1k_con666_ctx6_dep4_gen1_dep16_eplb384_mtp3_ccb-NIXL] SKIP (https://nvbugs/6581075)
perf/test_perf_sanity.py::test_e2e[disagg_upload-gen_only-gb200_gpt-oss-120b-fp4_8k1k_con1024_ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-NIXL] SKIP (https://nvbugs/6581075)
perf/test_perf_sanity.py::test_e2e[disagg_upload-gen_only-gb200_kimi-k25-thinking-fp4_8k1k_con1024_ctx1_dep4_gen1_dep32_eplb416_mtp3_ccb-NIXL] SKIP (https://nvbugs/6581075)
perf/test_perf_sanity.py::test_e2e[disagg_upload-gen_only-gb300_deepseek-r1-fp4_128k8k_con256_ctx1_pp4_gen1_dep8_eplb0_mtp1_ccb-NIXL] SKIP (https://nvbugs/6581075)
perf/test_perf_sanity.py::test_e2e[disagg_upload-gen_only-gb300_deepseek-v4-pro-fp4_8k1k_con4301_ctx12_dep4_gen1_dep8_eplb384_mtp1_ccb-NIXL] SKIP (https://nvbugs/6581075)
perf/test_perf_sanity.py::test_e2e[disagg_upload-gen_only-gb300_deepseek-v4-pro-fp4_8k1k_con666_ctx6_dep4_gen1_dep16_eplb384_mtp3_ccb-NIXL] SKIP (https://nvbugs/6581075)
perf/test_perf_sanity.py::test_e2e[disagg_upload-gen_only-gb300_glm-5-fp4_8k1k_con1024_ctx1_dep2_gen1_dep8_eplb256_mtp1_ccb-NIXL] SKIP (https://nvbugs/6581075)
perf/test_perf_sanity.py::test_e2e[disagg_upload-gen_only-gb300_glm-5-fp4_8k1k_con1_ctx1_dep2_gen1_tep8_eplb0_mtp3_ccb-NIXL] SKIP (https://nvbugs/6581075)
perf/test_perf_sanity.py::test_e2e[disagg_upload-gen_only-gb300_glm-5-fp4_8k1k_con512_ctx1_dep2_gen1_dep32_eplb0_mtp3_ccb-NIXL] SKIP (https://nvbugs/6581075)
unittest/_torch/modules/test_w4a16_nvfp4_linear.py::test_nvfp4_attention_keeps_high_precision_output_for_hopper_marlin SKIP (https://nvbugs/6581071)
unittest/_torch/sampler -k "not test_speculative_d2h_parity_real_predictor" SKIP (https://nvbugs/6581047)
unittest/_torch/sampler/test_beam_search.py::test_beam_search_e2e[multi_process-TorchSampler-no_cuda_graph_and_overlap-stop_token_ids0-1-1-True-True-True] SKIP (https://nvbugs/6581048)
unittest/_torch/thop/parallel/test_fp4_linear.py::test_fp4_gemm_bias_per_backend[mnk2-cublaslt] SKIP (https://nvbugs/6581067)
unittest/_torch/thop/parallel/test_fp4_linear.py::test_fp4_gemm_bias_per_backend[mnk2-cutlass] SKIP (https://nvbugs/6581067)
unittest/_torch/thop/parallel/test_fp4_linear.py::test_fp4_gemm_bias_per_backend[mnk3-cublaslt] SKIP (https://nvbugs/6581067)
unittest/_torch/thop/parallel/test_fp4_linear.py::test_fp4_gemm_bias_per_backend[mnk3-cutlass] SKIP (https://nvbugs/6581067)

Already waived (skipped)

accuracy/test_llm_api_pytorch.py::TestNemotronV3Ultra::test_nvfp4_8gpus[attention_dp_off-trtllm]

This PR was auto-generated by TensorRT LLM CI Report. Please review the waive entries before merging.

Dev Engineer Review

Added 23 SKIP entries to tests/integration/test_lists/waives.txt.
Entries cover DeepSeek, Llama, Nemotron, multimodal, visual-generation, performance, sampler, FP4 linear, and unit tests.
Entries reference 11 NVIDIA bugs.
No code or public API changes are included.
Review should confirm valid test paths and absence of duplicate waivers.

QA Engineer Review

Modified file: tests/integration/test_lists/waives.txt.
Added 23 waiver entries. No entries were removed.
No test-db/ or qa/ files were modified.
CBTS coverage data is unavailable.
Verdict: needs follow-up.

@mzweilz

Bug(s): 6572800, 6581047, 6581048, 6581049, 6581063, 6581065, 6581066, 6581067, 6581071, 6581073, 6581075 Requested by: @mzweilz Signed-off-by: trtllm-agent <296075020+trtllm-agent@users.noreply.github.com>

mzweilz · 2026-08-10T03:06:45Z

/bot skip --comment "waive only"

coderabbitai · 2026-08-10T03:09:14Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: e40f1c31-a74c-4ee4-b55c-20897798be48

📥 Commits

Reviewing files that changed from the base of the PR and between a84d6ef and 96253c8.

📒 Files selected for processing (1)

tests/integration/test_lists/waives.txt

💤 Files with no reviewable changes (1)

tests/integration/test_lists/waives.txt

Walkthrough

The integration waiver list adds 23 skip entries for newly affected model, multimodal, performance, sampler, attention precision, and FP4 GEMM tests.

Changes

Integration test waivers

Layer / File(s)	Summary
Model and multimodal test waivers `tests/integration/test_lists/waives.txt`	Adds skip entries for DeepSeek, Llama, Nemotron, NanoV3 Omni, and FastWan tests.
Performance and sampler test waivers `tests/integration/test_lists/waives.txt`	Adds waivers for GLM-5, DeepSeek, GPT-OSS, Kimi K2, Marlin NVFP4, sampler, and FP4 GEMM tests.

Estimated code review effort: 1 (Trivial) | ~3 minutes

Possibly related PRs

NVIDIA/TensorRT-LLM#17384: Adds different skip waivers to the same integration waiver list.
NVIDIA/TensorRT-LLM#17407: Updates overlapping visual-generation and performance waivers.
NVIDIA/TensorRT-LLM#17442: Updates performance-test waiver entries.

Suggested reviewers: qijune, schetlur-nv

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies an infrastructure change that waives 23 failed cases for the main branch.
Description check	✅ Passed	The description explains the automated waiver purpose, affected tests, bug links, target branch, and existing waiver; formal template sections are not used.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/integration/test_lists/waives.txt`:
- Line 58: Add the repository-standard NVIDIA copyright header at the top of the
modified waives.txt file, using 2026 as the latest meaningful modification year;
leave the existing test entry unchanged.
- Line 58: Add the NVIDIA copyright header to
tests/integration/test_lists/waives.txt and align its waiver entries with the PR
objectives by adding bug 6581066 if it remains in scope; otherwise remove that
objective. Preserve existing waiver entries unless they are part of this scope
adjustment.
- Line 409: Update the sampler waiver entry to target only the failing test by
replacing the broad unittest path and exclusion expression with the fully
qualified pytest node ID for test_speculative_d2h_parity_real_predictor.
Preserve the existing nvbugs reference.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 1b341110-9cd4-45ff-953d-28ff9328db26

📥 Commits

Reviewing files that changed from the base of the PR and between 5761d00 and a84d6ef.

📒 Files selected for processing (1)

tests/integration/test_lists/waives.txt

tensorrt-cicd · 2026-08-10T03:12:34Z

PR_Github #64936 [ skip ] triggered by Bot. Commit: a84d6ef Link to invocation

tensorrt-cicd · 2026-08-10T03:21:18Z

PR_Github #64936 [ skip ] completed with state SUCCESS. Commit: a84d6ef
Skipping testing for commit a84d6ef

Link to invocation

Signed-off-by: Abby Wei <18545893+mzweilz@users.noreply.github.com>

mzweilz · 2026-08-10T03:50:09Z

/bot skip --comment "waive only"

tensorrt-cicd · 2026-08-10T03:55:48Z

PR_Github #64945 [ skip ] triggered by Bot. Commit: 96253c8 Link to invocation

tensorrt-cicd · 2026-08-10T04:02:28Z

PR_Github #64945 [ skip ] completed with state SUCCESS. Commit: 96253c8
Skipping testing for commit 96253c8

Link to invocation

[None][infra] Waive 23 failed cases for main in post-merge 2894

a84d6ef

Bug(s): 6572800, 6581047, 6581048, 6581049, 6581063, 6581065, 6581066, 6581067, 6581071, 6581073, 6581075 Requested by: @mzweilz Signed-off-by: trtllm-agent <296075020+trtllm-agent@users.noreply.github.com>

trtllm-agent requested review from a team as code owners August 10, 2026 03:05

trtllm-agent requested review from QiJune, StanleySun639, mlefeb01, mzweilz and schetlur-nv August 10, 2026 03:05

mzweilz approved these changes Aug 10, 2026

View reviewed changes

github-actions Bot assigned trtllm-agent Aug 10, 2026

mzweilz enabled auto-merge (squash) August 10, 2026 03:07

coderabbitai Bot reviewed Aug 10, 2026

View reviewed changes

Comment thread tests/integration/test_lists/waives.txt

Comment thread tests/integration/test_lists/waives.txt Outdated

mzweilz disabled auto-merge August 10, 2026 03:48

Update waives.txt

96253c8

Signed-off-by: Abby Wei <18545893+mzweilz@users.noreply.github.com>

mzweilz enabled auto-merge (squash) August 10, 2026 03:49

mzweilz merged commit fa73933 into NVIDIA:main Aug 10, 2026
10 of 11 checks passed

This was referenced Aug 10, 2026

[None][test] Waive 5 failed cases for main in QA CI #17467

Merged

[None][test] Waive 2 failed cases for main in QA CI #17468

Open

[None][test] Remove 85 closed-bug waive entries for main #17444

Open

chenfeiz0326 mentioned this pull request Aug 10, 2026

[https://nvbugs/6517846][fix] Raise AGG server-ready timeout to 3600s and unwaive 4 perf-sanity cases #17459

Merged

3 tasks

Navigation Menu

Uh oh!

[None][infra] Waive 23 failed cases for main in post-merge 2894 - #17458

[None][infra] Waive 23 failed cases for main in post-merge 2894#17458
mzweilz merged 2 commits into
NVIDIA:mainfrom
trtllm-agent:trtllm-ci-report/waive-20260810-030541

trtllm-agent commented Aug 10, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

mzweilz commented Aug 10, 2026

Uh oh!

coderabbitai Bot commented Aug 10, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

Uh oh!

tensorrt-cicd commented Aug 10, 2026

Uh oh!

tensorrt-cicd commented Aug 10, 2026

Uh oh!

mzweilz commented Aug 10, 2026

Uh oh!

tensorrt-cicd commented Aug 10, 2026

Uh oh!

tensorrt-cicd commented Aug 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

trtllm-agent commented Aug 10, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Auto-generated Waive PR

Waive entries added

Already waived (skipped)

Uh oh!

mzweilz commented Aug 10, 2026

Uh oh!

coderabbitai Bot commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

tensorrt-cicd commented Aug 10, 2026

Uh oh!

tensorrt-cicd commented Aug 10, 2026

Uh oh!

mzweilz commented Aug 10, 2026

Uh oh!

tensorrt-cicd commented Aug 10, 2026

Uh oh!

tensorrt-cicd commented Aug 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

trtllm-agent commented Aug 10, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Aug 10, 2026 •

edited

Loading