[None][test] Unwaive DeepSeekV3.2 nvfp4 mtp3_fp8kv_chunked test by yizhang-nv · Pull Request #15973 · NVIDIA/TensorRT-LLM

yizhang-nv · 2026-07-06T09:51:08Z

Description

Unwaives accuracy/test_llm_api_pytorch.py::TestDeepSeekV32::test_nvfp4_multi_gpus_piecewise_cuda_graph[mtp3_fp8kv_chunked] (NVBug 5989920).

Two separate defects kept this test failing; both are fixed elsewhere:

Scheduler token budget — total_num_tokens (8195) should be <= max_num_tokens (8192) under one-model MTP with chunked prefill + overlap scheduler. Fixed on main by [https://nvbugs/6427240][fix] Reserve MTP draft tokens in scheduler for one-model speculative decoding #16101.
Stale CUDA graph pool — with torch.compile enabled, Backend._graph_pool_handle is class state shared by every engine in the process. Engine teardown resets the graphs captured into that pool, dropping its use_count to zero, but the caching allocator only erases the entry once the pool has no blocks left. The next engine in the same worker process then captured into the retired handle and hit it->second->use_count > 0 INTERNAL ASSERT FAILED in beginAllocateToPool. Fixed by [https://nvbugs/6475346][fix] Avoid stale CUD… #16952, which removes the shared class handle entirely and lets each capture create its own private pool.

This PR depends on #16952 landing first. Verified on 8x B200 that with that class of fix applied, running [baseline] and [mtp3_fp8kv_chunked] back to back in one pytest session goes from 1 failed, 1 passed to 2 passed. Without it the test still fails, because the CI stage runs [baseline] first and the MPI workers are reused across both — which is also why the test passes when run standalone.

Test Coverage

accuracy/test_llm_api_pytorch.py::TestDeepSeekV32::test_nvfp4_multi_gpus_piecewise_cuda_graph[mtp3_fp8kv_chunked], re-enabled by this PR, runs in DGX_B200-8_GPUs-PyTorch-*.

Post-fix accuracy measured on 8x B200: MMLU 87.865 (reference 87.200), GSM8K 95.072 (reference 95.600).

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

coderabbitai · 2026-07-06T09:56:37Z

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

@coderabbitai resume to resume automatic reviews.
@coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

▶️ Resume reviews
🔍 Trigger review

Walkthrough

The backend now creates a fresh CUDA graph pool handle after graph teardown. Model engine cleanup invokes this rotation after clearing graph runners. One waived DeepSeek integration test entry was removed.

Changes

CUDA graph pool lifecycle

Layer / File(s)	Summary
Rotate the shared graph pool during teardown `tensorrt_llm/_torch/compilation/backend.py`, `tensorrt_llm/_torch/pyexecutor/model_engine.py`	`Backend.rotate_graph_pool_handle()` creates a new CUDA graph pool handle. `_release_cuda_graphs` invokes it after clearing captured graphs.
Remove the resolved test waiver `tests/integration/test_lists/waives.txt`	The waiver for `TestDeepSeekV32::test_nvfp4_multi_gpus_piecewise_cuda_graph[mtp3_fp8kv_chunked]` was removed.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

NVIDIA/TensorRT-LLM#17313: Both changes update CUDA graph pool handle lifecycle management.
NVIDIA/TensorRT-LLM#17329: Both changes modify integration-test waiver entries.

Suggested reviewers: schetlur-nv, mlefeb01, qijune

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly identifies the test change and follows the required ticket and type format.
Description check	✅ Passed	The description explains the issue, dependent fixes, test coverage, results, and checklist status in sufficient detail.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

yizhang-nv · 2026-07-07T03:13:18Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-07T03:19:15Z

PR_Github #57916 [ run ] triggered by Bot. Commit: de560e6 Link to invocation

tensorrt-cicd · 2026-07-07T08:59:16Z

PR_Github #57916 [ run ] completed with state FAILURE. Commit: de560e6
/LLM/main/L0_MergeRequest_PR pipeline #46608 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

yizhang-nv · 2026-07-08T05:31:45Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-08T05:37:24Z

PR_Github #58156 [ run ] triggered by Bot. Commit: e34fe34 Link to invocation

yizhang-nv · 2026-07-08T05:38:06Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-08T05:44:13Z

PR_Github #58158 [ run ] triggered by Bot. Commit: b71f9f0 Link to invocation

tensorrt-cicd · 2026-07-08T05:47:44Z

PR_Github #58156 [ run ] completed with state ABORTED. Commit: e34fe34

Link to invocation

tensorrt-cicd · 2026-07-08T11:08:30Z

PR_Github #58158 [ run ] completed with state FAILURE. Commit: b71f9f0
/LLM/main/L0_MergeRequest_PR pipeline #46809 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

yizhang-nv · 2026-07-09T03:40:05Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-09T03:46:22Z

PR_Github #58362 [ run ] triggered by Bot. Commit: b71f9f0 Link to invocation

tensorrt-cicd · 2026-07-09T05:36:18Z

PR_Github #58362 [ run ] completed with state SUCCESS. Commit: b71f9f0
/LLM/main/L0_MergeRequest_PR pipeline #46988 completed with status: 'SUCCESS'

CI Report

Link to invocation

…upstream PR NVIDIA#15973) Squash of upstream PR NVIDIA#15973 (4 commits). One-engine speculative decoders (MTP / DSpark) keep the draft model inside ModelEngine, so PyExecutor.drafter is None; under the overlap scheduler the next draft tokens are produced by the previous batch and their count may not be copied to the C++ request before the next scheduling pass, so the scheduler can budget a gen request as 1 token though the model consumes 1+K -> oversized mixed context/generation batch (total_num_tokens > max_num_tokens). Run scheduler-facing draft-token reservation for every speculative decoder immediately before scheduling (_prepare_scheduler_draft_tokens); placeholder token values are not consumed, their length reserves the correct micro-batch capacity. Includes: skip when spec decode is disabled, handle lightweight executor fixtures, and unwaive the DeepSeekV3 nvfp4 mtp3_fp8kv_chunked test that exposed the bug. Signed-off-by: chungen28 <chung-en@deepinfra.com>

coderabbitai · 2026-08-05T06:38:47Z

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

yizhang-nv · 2026-08-05T06:41:15Z

/bot run --disable-fail-fast --stage-list "DGX_B200-8_GPUs-PyTorch-1, DGX_B200-8_GPUs-PyTorch-2, DGX_B200-8_GPUs-PyTorch-3, DGX_B200-8_GPUs-PyTorch-4"

yizhang-nv · 2026-08-05T13:07:06Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-08-05T13:11:45Z

PR_Github #64056 [ run ] triggered by Bot. Commit: c4a5959 Link to invocation

tensorrt-cicd · 2026-08-05T13:12:20Z

PR_Github #64057 [ run ] triggered by Bot. Commit: c4a5959 Link to invocation

tensorrt-cicd · 2026-08-05T13:12:22Z

PR_Github #64056 [ run ] completed with state ABORTED. Commit: c4a5959

Link to invocation

tensorrt-cicd · 2026-08-05T16:49:17Z

PR_Github #64057 [ run ] completed with state FAILURE. Commit: c4a5959
/LLM/main/L0_MergeRequest_PR pipeline #51986 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

yizhang-nv · 2026-08-06T02:18:33Z

/bot run --disable-fail-fast

coderabbitai · 2026-08-06T02:18:45Z

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

tensorrt-cicd · 2026-08-06T02:24:05Z

PR_Github #64174 [ run ] triggered by Bot. Commit: 4faea3e Link to invocation

tensorrt-cicd · 2026-08-06T07:39:58Z

PR_Github #64174 [ run ] completed with state FAILURE. Commit: 4faea3e
/LLM/main/L0_MergeRequest_PR pipeline #52087 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

yizhang-nv · 2026-08-06T07:41:34Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-08-06T07:47:08Z

PR_Github #64267 [ run ] triggered by Bot. Commit: 4faea3e Link to invocation

yizhang-nv · 2026-08-06T07:58:15Z

/bot kill

tensorrt-cicd · 2026-08-06T08:04:33Z

PR_Github #64274 [ kill ] triggered by Bot. Commit: fdebd71 Link to invocation

tensorrt-cicd · 2026-08-06T08:08:19Z

PR_Github #64267 [ run ] completed with state ABORTED. Commit: 4faea3e

Link to invocation

tensorrt-cicd · 2026-08-06T08:08:26Z

PR_Github #64274 [ kill ] completed with state SUCCESS. Commit: fdebd71
Successfully killed previous jobs for commit fdebd71

Link to invocation

Unwaive TestDeepSeekV32::test_nvfp4_multi_gpus_piecewise_cuda_graph[mtp3_fp8kv_chunked] (nvbugs/5989920) to re-enable it in CI. Signed-off-by: Yi Zhang <187001205+yizhang-nv@users.noreply.github.com>

yizhang-nv · 2026-08-11T06:56:50Z

/bot run --disable-fail-fast

yizhang-nv · 2026-08-11T09:56:58Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-08-11T10:02:58Z

PR_Github #65267 [ run ] triggered by Bot. Commit: ee0455d Link to invocation

tensorrt-cicd · 2026-08-11T11:23:42Z

PR_Github #65267 [ run ] completed with state SUCCESS. Commit: ee0455d
/LLM/main/L0_MergeRequest_PR pipeline #53044 completed with status: 'SUCCESS'

CI Report

Link to invocation

yizhang-nv requested a review from a team as a code owner July 6, 2026 09:51

yizhang-nv requested a review from achartier July 6, 2026 09:51

github-actions Bot assigned yizhang-nv Jul 6, 2026

achartier approved these changes Jul 7, 2026

View reviewed changes

Comment thread tensorrt_llm/_torch/pyexecutor/py_executor.py Outdated

yizhang-nv force-pushed the codex/fix-one-engine-spec-scheduler-budget branch from b71f9f0 to 1639857 Compare August 5, 2026 06:38

yizhang-nv requested review from a team as code owners August 5, 2026 06:38

yizhang-nv requested review from QiJune, StanleySun639, mlefeb01, mzweilz and schetlur-nv August 5, 2026 06:38

yizhang-nv changed the title ~~[None][fix] Reserve one-engine speculative tokens before scheduling~~ [None][test] Unwaive DeepSeekV3.2 nvfp4 mtp3_fp8kv_chunked test Aug 5, 2026

yizhang-nv force-pushed the codex/fix-one-engine-spec-scheduler-budget branch from c4a5959 to 4faea3e Compare August 6, 2026 02:18

liji-nv reviewed Aug 6, 2026

View reviewed changes

Comment thread tensorrt_llm/_torch/compilation/backend.py Outdated

yizhang-nv force-pushed the codex/fix-one-engine-spec-scheduler-budget branch from 4faea3e to fdebd71 Compare August 6, 2026 07:58

yizhang-nv changed the title ~~[None][fix] Retire the shared CUDA graph pool handle on engine teardown~~ [None][test] Unwaive DeepSeekV3.2 nvfp4 mtp3_fp8kv_chunked test Aug 6, 2026

StanleySun639 approved these changes Aug 6, 2026

View reviewed changes

[None][test] Unwaive DeepSeekV3 nvfp4 mtp3_fp8kv_chunked test

ee0455d

Unwaive TestDeepSeekV32::test_nvfp4_multi_gpus_piecewise_cuda_graph[mtp3_fp8kv_chunked] (nvbugs/5989920) to re-enable it in CI. Signed-off-by: Yi Zhang <187001205+yizhang-nv@users.noreply.github.com>

yizhang-nv force-pushed the codex/fix-one-engine-spec-scheduler-budget branch from fdebd71 to ee0455d Compare August 11, 2026 06:56

yizhang-nv disabled auto-merge August 11, 2026 09:56

yizhang-nv enabled auto-merge (squash) August 11, 2026 09:56

yizhang-nv merged commit 26842d1 into NVIDIA:main Aug 11, 2026
8 checks passed

coderabbitai Bot mentioned this pull request Aug 11, 2026

[https://nvbugs/6546909][fix] Make the handle's lifetime match its graphs — lazy… #17313

Open

2 tasks

Uh oh!

Conversation

yizhang-nv commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Test Coverage

PR Checklist

GitHub Bot Help

Uh oh!

coderabbitai Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviews paused

Walkthrough

Changes

Uh oh!

yizhang-nv commented Jul 7, 2026

Uh oh!

tensorrt-cicd commented Jul 7, 2026

Uh oh!

Uh oh!

tensorrt-cicd commented Jul 7, 2026

Uh oh!

yizhang-nv commented Jul 8, 2026

Uh oh!

tensorrt-cicd commented Jul 8, 2026

Uh oh!

yizhang-nv commented Jul 8, 2026

Uh oh!

tensorrt-cicd commented Jul 8, 2026

Uh oh!

tensorrt-cicd commented Jul 8, 2026

Uh oh!

tensorrt-cicd commented Jul 8, 2026

Uh oh!

yizhang-nv commented Jul 9, 2026

Uh oh!

tensorrt-cicd commented Jul 9, 2026

Uh oh!

tensorrt-cicd commented Jul 9, 2026

Uh oh!

coderabbitai Bot commented Aug 5, 2026

Uh oh!

yizhang-nv commented Aug 5, 2026

Uh oh!

yizhang-nv commented Aug 5, 2026

Uh oh!

tensorrt-cicd commented Aug 5, 2026

Uh oh!

tensorrt-cicd commented Aug 5, 2026

Uh oh!

tensorrt-cicd commented Aug 5, 2026

Uh oh!

tensorrt-cicd commented Aug 5, 2026

Uh oh!

yizhang-nv commented Aug 6, 2026

Uh oh!

coderabbitai Bot commented Aug 6, 2026

Uh oh!

tensorrt-cicd commented Aug 6, 2026

Uh oh!

Uh oh!

tensorrt-cicd commented Aug 6, 2026

Uh oh!

yizhang-nv commented Aug 6, 2026

Uh oh!

tensorrt-cicd commented Aug 6, 2026

Uh oh!

yizhang-nv commented Aug 6, 2026

Uh oh!

tensorrt-cicd commented Aug 6, 2026

Uh oh!

tensorrt-cicd commented Aug 6, 2026

Uh oh!

tensorrt-cicd commented Aug 6, 2026

Uh oh!

yizhang-nv commented Aug 11, 2026

Uh oh!

yizhang-nv commented Aug 11, 2026

Uh oh!

tensorrt-cicd commented Aug 11, 2026

yizhang-nv commented Jul 6, 2026 •

edited

Loading

coderabbitai Bot commented Jul 6, 2026 •

edited

Loading