Skip to content

[NV] Add GB300 Qwen3.5 NVFP4 Dynamo-TensorRT-LLM disaggregated benchmarks / [NVIDIA] 添加 GB300 Qwen3.5 NVFP4 Dynamo-TensorRT-LLM 分离式基准测试 - #2206

Merged
adibarra merged 10 commits into
mainfrom
dynamo-trtllm-gb300-qwen3.5-nvfp4
Aug 10, 2026
Merged

[NV] Add GB300 Qwen3.5 NVFP4 Dynamo-TensorRT-LLM disaggregated benchmarks / [NVIDIA] 添加 GB300 Qwen3.5 NVFP4 Dynamo-TensorRT-LLM 分离式基准测试#2206
adibarra merged 10 commits into
mainfrom
dynamo-trtllm-gb300-qwen3.5-nvfp4

Conversation

@richardhuo-nv

@richardhuo-nv richardhuo-nv commented Jul 14, 2026

Copy link
Copy Markdown
Collaborator

Summary

Add Qwen3.5-397B-A17B-NVFP4 GB300 disaggregated TensorRT-LLM + Dynamo multi-node benchmarks (STP, no MTP) for both 1k1k and 8k1k scenarios.

  • 18 srt-slurm recipe configs checked into benchmarks/multi_node/srt-slurm-recipes/trtllm/qwen3.5/gb300-fp4/disagg/ (9 per scenario), sourced from validated experiment runs
  • Master config entry qwen3.5-fp4-gb300-dynamo-trt covering the full topology sweep across both ISL/OSL scenarios
  • Runner launch_gb300-nv.sh extended to check out NVIDIA/srt-slurm@v1.0.29 and overlay the local TensorRT-LLM recipes for the dynamo-trt + qwen3.5 + fp4 path
  • Perf changelog entry to trigger the full GB300 sweep

Config details

Image: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc18
Dynamo: installed at runtime (version: 1.3.0.dev20260708, TCP request plane)
Model checkpoint: /scratch/models/Qwen3.5-397B-A17B-NVFP4 (node-local NVMe)
Model alias: qwen3.5-fp4 (matches model.path in all recipes)

All prefill workers use TP2/EP2 with attention DP (DEP2, 2 GPUs/worker). Decode topology varies by concurrency point:

1k1k (ISL=1024, OSL=1024) — 9 configs

Config Prefill Decode Conc
ctx1dep2_gen17dep4_mtp0_c68 1×DEP2 17×TP4/EP4 68
ctx1dep2_gen17dep4_mtp0_c136 1×DEP2 17×TP4/EP1 136
ctx1dep2_gen17dep4_mtp0_c272 1×DEP2 17×TP4/EP1 272
ctx1dep2_gen4dep16_mtp0_c16 1×DEP2 4×TP16/EP16 16
ctx1dep2_gen5dep8_mtp0_c640 1×DEP2 5×TP8/EP8 640
ctx1dep2_gen1dep16_mtp0_c1280 1×DEP2 1×DEP16 1280
ctx2dep2_gen3dep16_mtp0_c1536 2×DEP2 3×DEP16 1536
ctx5dep2_gen3dep16_mtp0_c7680 5×DEP2 3×DEP16 7680
ctx5dep2_gen7dep4_mtp0_c17920 5×DEP2 7×DEP4 17920

8k1k (ISL=8192, OSL=1024) — 9 configs

Config Prefill Decode Conc
ctx1dep2_gen13tp4_mtp0_c78 1×DEP2 13×TP4/EP1 78
ctx1dep2_gen4tep16_mtp0_c24 1×DEP2 4×TP16/EP16 24
ctx1dep2_gen8tep8_mtp0_c48 1×DEP2 8×TP8/EP8 48
ctx2dep2_gen15tp4_mtp0_c180 2×DEP2 15×TP4/EP1 180
ctx2dep2_gen9tp4_mtp0_c216 2×DEP2 9×TP4/EP1 216
ctx3dep2_gen1dep4_mtp0_c1536 3×DEP2 1×TP4/EP4 1536
ctx7dep2_gen12tep4_mtp0_c1152 7×DEP2 12×TP4/EP4 1152
ctx8dep2_gen9tep4_mtp0_c1728 8×DEP2 9×TP4/EP4 1728
ctx10dep2_gen3dep8_mtp0_c4608 10×DEP2 3×TP8/EP8 4608

Files changed

  • benchmarks/multi_node/srt-slurm-recipes/trtllm/qwen3.5/gb300-fp4/disagg/1k1k/stp/ — 9 new recipe YAMLs
  • benchmarks/multi_node/srt-slurm-recipes/trtllm/qwen3.5/gb300-fp4/disagg/8k1k/stp/ — 9 new recipe YAMLs
  • configs/nvidia-master.yaml — new qwen3.5-fp4-gb300-dynamo-trt entry (18 search-space points)
  • runners/launch_gb300-nv.sh — new dynamo-trt + qwen3.5 + fp4 branch that overlays local recipes
  • perf-changelog.yaml — new entry to trigger the GB300 sweep

中文说明

概要

新增 Qwen3.5-397B-A17B-NVFP4 GB300 TensorRT-LLM + Dynamo 分离式多节点基准测试,覆盖 1k1k8k1k 两种场景,采用 STP(不启用 MTP)。

  • benchmarks/multi_node/srt-slurm-recipes/trtllm/qwen3.5/gb300-fp4/disagg/ 中新增 18 个 srt-slurm 配置(每种场景 9 个),均来自已验证的实验运行
  • 新增主配置项 qwen3.5-fp4-gb300-dynamo-trt,覆盖两种 ISL/OSL 场景下的完整拓扑扫描
  • 扩展启动器 launch_gb300-nv.sh:针对 dynamo-trt + qwen3.5 + fp4 路径检出 NVIDIA/srt-slurm@v1.0.29,并覆盖仓库内的 TensorRT-LLM 配置
  • 新增性能变更日志条目,用于触发完整的 GB300 扫描

配置详情

镜像: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc18
Dynamo: 运行时安装(version: 1.3.0.dev20260708,使用 TCP request plane)
模型检查点: /scratch/models/Qwen3.5-397B-A17B-NVFP4(节点本地 NVMe)
模型别名: qwen3.5-fp4(与所有配置中的 model.path 一致)

所有预填充工作进程均使用 TP2/EP2 并启用注意力数据并行(DEP2,每个工作进程使用 2 张 GPU)。解码拓扑随并发点变化:

1k1k(ISL=1024,OSL=1024)— 9 个配置

配置 预填充 解码 并发数
ctx1dep2_gen17dep4_mtp0_c68 1×DEP2 17×TP4/EP4 68
ctx1dep2_gen17dep4_mtp0_c136 1×DEP2 17×TP4/EP1 136
ctx1dep2_gen17dep4_mtp0_c272 1×DEP2 17×TP4/EP1 272
ctx1dep2_gen4dep16_mtp0_c16 1×DEP2 4×TP16/EP16 16
ctx1dep2_gen5dep8_mtp0_c640 1×DEP2 5×TP8/EP8 640
ctx1dep2_gen1dep16_mtp0_c1280 1×DEP2 1×DEP16 1280
ctx2dep2_gen3dep16_mtp0_c1536 2×DEP2 3×DEP16 1536
ctx5dep2_gen3dep16_mtp0_c7680 5×DEP2 3×DEP16 7680
ctx5dep2_gen7dep4_mtp0_c17920 5×DEP2 7×DEP4 17920

8k1k(ISL=8192,OSL=1024)— 9 个配置

配置 预填充 解码 并发数
ctx1dep2_gen13tp4_mtp0_c78 1×DEP2 13×TP4/EP1 78
ctx1dep2_gen4tep16_mtp0_c24 1×DEP2 4×TP16/EP16 24
ctx1dep2_gen8tep8_mtp0_c48 1×DEP2 8×TP8/EP8 48
ctx2dep2_gen15tp4_mtp0_c180 2×DEP2 15×TP4/EP1 180
ctx2dep2_gen9tp4_mtp0_c216 2×DEP2 9×TP4/EP1 216
ctx3dep2_gen1dep4_mtp0_c1536 3×DEP2 1×TP4/EP4 1536
ctx7dep2_gen12tep4_mtp0_c1152 7×DEP2 12×TP4/EP4 1152
ctx8dep2_gen9tep4_mtp0_c1728 8×DEP2 9×TP4/EP4 1728
ctx10dep2_gen3dep8_mtp0_c4608 10×DEP2 3×TP8/EP8 4608

变更文件

  • benchmarks/multi_node/srt-slurm-recipes/trtllm/qwen3.5/gb300-fp4/disagg/1k1k/stp/ — 新增 9 个配置 YAML
  • benchmarks/multi_node/srt-slurm-recipes/trtllm/qwen3.5/gb300-fp4/disagg/8k1k/stp/ — 新增 9 个配置 YAML
  • configs/nvidia-master.yaml — 新增 qwen3.5-fp4-gb300-dynamo-trt 配置项(18 个搜索空间点)
  • runners/launch_gb300-nv.sh — 新增 dynamo-trt + qwen3.5 + fp4 分支,用于覆盖仓库内配置
  • perf-changelog.yaml — 新增触发 GB300 扫描的条目

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional findings (outside current diff — PR may have been updated during review):

  • 🔴 perf-changelog.yaml:4355 — The new entry's pr-link is https://github.com/SemiAnalysisAI/InferenceX/pull/{TBD}, but the changelog merge tooling only accepts a real pull/<PR> link or the literal placeholder XXX; {TBD} matches neither. This won't fail the PR-time CI gate, but it will raise a ChangelogValidationError when utils/merge_with_reuse.sh runs prepare_perf_changelog_merge.py canonicalize at merge time, blocking the standard merge path until fixed. Fix: replace {TBD} with XXX (or the literal pull/2206).

    Extended reasoning...

    The newly appended perf-changelog.yaml entry (line 4355) sets:

    pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/{TBD}
    

    The accepted values for an appended entry's pr-link are the canonical pull/<PR-number> URL, or one of the literal placeholders in PR_LINK_PLACEHOLDERS ('XXX' or 'https://github.com/SemiAnalysisAI/InferenceX/pull/XXX', defined in utils/validate_perf_changelog.py lines 24-27). The string {TBD} is neither of these, so it will be rejected by validate_added_pr_link().

    Where it actually fires: it's worth correcting where this gets caught, since it's not where it would naively be assumed. The PR-time CI gate (run-sweep.yml's "Validate perf-changelog matrix" step) invokes validate_perf_changelog.py's main(), which only calls validate_matrix_compatible_changevalidate_generated_config (process_changelog.py). That code path never inspects pr-link at all — ChangelogEntry.pr_link (utils/matrix_logic/validation.py:497) is a plain str field with no format validator — so {TBD} sails through the PR CI check without complaint.

    The rejection instead happens at merge time. utils/merge_with_reuse.sh (the documented, required merge entrypoint per README.md and the merge-prs skill) unconditionally runs prepare_perf_changelog_merge.py canonicalize --pr-number <PR> as part of the merge flow. That script calls canonicalize_appended_links()compare_entries()validate_added_pr_link(link, pr_number). With pr_number supplied, the expected value is the real pull/2206 link, and the only alternative accepted is the XXX placeholder (meant to be auto-substituted with the real PR number during canonicalization). {TBD} matches neither, so validate_added_pr_link raises ChangelogValidationError('new PR entry must use ... or an XXX placeholder; found ...{TBD}'), and the merge script (running under set -euo pipefail) hard-fails.

    Step-by-step proof:

    1. PR #2206 is opened with perf-changelog.yaml's new entry containing pr-link: .../pull/{TBD}.
    2. The PR-time CI check (validate_perf_changelog.py main()) runs and passes, since it never looks at pr-link.
    3. A maintainer runs the standard merge flow: utils/merge_with_reuse.sh <PR>.
    4. That script invokes prepare_perf_changelog_merge.py canonicalize --pr-number 2206.
    5. canonicalize_appended_links iterates the appended changelog entries, calling compare_entries(base, head, 2206) for each.
    6. For this entry, link = 'https://github.com/SemiAnalysisAI/InferenceX/pull/{TBD}'. It is compared against the expected canonical link (pull/2206) and against PR_LINK_PLACEHOLDERS = {'XXX', '.../pull/XXX'}. It matches neither.
    7. validate_added_pr_link raises ChangelogValidationError, aborting the merge script before the rest of the PRs in the batch (if any) are processed.

    Impact: the PR looks green in CI but cannot be merged via the standard/enforced merge tooling until the placeholder is fixed — someone has to notice the failure, edit perf-changelog.yaml, and re-push before merge can proceed.

    Fix: replace {TBD} with the XXX placeholder (matching the convention used by essentially every other entry in this file, which lets canonicalization substitute the real PR number automatically) or with the literal pull/2206 link.

  • 🔴 runners/launch_gb300-nv.sh:205-211 — In the new dynamo-trt + qwen3.5 + fp4 branch, the runner checks out v1.0.29 from NVIDIA/srt-slurm, but every other sibling branch in this same if/elif chain (including the closest analog, dsv4/dynamo-trt directly above) checks out sa-submission-q2-2026 — and this PR's own perf-changelog entry states the runner clones @sa-submission-q2-2026. If unintentional, the recipes overlaid into this checkout (authored against the submission branch's srtctl schema/layout) will likely fail to apply against the public v1.0.29 tag, breaking the sweep at launch.

    Extended reasoning...

    The bug: runners/launch_gb300-nv.sh:208 (the new elif for FRAMEWORK==dynamo-trt && MODEL_PREFIX==qwen3.5 && PRECISION==fp4) does git checkout v1.0.29 after cloning NVIDIA/srt-slurm. Every other branch of this same if/elif chain that touches dynamo-trt or dynamo-sglang — the dsv4/dynamo-trt branch directly above it (line 204), the glm5/qwen3.5 dynamo-sglang branches, and the final else fallback — all check out sa-submission-q2-2026. This PR's own perf-changelog.yaml entry, added in the very same diff, spells this out explicitly: "Runner: launch_gb300-nv.sh updated to clone NVIDIA/srt-slurm@sa-submission-q2-2026 ... for the dynamo-trt+qwen3.5+fp4 path." So the code, the changelog, and every sibling code path in the file all point to sa-submission-q2-2026, and only this one new branch diverges to a different ref.

    Why this looks like a real bug, not a stale description: normally a description/code mismatch is more likely to be an imprecise PR write-up than a code defect (per the "nit by default" guidance). That reasoning doesn't hold as cleanly here, because the "description" in question isn't a vague high-level PR summary — it's a changelog entry added in the same commit that describes this exact new branch's behavior, and it agrees with five other code paths written in the same file, in the same PR history, using the same clone-and-checkout idiom. The odds that all of those independently converged on the wrong branch name, while only the new elif happens to be correct, are low. The much more parsimonious explanation is a copy/paste or typo slip when this branch was authored (e.g., pinning a tag while drafting/testing locally, then forgetting to switch back to the shared submission branch before opening the PR).

    Code path / mechanism: launch_gb300-nv.sh clones NVIDIA/srt-slurm fresh into $SRT_REPO_DIR, checks out a ref, and then this new branch does mkdir -p recipes/trtllm/qwen3.5/gb300-fp4/disagg followed by cp -rT .../benchmarks/multi_node/srt-slurm-recipes/trtllm/qwen3.5/gb300-fp4/disagg recipes/trtllm/qwen3.5/gb300-fp4/disagg, overlaying the 18 new recipe YAMLs into the checked-out repo. srtctl apply is then run against that recipe tree. The recipes carry provenance headers (# Source: /lustre/.../configs-variant1/...yaml) showing they were authored/validated against a specific run of srtctl/TensorRT-LLM — but that provenance says nothing about which srt-slurm git ref was used, so it doesn't establish that v1.0.29 (a public upstream tag) is compatible, and it cuts equally in favor of sa-submission-q2-2026 since that's the ref the PR author says was used for the whole PR.

    Why nothing in the existing code prevents this: the if/elif structure has no shared "default srt-slurm ref" variable — each branch independently hardcodes its own git checkout <ref>, so there is no single point of truth that would catch a rogue ref, and nothing in srtctl apply validates ahead of time that the recipe's on-disk schema matches the checked-out srt-slurm version until the actual apply/parse step runs.

    Impact if this is indeed a mistake: srtctl apply would attempt to parse/apply the 18 new recipe YAMLs (each with a trtllm_config, cache_transceiver_config, moe_config, dynamo: block, etc.) against whatever schema/CLI v1.0.29 ships, which may differ from the submission branch's schema the recipes were actually written for. In the best case this fails loudly at srtctl apply time (unknown field / validation error) and the whole GB300 qwen3.5 sweep — the entire point of this PR — never launches. In a worse case, if v1.0.29's schema happens to overlap partially, it could silently apply a subset of settings and produce misleading benchmark numbers instead of a clean failure.

    Step-by-step proof of the inconsistency (this part is fully verifiable from the diff alone, no external repo access needed):

    1. Diff line ~204 (dsv4/dynamo-trt, the closest analog — same FRAMEWORK): git checkout sa-submission-q2-2026.
    2. Diff line ~208 (new qwen3.5/dynamo-trt/fp4 branch): git checkout v1.0.29.
    3. Diff to perf-changelog.yaml, added in this same PR, describing this same new branch: "Runner: launch_gb300-nv.sh updated to clone NVIDIA/srt-slurm@sa-submission-q2-2026 and overlay local recipes ... for the dynamo-trt+qwen3.5+fp4 path."
    4. The else fallback (line ~215, catches every framework/model combo not otherwise matched) also uses sa-submission-q2-2026.
    5. Four independent sources within this one file/PR (three sibling code branches + the fallback + the changelog prose) agree on sa-submission-q2-2026; only the new branch's actual executed git checkout disagrees.

    Suggested fix: change line 208 from git checkout v1.0.29 to git checkout sa-submission-q2-2026, matching the sibling dynamo-trt branch and the PR's own changelog — unless the author confirms v1.0.29 was a deliberate, tested pin, in which case the changelog line should be corrected instead so the two aren't contradictory.

    One counterpoint raised during review deserves acknowledgment: it's possible v1.0.29 is an intentional, deliberate pin to a stable release tag (arguably more reproducible than tracking a moving branch), and the changelog prose is simply reused boilerplate that wasn't updated for this specific branch. I can't rule this out without access to the upstream NVIDIA/srt-slurm repo to check whether v1.0.29 actually supports the TensorRT-LLM disagg recipe schema these 18 YAMLs use. That uncertainty is exactly why this should be flagged for the author to confirm before merge, rather than silently waved through — but the concrete failure mode (sweep launch fails) if it's wrong, combined with the accumulated weight of evidence (5 independent agreements vs. 1 outlier), pushes this from a nice-to-have nit to something the author should explicitly resolve before merging.

@github-actions

Copy link
Copy Markdown
Contributor

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs


感谢你的贡献!请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md,然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人,你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板,包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have

如需进行 PR 验证,请为此 PR 添加 full-sweep-fail-fast 标签(强烈推荐)— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动(flake),重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

@github-actions

Copy link
Copy Markdown
Contributor

@github-actions

Copy link
Copy Markdown
Contributor

2 similar comments
@github-actions

Copy link
Copy Markdown
Contributor

@github-actions

Copy link
Copy Markdown
Contributor

@Ankur-singh Ankur-singh changed the title [NV] Add dynamo trtllm gb300 qwen3.5 nvfp4 recipes for 8k1k and 1k1k [NV] Add GB300 Qwen3.5 NVFP4 Dynamo-TensorRT-LLM disaggregated benchmarks / [NVIDIA] 添加 GB300 Qwen3.5 NVFP4 Dynamo-TensorRT-LLM 分离式基准测试 Jul 27, 2026
@Ankur-singh

Copy link
Copy Markdown
Collaborator

As a PR reviewer and CODEOWNER, I have reviewed this and have:

  • Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
  • Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
  • Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29369774630
  • Verified that this PR passes evals. Please link to GitHub Action workflow that shows this. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29369774630
  • Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
  • For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
  • Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; FLOPs at lower precisions is fine, given that the config passes private evals. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
  • If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
  • If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
  • Verified that every single-node vLLM/SGLang recipe in this PR is documented in the official vLLM recipes and/or the SGLang cookbook:
    • I linked the corresponding upstream PR in the vLLM recipe repo or SGLang repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
  • Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/<PR_NUMBER>.md — named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section.
  • If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.

Additional detail section:

Scope. New config key qwen3.5-fp4-gb300-dynamo-trt in configs/nvidia-master.yaml (+271 lines, additive only), 18 new srt-slurm recipe YAMLs under benchmarks/multi_node/srt-slurm-recipes/trtllm/qwen3.5/gb300-fp4/disagg/ (9 x 1k1k, 9 x 8k1k, all STP / no MTP), one perf-changelog.yaml entry, and one new elif branch in runners/launch_gb300-nv.sh. Model nvidia/Qwen3.5-397B-A17B-NVFP4, image nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc18, GB300, disaggregated, NIXL KV P2P. No existing config key, recipe, or launcher branch is modified. 21 files, 4 commits.

Validation evidence. Run Sweep 29369774630 (https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29369774630) ran at the exact current head SHA 36dd0126829a3e47eb7ff1701f59d92b86276b0f. This is a real run, not a reuse-gated no-op: setup = success, reuse-ingest-artifacts = skipped, and the multi-node eval lane's if: guard requires needs.setup.outputs.reuse-enabled != 'true', which it satisfied. Non-skipped successful job counts:

  • multi-node 1k1k /: 9 / 9 success (conc 16, 68, 136, 272, 640, 1280, 1536, 7680, 17920) — durations 33–265 min.
  • multi-node 8k1k /: 9 / 9 success (conc 24, 48, 78, 180, 216, 1152, 1536, 1728, 4608) — durations 20–300 min.
  • multi-node eval / (eval-only): 9 / 9 success (conc 24, 48, 78, 180, 216, 1152, 1536, 1728, 4608) — durations 19–89 min.
  • Total ~30 GPU-hours of wall-clock across 27 real jobs. Zero failures, zero cancellations in the sweep.
  • Aggregators collect-results, collect-evals, compare-results, calc-success-rate, upload-changelog-metadata, check-changelog all success. CodeQL Analyze (python) and Analyze (actions) success at head.
  • Skipped lanes are all correctly-inapplicable: single-node 1k1k /, single-node 8k1k /, eval / (single-node), canary /, agentic /, multi-node agentic /, trigger-ingest, trigger-agentic-ingest, reuse-ingest-artifacts. This config is multinode: true with no single-node or agentic scenarios, so those lanes have no matrix entries.

Eval coverage is complete, not partial. Evals ran only on the 8k1k configs. Per utils/evals/EVALS.md ("Selection"), multi-node evals are 8k1k only, one job per parallelism topology at its highest eligible concurrency. This PR contributes exactly 9 distinct 8k1k topologies and all 9 got an eval job, all successful. The 1k1k lane is not eligible for evals by design, so there is no missing eval coverage here.

Image / framework reasoning. The pinned image nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc18 is an official NVIDIA NGC TensorRT-LLM release image. The upstream-Docker-repo item (hub.docker.com/u/vllm / hub.docker.com/u/lmsysorg) governs vLLM and SGLang submissions; this PR contains no vLLM or SGLang submission, so that item is not applicable and is left unchecked rather than falsely checked. The image string in configs/nvidia-master.yaml at head, in all 18 recipe YAMLs (identity.container.image, model.container), and in every single sweep job name is byte-identical: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc18. The config at head is exactly what the sweep ran.

vLLM/SGLang-first ordering — satisfied. qwen3.5-fp4-gb300-dynamo-sglang already exists on main (configs/nvidia-master.yaml line 6586 on main), i.e. the SGLang submission for the same model + precision + SKU landed before this TensorRT-LLM submission. Ordering requirement met.

Recipe requirement — not applicable. All 18 recipes in this PR live under benchmarks/multi_node/srt-slurm-recipes/...; there are zero single-node recipes, and both single-node sweep lanes were skipped for having no matrix entries. The vLLM-recipes / SGLang-cookbook requirement is scoped to single-node vLLM/SGLang recipes, so it does not apply to this multi-node TensorRT-LLM submission. Both boxes left unchecked as N/A rather than checked.

Speculative decoding — not applicable. This submission has no speculative decoding of any kind. Every recipe filename carries mtp0; grepping the full effective diff for speculative, spec_config, mtp, nextn, eagle, draft, SGLANG_SIMULATE_ACC_LEN, synthetic_acceptance_length, and TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS returns no config keys or env vars — only the mtp0 substring inside filenames and one changelog line that explicitly says "no MTP". Consequently: no chat-template flag to verify, and no golden-AL value to check against golden_al_distribution/. This is also not an agentic/AgentX workload (agentic / and multi-node agentic / lanes both skipped, no agentic scenario in the config). Both spec-decode items are left unchecked as N/A.

Model architecture — unchanged. No --hf-overrides, no hf_overrides, no layer-skipping, no indexer-skipping, no MoE-layer dropping, and no reduction in architecture FLOPs anywhere in the diff. The only precision-related settings are kv_cache_config.dtype: fp8, moe_config.use_low_precision_moe_combine: true, and the NVFP4 GEMM backend allow-list (cutlass, cublaslt, cutedsl, cuda_core) — these are lower-precision / kernel-backend choices, which the checklist explicitly permits provided evals pass, and all 9 eval jobs passed. trust_remote_code: true is used, which is standard for this model family in this repo.

Patch / waiver — nothing patched, with full disclosure of the two borderline items. Grepping every added line in the effective diff for git apply, .patch, patch -p, sed -i, site-packages, monkey-patching, heredoc rewrites of engine sources, and forked/rebuilt engine wheel installs returns zero hits. docs/waiver/ at head contains only .gitkeep; no waiver is needed and none is claimed. Two things a reviewer should see and judge for themselves, both disclosed rather than hidden:

  1. runners/launch_gb300-nv.sh clones NVIDIA/srt-slurm at tag v1.0.29 and then cp -rT's this PR's in-repo recipe directory over recipes/trtllm/qwen3.5/gb300-fp4/disagg in that checkout. This overlays harness configuration files, not engine or serving-stack source. It is the established pattern already used by six other framework/model branches in the very same launcher at head (lines 161, 168, 175, 184, 208, 214, 224 of runners/launch_gb300-nv.sh), and the copied files are exactly the 18 YAMLs reviewable in this diff.
  2. Each recipe carries dynamo: {install: true, version: 1.3.0.dev20260708, request_plane: tcp}, i.e. the Dynamo disaggregated-serving frontend is installed into the pinned TRT-LLM container at runtime. Dynamo is an official upstream NVIDIA release consumed as-is — it is not a fork, not a rebuild, and not a modification of the TensorRT-LLM engine. This is the standing convention for every dynamo-* framework config in the repo (191 existing in-repo srt-slurm recipes on main already carry install: true). Flagging it explicitly so the decision to treat it as "as shipped" is made knowingly rather than by omission.

The launcher branch also adds --no-preflight to SRTCTL_APPLY_ARGS for dynamo-trt + qwen3.5 + fp4, matching the existing carve-outs for agentic, glm5.1, and qwen3.5+fp8. This suppresses a login-node preflight check only; it does not alter what runs on the compute nodes.

Why "general code quality" is left unchecked. Two factual inaccuracies in committed text, both cosmetic and both one-line fixes, but neither should be silently checked off:

  1. perf-changelog.yaml states "launch_gb300-nv.sh updated to clone NVIDIA/srt-slurm@sa-submission-q2-2026", but the code added in the same PR checks out v1.0.29, not sa-submission-q2-2026. perf-changelog.yaml is a published artifact, so this line should be corrected before merge. The changelog entry also omits the --no-preflight change.
  2. benchmarks/multi_node/srt-slurm-recipes/trtllm/qwen3.5/gb300-fp4/disagg/8k1k/stp/ctx7dep2_gen12tep4_mtp0_c1152.yaml has a header comment reading "9th point (C768) for exp3 8k1k mtp0 srt-slurm" while the file's name, benchmark.concurrencies, and matching conc-list are all 1152. Stale provenance comment, presumably from a retarget.

Neither affects what the sweep executed. Independently verified that all 18 recipes' benchmark.concurrencies match both their filenames and the corresponding conc-list entries in configs/nvidia-master.yaml at head — 18/18 consistent.

Other notes for the record. Base repo is SemiAnalysisAI/InferenceX, head repo is the same (not a fork), base branch main, label full-sweep-enabled, not a draft. No /reuse-sweep-run comment exists on this PR and none is required, since a qualifying full sweep exists at the exact current head SHA. There are no prior approving reviews at this head. Earlier Run Sweep attempts on this branch (29358114617 success and 29358360443 cancelled at 376b8918, 29357761821 and 29357696360 failures at earlier SHAs) are superseded and were not relied on for any claim above.

Signed: Ankur-singh

@Klaud-Cold

Copy link
Copy Markdown
Collaborator

❌❌❌ REJECTED ❌❌❌

@Ankur-singh — one blocker: no authorized /reuse-sweep-run command has been posted on this PR. The green sweep at head is real, but the reuse command must be on record so merge_with_reuse.sh consumes it instead of re-running it. An authorized maintainer (OWNER/MEMBER/COLLABORATOR) should comment /reuse-sweep-run 29369774630; everything else verified clean.

✅ Check 0 (CODEOWNER): PASS — Ankur-singh is a listed owner of configs/nvidia-master.yaml; remaining paths fall under the catch-all.
✅ Check 1 (sweep on in-PR commit): PASS — head 36dd012 has 9/9 multi-node 1k1k /, 9/9 multi-node 8k1k /, 9/9 multi-node eval / all success (executed, not skipped) in run 29369774630; single-node lanes correctly have no matrix entries.
✅ Check 2 (evals pass): PASS — agg_eval_all.json from that run: 10 gsm8k results for Qwen3.5-397B-A17B-NVFP4/dynamo-trt/GB300, em_strict 0.964–0.974, all above the 0.94 qwen3.5 bar in utils/evals/thresholds.yaml, on the PR's exact image nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc18.
➖ Check 3 (recipe link): N/A — disaggregated/multi-node submission (multinode: true, disagg: true, all recipes under benchmarks/multi_node/srt-slurm-recipes/**); the recipe-link requirement applies to single-node recipes only.
❌ Check 4 (reuse command): FAIL — no authorized /reuse-sweep-run command has been posted on this PR; a green sweep alone is not enough. An authorized maintainer must comment /reuse-sweep-run before this PR can merge via reuse.
✅ Check 5 (latest checklist): PASS — all current-template items present; each unchecked box (code quality, spec-decode, agentic-AL, vLLM/SGLang image, recipe link) has an explicit explanation in the additional detail section.
✅ Check 6 (upstream image / engine-first): PASS — new entry is framework: dynamo-trt (official NGC TRT-LLM release image), so the vLLM/SGLang upstream-image rule doesn't apply; engine-first satisfied by qwen3.5-fp4-gb300-dynamo-sglang (lmsysorg image, same model-prefix + GB300) already on main.
✅ Check 7 (no architecture hacks): PASS — no --hf-overrides/layer-trimming anywhere in the diff; only precision/kernel-backend knobs (kv-cache fp8, low-precision MoE combine, NVFP4 GEMM backend list), which evals cover.
➖ Check 8 (spec-decode chat template): N/A — no speculative decoding in this PR (all configs mtp0/STP; no spec flags in the diff).
✅ Check 9 (no engine patches): PASS — no patch/sed/site-packages/wheel-install hits in the diff; the launcher overlays in-repo recipe YAMLs into the srt-slurm@v1.0.29 checkout (harness config, established pattern) and the runtime Dynamo install is the standing upstream convention for dynamo-* configs.
➖ Check 10 (agentic golden AL): N/A — no agentic speculative-decoding changes.

Informational, non-blocking (already disclosed in the sign-off): perf-changelog.yaml says NVIDIA/srt-slurm@sa-submission-q2-2026 but the launcher checks out v1.0.29, and ctx7dep2_gen12tep4_mtp0_c1152.yaml carries a stale "C768" header comment — both worth a one-line fix before or after merge.

…-trt / 移除已弃用的 Single-turn 1k1k 场景

Single-turn 1k1k (isl 1024 / osl 1024) is deprecated for all models since
2026-07-17 (#2263) per MODELS.md, and the deprecation was enacted on
2026-08-04 by #2493, which removed 54 config keys from the active master
configs. This submission still added a 1k1k scenario, so it is removed here.

- Delete the 9 disagg 1k1k STP recipes under
  benchmarks/multi_node/srt-slurm-recipes/trtllm/qwen3.5/gb300-fp4/disagg/1k1k/
- Remove the isl 1024 / osl 1024 scenario (9 search-space points) from the
  qwen3.5-fp4-gb300-dynamo-trt block in configs/nvidia-master.yaml
- Align the perf-changelog entry to describe an 8k1k-only submission

The Single-turn 8k1k half is unchanged: 9 recipes and 9 search-space points
remain byte-identical.

中文:根据 MODELS.md,Single-turn 1k1k(isl 1024 / osl 1024)自 2026-07-17
起对所有模型弃用(#2263),并已于 2026-08-04 由 #2493 正式执行(从活跃主配置中
移除了 54 个 config key)。本次提交仍新增了 1k1k 场景,故在此移除:删除
gb300-fp4/disagg/1k1k/ 下的 9 个 STP recipe,从 configs/nvidia-master.yaml 的
qwen3.5-fp4-gb300-dynamo-trt 块中删除 isl 1024 / osl 1024 场景及其 9 个搜索空间
配置点,并同步修改 perf-changelog 描述为仅 8k1k。Single-turn 8k1k 部分保持不变,
9 个 recipe 与 9 个搜索空间配置点逐字节保留。
MODELS.md deprecates Single-turn 1k1k for all models (2026-07-17), enacted
2026-08-04 by #2493, which removed 54 config keys and archived them. This PR
still added 1k1k, so remove that half:

- delete the 9 disagg/1k1k/stp recipes
- drop the isl:1024/osl:1024 scenario and its 9 search-space points from
  qwen3.5-fp4-gb300-dynamo-trt, leaving the 8k1k entry byte-identical
- reword the perf-changelog entry to describe an 8k1k-only submission

The main merge also conflicted in configs/nvidia-master.yaml: this branch
anchored the new block before glm5-fp4-gb300-dynamo-sglang, which main
deleted with the rest of the GLM-5 family. Resolved by keeping main's
replacement key (qwen3.5-fp4-gb300-dynamo-sglang-agentic-agg) and placing
the new dynamo-trt block immediately before it, so the file diff against
main is purely the added block.
Two label defects, no behaviour change:

- configs/nvidia-master.yaml declared kv-p2p-transfer: nixl, but all nine
  retained recipes set cache_transceiver_config.backend: UCX on both prefill
  and decode. That value is propagated verbatim into every result row as
  kv_p2p_transfer, so each published point was labelled NIXL while running
  UCX. Corrected to ucx to describe what actually runs.

- The perf-changelog entry said the runner clones srt-slurm@sa-submission-q2-2026
  while launch_gb300-nv.sh checks out the immutable tag v1.0.29 on this path.
  Corrected, and the --no-preflight change this PR also makes is now recorded.
…og last / 合并 origin/main:保留两个 GB300 TRT 配置键并将 changelog 条目追加到末尾

check-changelog rejected the previous head because perf-changelog.yaml showed a
deleted line (- qwen3.5-fp4-gb300-dynamo-trt-mtp): main had since gained the
#2218 and #2438 entries, which this branch did not carry. Deletions are never
permitted in perf-changelog.yaml.

Resolution keeps main's entries intact and re-appends this PR's entry at the
very end, so the diff against main is additions-only. configs/nvidia-master.yaml
was an add/add conflict between this branch's qwen3.5-fp4-gb300-dynamo-trt and
main's qwen3.5-fp4-gb300-dynamo-trt-mtp (#2218); both keys are kept, each with
its own trailing dp-attn line.
@Ankur-singh

Copy link
Copy Markdown
Collaborator

Revoking the /reuse-sweep-run 29369774630 authorization on this PR (removing the pinned command comment from 2026-07-29). This PR needs a genuine fresh sweep, not a reuse.

Why that pin is no longer safe. Run 29369774630 is from 2026-07-14, at head 36dd0126, and it measured 9 multi-node 1k1k jobs alongside its 9 multi-node 8k1k jobs. Since then this PR deleted the 1k1k scenario entirely — Single-turn 1k1k is deprecated for every model as of #2493, and qwen3.5-fp4-gb300-dynamo-trt now declares only isl: 8192 / osl: 1024. That run therefore describes a config shape this PR no longer submits.

Why the gate would not have caught it. utils/find_reusable_sweep_run.py performs no config-drift check on a pinned run: validate_reusable_run only asserts event == pull_request, status == completed, and a conclusion in the allowed set — and for a pinned run it passes allow_failed=True, widening that set to {success, failure, cancelled}. A pinned ID is treated as an explicit maintainer choice and consumed as-is. So the next synchronize on this PR would have bound 1k1k-era artifacts and set skip-pr-sweep=true, publishing points the current config does not declare — with nothing failing to signal it.

The 8k1k half of that old run is not a substitute either: the config has changed underneath it (kv-p2p-transfer corrected nixlucx to match the recipes' UCX backend), so it no longer describes the configuration that would be published.

Re-authorize with an explicit run ID once a fresh sweep lands at the current head.

…main:在 #2492 之后重新追加 changelog 条目

PR #2492 merged while this branch was being fixed, adding another
perf-changelog.yaml entry and re-conflicting this PR before its sweep could
start. Sweeps do not trigger on a conflicted PR, so no run was ever created at
the previous head.

Resolution keeps main's #2492 entry and re-appends this PR's entry last, so the
diff against main stays additions-only (142/0 on nvidia-master.yaml, 9/0 on
perf-changelog.yaml).
@github-actions

github-actions Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

1 similar comment
@github-actions

github-actions Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

@github-actions

github-actions Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

@xinli-sw

xinli-sw commented Aug 9, 2026

Copy link
Copy Markdown
Collaborator

/reuse-sweep-run

@xinli-sw

xinli-sw commented Aug 9, 2026

Copy link
Copy Markdown
Collaborator

@Ankur-singh please help sign off, thanks!

@Ankur-singh

Copy link
Copy Markdown
Collaborator

/reuse-sweep-run 31131975660

@Ankur-singh

Copy link
Copy Markdown
Collaborator

As a PR reviewer and CODEOWNER, I have reviewed this and have:

  • Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
  • Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
  • Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/31131975660
  • Verified that this PR passes evals. Please link to GitHub Action workflow that shows this. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/31131975660
  • Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
  • For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
  • Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
  • Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; FLOPs at lower precisions is fine, given that the config passes private evals. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
  • If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
  • If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
  • Verified that every single-node vLLM/SGLang recipe in this PR is documented in the official vLLM recipes and/or the SGLang cookbook:
    • I linked the corresponding upstream PR in the vLLM recipe repo or SGLang repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
  • Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/<PR_NUMBER>.md — named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section.
  • If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.

Additional detail section:

Validation and eval evidence. Run 31131975660 ran on the exact PR head 0ac8ea4c, attempt 3, and settled with 28 success / 10 skipped / 0 failures: 9/9 multi-node 8k1k / benchmark jobs and 9/9 multi-node eval / jobs all executed and succeeded, with collect-results, collect-evals, calc-success-rate, and compare-results green. The skipped lanes (single-node, multi-node 1k1k, agentic, canary) have no matrix entries for this config and are correctly not applicable. Earlier attempts on this head failed only on a cluster-infrastructure fault — flock: 9: Protocol family not supported while importing the nginx squash on node im-gb300-r01-c003, whose /data/home/sa-shared mount does not support file locking — which never reached container start or the benchmark; the reruns cleared it and no configuration in this PR is implicated.

DISAGG master↔recipe parity. Re-derived independently for all nine points rather than taken from any creator packet; every nvidia-master.yaml variant agrees with its CONFIG_FILE= recipe, including the naming convention where dep denotes dp-attention enabled and tep/tp denote it disabled:

conc master prefill master decode recipe
78 1 x TP2/EP2, dp-attn 13 x TP4/EP1 ctx1dep2_gen13tp4_mtp0_c78
24 1 x TP2/EP2, dp-attn 4 x TP16/EP16 ctx1dep2_gen4tep16_mtp0_c24
48 1 x TP2/EP2, dp-attn 8 x TP8/EP8 ctx1dep2_gen8tep8_mtp0_c48
180 2 x TP2/EP2, dp-attn 15 x TP4/EP1 ctx2dep2_gen15tp4_mtp0_c180
216 2 x TP2/EP2, dp-attn 9 x TP4/EP1 ctx2dep2_gen9tp4_mtp0_c216
1536 3 x TP2/EP2, dp-attn 1 x TP4/EP4, dp-attn ctx3dep2_gen1dep4_mtp0_c1536
1152 7 x TP2/EP2, dp-attn 12 x TP4/EP4 ctx7dep2_gen12tep4_mtp0_c1152
1728 8 x TP2/EP2, dp-attn 9 x TP4/EP4 ctx8dep2_gen9tep4_mtp0_c1728
4608 10 x TP2/EP2, dp-attn 3 x TP8/EP8, dp-attn ctx10dep2_gen3dep8_mtp0_c4608

Recipe resource blocks agree with the master worker/TP counts on GB300's 4-GPU nodes (c4608: 10 prefill workers x 2 GPUs = 20 GPUs = 5 prefill nodes; 3 decode workers x 8 GPUs = 24 GPUs = 6 decode nodes). KV transport is consistent end to end: master declares kv-p2p-transfer: ucx and all nine recipes set backend: UCX on both the prefill and decode halves, with no nixl anywhere.

Model and scenario scope. MODELS.md lists Qwen3.5-397B-A17B as active for Single-turn 8k1k at fp8/fp4. This submission is 8k1k NVFP4 only. Single-turn 1k1k was deprecated on 2026-08-04 (#2493) and this PR correctly carries no 1k1k lane, in either the master entry or the sweep matrix.

Engine-first ordering — satisfied. This is a framework: dynamo-trt submission, and qwen3.5-fp4-gb300-dynamo-sglang (upstream lmsysorg image, same model-prefix, same GB300 SKU) is already on main ahead of it, so the upstream SGLang engine led this hardware/model combination. MODELS.md records Qwen3.5's engine expectation as the native/upstream SGLang engine with additional non-vLLM/SGLang engines permitted under the ordering guideline.

Scope exclusions, left unchecked deliberately. Speculative decoding and the agentic golden-AL item are not applicable: every config is mtp0/STP and the diff contains no speculative, draft, EAGLE, or MTP settings. The vLLM/SGLang upstream-image item is not applicable because this PR adds no vLLM or SGLang submission; it pins the official NGC release image nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc18. The single-node recipe-publication item and its nested sub-item are not applicable because all nine recipes are multi-node srt-slurm recipes under benchmarks/multi_node/srt-slurm-recipes/** (multinode: true, disagg: true).

No engine or serving-stack patching. The diff contains no .patch, git apply, sed -i, site-packages edit, monkey-patch, or forked/rebuilt engine wheel. The pinned NGC TRT-LLM image runs as shipped. launch_gb300-nv.sh clones NVIDIA/srt-slurm@v1.0.29 and overlays this repo's recipe YAMLs into its recipes/ tree — harness configuration, the established pattern for srt-slurm lanes — and that launcher branch already exists on main and is not modified by this PR. The runtime Dynamo install is the standing convention for dynamo-* configs.

Informational, non-blocking. Two cosmetic documentation nits, neither affecting what was benchmarked or published: the changelog bullet renders the 2P15D (conc 180) and 2P9D (conc 216) points as a single "2P15D DEP2+15D TP4 (conc 180/216)" entry, when 216 is the 2P9D point; and the first header line of ctx7dep2_gen12tep4_mtp0_c1152.yaml still labels it "(C768)", though the following lines document the 768 -> 1152 sysc x1.5 transform and the recipe name and contents are c1152. The srt-slurm@sa-submission-q2-2026 versus v1.0.29 mismatch raised on the previous sign-off is resolved at this head: the changelog now states v1.0.29, matching the launcher. The changelog's runner bullet describes the already-landed dynamo-trt + qwen3.5 + fp4 launcher path rather than a change introduced by this PR.

Signed: Ankur-singh

@Klaud-Cold

Copy link
Copy Markdown
Collaborator

✅✅✅ Verdict: PASS ✅✅✅

✅ Check 0 (CODEOWNER): PASS — @Ankur-singh is a named owner of configs/nvidia-master.yaml; remaining paths carry only the * @InferenceX/core catch-all, satisfied by a recognized CODEOWNER.
✅ Check 1 (sweep on in-PR commit): PASS — head 0ac8ea4c carries run 31131975660 with 9/9 multi-node 8k1k / and 9/9 multi-node eval / per-config check-runs executed and success (skipped lanes have no matrix entries for this config).
✅ Check 2 (evals pass): PASS — all 9 configs scored GSM8K em_strict 0.9651–0.9697 (n_eff 1319), above the 0.94 qwen3.5 bar in utils/evals/thresholds.yaml, on the PR's image nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc18.
➖ Check 3 (recipe link): N/A — disaggregated/multi-node submission (benchmarks/multi_node/srt-slurm-recipes/**, multinode: true, disagg: true, dynamo-trt); the recipe-link requirement applies to single-node recipes only.
✅ Check 4 (reuse command): PASS — /reuse-sweep-run 31131975660 posted by Ankur-singh (COLLABORATOR); the earlier revocation covered only the stale pin 29369774630.
✅ Check 5 (latest checklist): PASS — every current-template item is present; the four unchecked items are each explained as inapplicable in the additional detail section.
✅ Check 6 (upstream image / engine-first): PASS — entry is framework: dynamo-trt (upstream-image rule targets vLLM/SGLang entries), and qwen3.5-fp4-gb300-dynamo-sglang (upstream lmsysorg/sglang image, same model-prefix and GB300 runner) is already on main, so the SGLang engine led.
✅ Check 7 (deprecated models): PASS — Qwen3.5 Single-turn 8k1k fp4 is active per MODELS.md; the PR carries no 1k1k lane (deprecated 2026-08-04).
✅ Check 8 (no architecture hacks): PASS — no --hf-overrides/model-config edits; recipes are plain TRT-LLM configs (fp8 KV cache is a precision choice, evals pass).
➖ Check 9 (spec-decode chat templates): N/A — all configs are mtp0/STP with no speculative flags in the diff.
✅ Check 10 (no engine patches): PASS — no .patch/git apply/sed -i/site-packages edits; the pinned NGC image runs as shipped, and the runtime Dynamo install is frontend/orchestrator plumbing per the standing dynamo-* convention.
➖ Check 11 (agentic golden AL): N/A — no agentic speculative-decoding changes.

@adibarra
adibarra merged commit 4830e33 into main Aug 10, 2026
27 checks passed
@adibarra
adibarra deleted the dynamo-trtllm-gb300-qwen3.5-nvfp4 branch August 10, 2026 22:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Development

Successfully merging this pull request may close these issues.

5 participants