perf: add GB200 DeepSeek V4 FP4 vLLM MTP AgentX - #2567
Conversation
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
cquil11
added
the
agentx-fast
|
Claude finished @cquil11's task in 6m 57s —— View job Review of PR #2567
SummaryOne 🟡 WARNING posted inline: the new launcher branch in Everything else checked out: |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| # Agentic DeepSeek-V4 uses the latest released srt-slurm. v1.0.45 carries | ||
| # custom-benchmark support plus ordered logical-worker Prometheus endpoint | ||
| # injection, so disaggregated AIPerf runs receive every P/D metrics URL. | ||
| if [[ "$IS_AGENTIC" == "1" && "$MODEL_PREFIX" == "dsv4" && "$FRAMEWORK" == "dynamo-vllm" ]]; then |
There was a problem hiding this comment.
🟡 WARNING: This condition doesn't only route the new MTP recipes — it also captures the existing non-MTP DSV4 GB200 agentic lanes (dsv4-fp4-gb200-dynamo-vllm-agentic-2p1d-dep8-dep8 and ...-3p2d-tep8-tp8, configs/nvidia-master.yaml ~lines 7240–7326, both agentic-coding/dynamo-vllm/dsv4, so IS_AGENTIC=1). They previously fell through to the elif below and cloned the pinned cquil11/srt-slurm-nv@de59739; after this PR they clone NVIDIA/srt-slurm v1.0.45.
Why it matters: Those existing recipes pin dynamo.wheel: "1.3.0.dev20260618" — a field this file's own comment attributes to the fork ("DynamoConfig.wheel (recipes pin the ai-dynamo wheel)"). If upstream v1.0.45 doesn't accept dynamo.wheel, the existing lanes break at srtctl apply on their next main-branch run; if it silently ignores it, they run an unpinned dynamo build — a silent toolchain change to already-published lanes. The PR states only the four new MTP recipes were dry-run against v1.0.45, and the perf-changelog entry doesn't list the existing config keys as affected.
Fix: Scope the new branch to the new recipes (the stripped CONFIG_FILE basename is a clean discriminator — only the new GB200 recipes end in -mtp-agentic.yaml on this runner):
| if [[ "$IS_AGENTIC" == "1" && "$MODEL_PREFIX" == "dsv4" && "$FRAMEWORK" == "dynamo-vllm" ]]; then | |
| if [[ "$IS_AGENTIC" == "1" && "$MODEL_PREFIX" == "dsv4" && "$FRAMEWORK" == "dynamo-vllm" && "${CONFIG_FILE%%:*}" == *-mtp-agentic.yaml ]]; then |
Alternatively, if migrating the existing non-MTP dsv4 lanes to v1.0.45 is intentional, dry-run those two recipes against it (including the dynamo.wheel field) and add their config keys to the perf-changelog entry.
| # Agentic DeepSeek-V4 uses the latest released srt-slurm. v1.0.45 carries | ||
| # custom-benchmark support plus ordered logical-worker Prometheus endpoint | ||
| # injection, so disaggregated AIPerf runs receive every P/D metrics URL. | ||
| if [[ "$IS_AGENTIC" == "1" && "$MODEL_PREFIX" == "dsv4" && "$FRAMEWORK" == "dynamo-vllm" ]]; then | ||
| git clone --branch v1.0.45 --single-branch https://github.com/NVIDIA/srt-slurm.git "$SRT_REPO_DIR" | ||
| cd "$SRT_REPO_DIR" | ||
| test "$(git rev-parse HEAD)" = "9d8d92b20c350a5d42f0709f5a0b64e30eb37d33" || { | ||
| echo "Error: NVIDIA/srt-slurm v1.0.45 resolved to an unexpected commit" >&2 | ||
| exit 1 | ||
| } | ||
| mkdir -p recipes/vllm/deepseek-v4/agentic | ||
| cp -rT "$GITHUB_WORKSPACE/benchmarks/multi_node/srt-slurm-recipes/vllm/deepseek-v4/agentic" \ | ||
| recipes/vllm/deepseek-v4/agentic | ||
| # TODO(CJQ): migrate the remaining Agentic model paths to released srt-slurm. |
There was a problem hiding this comment.
🔴 The new branch guard [[ $IS_AGENTIC == 1 && $MODEL_PREFIX == dsv4 && $FRAMEWORK == dynamo-vllm ]] in runners/launch_gb200-nv.sh (lines 391-404) is broader than the MTP recipes it was added for: it also matches two pre-existing configs, dsv4-fp4-gb200-dynamo-vllm-agentic-3p2d-tep8-tp8 and dsv4-fp4-gb200-dynamo-vllm-agentic-2p1d-dep8-dep8 (configs/nvidia-master.yaml:7217, :7275), which previously fell into the elif [[ $IS_AGENTIC == 1 ]] branch that clones the cquil11 fork. Rerouting them onto NVIDIA/srt-slurm v1.0.45 will likely break them, since their recipes still declare dynamo: { install: true, wheel: "1.3.0.dev20260618" } — a fork-specific field the new v1.0.45 recipes deliberately dropped in favor of dynamo: { version, install }. The condition should be scoped to only the new MTP recipes (e.g. gate on an MTP-specific marker) instead of every dsv4/dynamo-vllm/agentic-coding config.
Extended reasoning...
The bug: The new first branch in runners/launch_gb200-nv.sh (added at line 394) tests [[ "$IS_AGENTIC" == "1" && "$MODEL_PREFIX" == "dsv4" && "$FRAMEWORK" == "dynamo-vllm" ]]. This was written to route the four brand-new MTP recipes added in this PR (dsv4-fp4-gb200-dynamo-vllm-agentic-mtp-agg and -mtp-disagg) onto the newly-released NVIDIA/srt-slurm v1.0.45. However, the condition has no MTP-specific discriminator, so it also matches two pre-existing, previously-working configs that happen to share the same model-prefix: dsv4, framework: dynamo-vllm, and scenarios: agentic-coding combination: dsv4-fp4-gb200-dynamo-vllm-agentic-3p2d-tep8-tp8 and dsv4-fp4-gb200-dynamo-vllm-agentic-2p1d-dep8-dep8 (configs/nvidia-master.yaml lines ~7217 and ~7275, just above the new MTP entries added by this PR at line 7330+).\n\nWhy IS_AGENTIC/MODEL_PREFIX/FRAMEWORK line up for the old configs too: IS_AGENTIC is derived purely from scenario-type in .github/workflows/benchmark-multinode-tmpl.yml:247 (1 iff scenario-type == 'agentic-coding'). Both pre-existing configs declare scenarios: agentic-coding, model-prefix: dsv4, and framework: dynamo-vllm — exactly the same triple as the new MTP configs. Before this PR, they fell through to the retained elif [[ "$IS_AGENTIC" == "1" ]] branch (line ~405), which clones cquil11/srt-slurm-nv at commit de59739b.... After this PR, they hit the new, more specific first branch instead and get NVIDIA/srt-slurm v1.0.45 cloned in its place, with the entire benchmarks/multi_node/srt-slurm-recipes/vllm/deepseek-v4/agentic directory (old recipes included) copied in via cp -rT.\n\nWhy this breaks the old recipes: The two rerouted recipe files, disagg-gb200-3p2d-tep8-tp8-agentic.yaml and disagg-gb200-2p1d-dep8-dep8-agentic.yaml, still declare dynamo: { install: true, wheel: "1.3.0.dev20260618" }. The retained elif branch's own comment explicitly documents this as a feature that only the cquil11 fork provides: DynamoConfig.wheel (recipes pin the ai-dynamo wheel). The new v1.0.45 MTP recipes added by this very PR (e.g. agg-gb200-tp8-mtp-agentic.yaml) deliberately use dynamo: { version: "1.3.1", install: true } instead — no wheel field — strongly implying that stock v1.0.45's DynamoConfig schema does not accept wheel. Running srtctl apply against v1.0.45 with a recipe that still sets wheel will therefore either fail schema validation or silently ignore the pin and install a different Dynamo build than the one these sweeps were validated against.\n\nWhy nothing else catches this: The PR's Validation section only exercises the four new MTP recipes ("all four recipes parse and dry-run with NVIDIA srt-slurm v1.0.45"); it never re-validates the pre-existing 3p2d/2p1d configs under the new branch, since the author's intent was only to add MTP coverage, not to touch those sweeps. There is no test or guard in launch_gb200-nv.sh that distinguishes MTP configs from non-MTP ones within the dsv4/dynamo-vllm/agentic-coding space, so the reroute is silent — it will only surface as a build/apply failure (or a silently wrong Dynamo version) the next time those two sweeps run.\n\nStep-by-step proof:\n1. dsv4-fp4-gb200-dynamo-vllm-agentic-2p1d-dep8-dep8 (nvidia-master.yaml:7275) has scenarios: agentic-coding, model-prefix: dsv4, framework: dynamo-vllm.\n2. The workflow template sets IS_AGENTIC=1 for this scenario, and MODEL_PREFIX/FRAMEWORK env vars are populated from the config's own fields, so at runtime: IS_AGENTIC=1, MODEL_PREFIX=dsv4, FRAMEWORK=dynamo-vllm.\n3. In launch_gb200-nv.sh, the new condition [[ "$IS_AGENTIC" == "1" && "$MODEL_PREFIX" == "dsv4" && "$FRAMEWORK" == "dynamo-vllm" ]] evaluates true — matching this config even though it has no MTP recipe.\n4. The script clones NVIDIA/srt-slurm v1.0.45 and copies the whole agentic recipe directory into it, including disagg-gb200-2p1d-dep8-dep8-agentic.yaml.\n5. srtctl apply runs this recipe (which still has dynamo.wheel: "1.3.0.dev20260618") against v1.0.45, whose DynamoConfig schema (per the new recipes' switch to version+install) no longer models a wheel field — the previously-working sweep now either errors out or silently drops the pinned wheel install.\n\nThe fix: Scope the new branch's condition to something MTP-specific — e.g. check for an MTP marker in the config (a spec-decoding: mtp search-space key, a distinct model-prefix/config-key suffix like -mtp-, or an explicit env var set only by the new MTP configs) rather than the broad dsv4+dynamo-vllm+agentic-coding triple that also matches the pre-existing 3p2d/2p1d configs.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31540457767 |
…-mtp-agentx # Conflicts: # perf-changelog.yaml
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31558047670 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31562519200 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31564413096 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31572730249 |
1 participant
Summary
Adds a new GB200 DeepSeek-V4-Pro FP4 Dynamo-vLLM MTP AgentX submission using released vLLM v0.27.1, Dynamo 1.3.1, and NVIDIA srt-slurm v1.0.45.
The initial agentx-fast discovery matrix deliberately spans:
This follows the public vLLM GB200 recommendation that the checkpoint needs two NVL4 trays and uses DEP8 as the throughput-oriented baseline. The broad fast pass will be pruned to the measured Pareto frontier before the official full sweep.
Metrics correctness
Validation