perf(qwen3.5-fp4-b200-sglang-mtp): add TEP2 arm

[hshrivastava-droid]

@Klaud-Cold add PR description

[github-actions]

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

---

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

[claude]

LGTM — straightforward benchmark config addition.

Extended reasoning...

Overview

This PR adds a single new search-space arm (tp:2, ep:2, conc-list:[16,32,64], spec-decoding:mtp) to the existing qwen3.5-fp4-b200-sglang-mtp sweep in configs/nvidia-master.yaml, and appends a corresponding perf-changelog.yaml entry documenting the measured gains (+5-9% tok/s/GPU and tok/s/user across concurrencies 16/32/64) and correctness checks (GSM8K strict-match/flexible-extract above threshold, coherence pass, MTP acceptance baseline-like).

Security risks

None. This is a pure benchmark configuration change — no code paths, auth, or data handling are touched.

Level of scrutiny

Low. This follows the exact same mechanical pattern used throughout this file for adding sweep arms (confirmed conc-list syntax matches other entries in the config), and the changelog entry follows the established schema with the required fields (config-keys, scenario-type, description, pr-link) matching the format of adjacent entries.

Other factors

No outstanding review comments in the timeline beyond the standard automated recipe-reminder bot message. The perf-changelog description includes specific before/after benchmark numbers and correctness validation results, consistent with prior similar TEP-arm-addition PRs in the repo history.

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31432304739
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=31432304739

[Ankur-singh]

/reuse-sweep-run 31432304739

[Ankur-singh]

As a PR reviewer and CODEOWNER, I have reviewed this and have:

• Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
• Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
• Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/31432304739
• Verified that this PR passes evals. Please link to GitHub Action workflow that shows this. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/31432304739
• Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
• For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
• Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
• Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; FLOPs at lower precisions is fine, given that the config passes private evals. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
• If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
• If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
• Verified that every single-node vLLM/SGLang recipe in this PR is documented in the official vLLM recipes and/or the SGLang cookbook:
  • I linked the corresponding upstream PR in the vLLM recipe repo or SGLang repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
• Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/<PR_NUMBER>.md — named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section.
• If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.

Additional detail section:

Scope of the change. This PR adds a single search-space row to the existing qwen3.5-fp4-b200-sglang-mtp entry — { tp: 2, ep: 2, conc-list: [16, 32, 64], spec-decoding: mtp } — plus the corresponding perf-changelog.yaml entry. No benchmark script, recipe file, image pin, or launcher is touched.

Validation and eval evidence. Run 31432304739 ran on the exact PR head e3643999, attempt 1, and settled at 21 success / 12 skipped / 0 failures. All three new arm points executed and passed as distinct TP2/EP2 jobs at concurrency 16, 32, and 64, alongside the pre-existing TP4 c4 and TP2 c4/c8/c16/c32/c64 rows. Evals are non-skipped and green for the new arm specifically — eval / … TP2/EP2 mtp c32 and … TP2/EP2 mtp c64 — with collect-evals succeeding. The skipped lanes (multi-node, agentic, agentic-eval) have no matrix entries for this single-node config.

Speculative decoding and chat template. benchmarks/single_node/fixed_seq_len/qwen3.5_fp4_b200_mtp.sh passes --use-chat-template to the serving benchmark, so the MTP acceptance-length distribution is measured against chat-formatted prompts rather than raw tokens. The script runs EAGLE with --speculative-num-steps 3, --speculative-eagle-topk 1, --speculative-num-draft-tokens 4, unchanged by this PR. The golden-AL item is not applicable: this is a fixed-seq-len 8k/1k submission, not an agentic workload, so it uses measured acceptance rather than the simulated synthetic acceptance that item governs.

The new arm is a real configuration, not a relabel. The script consumes expert parallelism directly — --tensor-parallel-size=$TP --data-parallel-size=1 --expert-parallel-size=$EP_SIZE — so ep: 2 genuinely changes the served topology relative to the existing ep: 1 rows, and the sweep surfaced the points as separate TP2/EP2 jobs. Expert parallelism redistributes the same MoE computation across GPUs; it removes no model-architecture FLOPs, and the promoted configuration carries its own passing GSM8K evals in the linked run.

Model and scenario scope. MODELS.md lists Qwen3.5-397B-A17B as active for Single-turn 8k1k limited to fp8/fp4, and records the deprecations as Single-turn 1k1k, Single-turn 1k8k, and all bf16 recipes. This submission is 8k1k NVFP4 and adds no 1k1k lane. MODELS.md also records the engine expectation for this model as the native/upstream SGLang engine with native MTP, which is exactly what this arm exercises on the upstream lmsysorg/sglang:v0.5.14-cu130 image.

Single-node recipe publication — left unchecked, needs a scope call. This is a single-node SGLang submission, so the item is at least arguably in scope, and no upstream recipe or cookbook link has been provided on this PR. Against that: the PR introduces no recipe file and no new serving script — it extends an already-published single-node config's search space with a parallelism variant, and the underlying qwen3.5_fp4_b200_mtp.sh recipe is unchanged. I am not checking this box on my own reading of "recipe in this PR"; if the intent is that a new TP/EP arm on an existing single-node recipe requires a matching cookbook update, this PR needs that upstream link merged first.

No engine or serving-stack patching. The diff is two YAML files. No .patch, git apply, sed, site-packages edit, monkey-patch, container overwrite, or forked wheel, and the pinned upstream image is unchanged.

Signed: Ankur-singh

[Klaud-Cold]

❌❌❌ REJECTED ❌❌❌

@Ankur-singh Blocking issue: this is a single-node SGLang submission, but no merged upstream recipe PR or published cookbook page is linked in the sign-off (the recipe checklist item is unchecked). The new tp: 2, ep: 2 arm changes the deployment-defining parallelism of qwen3.5-fp4-b200-sglang-mtp, so it needs the corresponding SGLang cookbook / vLLM recipes documentation merged upstream and linked before merge.

✅ Check 0 (CODEOWNER): PASS — Ankur-singh owns configs/nvidia-master.yaml per CODEOWNERS; perf-changelog.yaml is catch-all-only and covered.
✅ Check 1 (sweep on in-PR commit): PASS — run 31432304739 on head e364399 has green, executed single-node 8k1k / (incl. TP2/EP2 c16/c32/c64) and eval / check-runs.
✅ Check 2 (evals real): PASS — GSM8K strict 0.968/0.970 on the new TP2/EP2 arm (0.971 on the EP1 reference) vs the 0.94 qwen3.5 bar, on the PR's image lmsysorg/sglang:v0.5.14-cu130.
❌ Check 3 (recipe linked & merged): FAIL — no upstream recipe/cookbook link in the sign-off's additional detail section for this single-node recipe change; the TP2/EP2 parallelism arm is a MAJOR (deployment-defining) config not shown in any linked merged upstream documentation.
✅ Check 4 (reuse command): PASS — /reuse-sweep-run 31432304739 posted by Ankur-singh (COLLABORATOR).
✅ Check 5 (latest checklist): PASS — all current-template items present; both unchecked items are explained in the additional detail section.
✅ Check 6 (upstream image / engine-first): PASS — framework: sglang with upstream lmsysorg/sglang:v0.5.14-cu130 on B200; no non-vLLM/SGLang entry added.
✅ Check 7 (deprecated models): PASS — Qwen3.5 Single-turn 8k1k fp4 is active per MODELS.md as of 2026-08-11.
✅ Check 8 (architecture hacks): PASS — diff is two YAML files; EP2 redistributes the same MoE FLOPs, no overrides.
✅ Check 9 (spec-decode chat template): PASS — qwen3.5_fp4_b200_mtp.sh benchmarks with --use-chat-template.
✅ Check 10 (engine patches): PASS — no patching of the serving stack; pinned image runs as shipped.
➖ Check 11 (agentic golden AL): N/A — fixed-seq-len submission, no agentic spec-decode changes; no synthetic-acceptance knobs present on this non-agentic config.

[Ankur-singh]

/reuse-sweep-run

[Ankur-singh]

/reuse-sweep-run

[Ankur-singh]

/reuse-sweep-run

[Ankur-singh]

/reuse-sweep-run