[AMD] [AgentX] Add MI325X Qwen3.5 FP8 SGLang MTP

[cquil11]

Summary

• Add a single-node MI325X AgentX recipe for Qwen/Qwen3.5-397B-A17B-FP8 with SGLang native EAGLE MTP.
• Use the official stable lmsysorg/sglang:v0.5.16-rocm720-mi30x image, AITER unified attention, FP8 KV cache, and the committed thinking-on golden AL 3.39.
• Select a 32-point official matrix across TP2/EP2, TP4, TEP4, TP8, and TEP8 from the completed 62-point agentx-fast discovery run (60 green; two low-concurrency time-coverage-only failures with complete metrics).
• Keep the submission aggregate and GPU-resident; the measured knees are TP2 c4, TP4/TEP4 c40, and TP8/TEP8 c64.
• Pass the one aggregate backend metrics endpoint explicitly through AIPERF_SERVER_METRICS_URLS and require the sglang: metric prefix.

Validation

• bash -n benchmarks/single_node/agentic/qwen3.5_fp8_mi325x_mtp.sh
• YAML load for configs/amd-master.yaml and perf-changelog.yaml
• 126 matrix validation tests passed
• Changelog validation passed and generated the 32-point official matrix
• Docker Hub manifest verified for v0.5.16-rocm720-mi30x
• MI325X cluster checked: eight idle Slurm nodes and nine online matching runners at PR creation time

[github-actions]

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

---

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

[claude]

🔴 The script appends --use-chat-template to REPLAY_CMD (lines 145-147), but that flag only exists on run_benchmark_serving's fixed-seq-len path, not on the aiperf CLI that build_replay_cmd invokes here. build_replay_cmd already sets --endpoint /v1/chat/completions and --endpoint-type chat, so aiperf will reject the unrecognized flag and fail the replay step for every throughput concurrency arm (62 points) in this recipe. Fix by deleting the REPLAY_CMD+=" --use-chat-template" line.

Extended reasoning...

What the bug is

benchmark_lib.sh's build_replay_cmd constructs REPLAY_CMD as an invocation of $AIPERF_CLI profile --scenario inferencex-agentx-mvp ... (benchmark_lib.sh:1927), where AIPERF_CLI="${AIPERF_VENV}/bin/aiperf" (benchmark_lib.sh:1721) — the external NVIDIA aiperf Python CLI. This is a completely different program from run_benchmark_serving, and --use-chat-template is not one of its recognized arguments.

--use-chat-template is defined only inside run_benchmark_serving's own bash arg-parsing loop (benchmark_lib.sh:520-526), which is consumed by the fixed-seq-len benchmarking path and appended to benchmark_cmd (not REPLAY_CMD) at benchmark_lib.sh:636-638. It has no meaning to aiperf profile.

The code path that triggers it

In qwen3.5_fp8_mi325x_mtp.sh (lines 144-149):

build_replay_cmd "$RESULT_DIR"
REPLAY_CMD+=" --use-chat-template"
run_agentic_replay_and_write_outputs "$RESULT_DIR"

build_replay_cmd already sets --endpoint /v1/chat/completions and --endpoint-type chat (benchmark_lib.sh:1929-1930), which is aiperf's own mechanism for chat-formatted prompts — nothing further is needed. The extra line simply appends a bogus flag to the string that later gets executed as $REPLAY_CMD inside run_agentic_replay_and_write_outputs.

Why nothing prevents it

There's no static validation of REPLAY_CMD before it's executed — it's a plain bash string built up with += and later eval'd/executed directly. aiperf's own argparse-based CLI is what will reject the unknown flag, but only at runtime, once the sweep is actually dispatched to a runner.

Impact

Every non-EVAL_ONLY throughput arm in this recipe (the full 62-point discovery grid across TP2/EP2, TP4, TEP4, TP8, TEP8) takes the else branch that calls build_replay_cmd + this bad append + run_agentic_replay_and_write_outputs. aiperf will exit non-zero on the unrecognized argument, so run_agentic_replay_and_write_outputs will fail for every one of those 62 points. Only EVAL_ONLY=true runs (which take the run_eval branch instead) are unaffected.

Root cause

This is a misapplication of the AGENTS.md rule "Every *_mtp.sh passes --use-chat-template to run_benchmark_serving." That rule is scoped specifically to run_benchmark_serving (the fixed-seq-len benchmarking path). This script is an agentic recipe that never calls run_benchmark_serving — it drives aiperf directly via build_replay_cmd/run_agentic_replay_and_write_outputs. A repo-wide check confirms no other agentic *_mtp.sh script (e.g. qwen3.5_fp8_b300_sglang_mtp.sh, glm5.2_fp4_mi355x_sglang_mtp.sh, dsv4_fp4_mi355x_vllm_mtp.sh) appends --use-chat-template to REPLAY_CMD; they only append legitimate aiperf flags like --server-metrics. In fact, the sibling kimik3_fp4_b300_vllm_mtp.sh carries an explicit comment noting this exact distinction and stating there's "Nothing to add here" for agentic recipes.

Step-by-step proof

1. Script reaches the else branch (non-EVAL_ONLY) at line 143.
2. build_replay_cmd "$RESULT_DIR" sets REPLAY_CMD="$AIPERF_VENV/bin/aiperf profile --scenario inferencex-agentx-mvp --url http://localhost:$PORT --endpoint /v1/chat/completions --endpoint-type chat --streaming --model $MODEL --concurrency $CONC --benchmark-duration $duration --stats-interval 30 ...".
3. Line 147 appends: REPLAY_CMD+=" --use-chat-template".
4. run_agentic_replay_and_write_outputs "$RESULT_DIR" executes $REPLAY_CMD, i.e. runs aiperf profile ... --use-chat-template.
5. aiperf's argparse-based CLI does not recognize --use-chat-template and exits non-zero, causing run_agentic_replay_and_write_outputs to report a failed replay for that concurrency arm.
6. This repeats identically for all 62 concurrency points in the qwen3.5-fp8-mi325x-sglang-agentic-mtp grid.

Fix

Simply delete line 147 (REPLAY_CMD+=" --use-chat-template"); build_replay_cmd's existing --endpoint-type chat already covers chat formatting for the aiperf replay.

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31544042402
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=31544042402

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31561404142
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=31561404142