diff --git a/KLAUD_DEBUG.md b/KLAUD_DEBUG.md index 6760b37f88..091b17fc00 100644 --- a/KLAUD_DEBUG.md +++ b/KLAUD_DEBUG.md @@ -52,7 +52,7 @@ and waits for the PR checks automatically. **Fix:** in `benchmarks/single_node/.sh`, either: 1. **Lower `--gpu-memory-utilization`** (`0.95 → 0.90`, sometimes 0.85). Matches the H100/H200/B200 NVIDIA pattern. Smallest blast radius. -2. **Disable the profiler entirely** for cases where lowering isn't enough: `export VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0` before `vllm serve`. Matches `benchmarks/single_node/agentic/kimik2.5_fp4_b200.sh:65`. +2. **Disable the profiler entirely** for cases where lowering isn't enough: `export VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0` before `vllm serve`. Matches `benchmarks/single_node/agentic/deprecated/kimik2.5_fp4_b200.sh:65`. Seen on: #1395 (kimik2.5-fp4-b200-vllm — needed env var), #1403 (gptoss-fp4-mi300x-vllm — needed 0.90), #1461 (dsv4-fp8-h200-vllm — needed 0.90). diff --git a/MODELS.md b/MODELS.md index 7775f717fb..d855c07a39 100644 --- a/MODELS.md +++ b/MODELS.md @@ -12,6 +12,8 @@ InferenceX-e2e runs on a fixed, limited pool of GPUs and is maintained by a smal **Monday, August 3, 2026** is the last day for the scenarios, precisions, and recipe variants listed below; they are deprecated after that date. +**Partially enacted on 2026-08-04** in [#2493](https://github.com/SemiAnalysisAI/InferenceX/pull/2493): the scenario and precision retirements in the first table were carried out — 54 config keys removed from the active master configs and archived under [`configs/deprecated/`](configs/deprecated/), with their benchmark scripts moved to the sibling `deprecated/` directories. The speculative-decoding A/B retirements in the second table are **not yet enacted**; see the note under that table. + Scenario and precision retirements: | Model | Deprecated | Remains | @@ -30,6 +32,8 @@ Speculative-decoding A/B retirements — in each pair below the spec-decode arm | GLM-5.2 (`glm5.2`) | Agentic coding, non-MTP | Agentic coding, MTP | | Kimi-K3 (`kimik3`) | Agentic coding, non-DSpark — deprecated from day 0 | Agentic coding, DSpark | +**Status: not yet enacted.** Every non-spec-decode agentic arm above still runs. Removing them today would leave MiniMax-M3 and GLM-5.2 with no active config at all — their EAGLE3 and MTP agentic arms have not landed yet — and would drop all AMD and all SGLang agentic coverage for DeepSeek-V4-Pro and Qwen3.5, neither of which has an MTP sibling on those platforms. This round runs once the replacement arms exist. + **Going forward we no longer benchmark non-spec-decode versus spec-decode as an A/B.** The non-spec-decode arm existed as a neutral baseline back when acceptance length wasn't standardized. That is now solved: [`golden_al_distribution/`](golden_al_distribution/) commits one golden acceptance-length curve per model, thinking mode, and draft length, measured on the SPEED-Bench `coding` category, and AgentX pins every submission to that curve through synthetic acceptance (vLLM `synthetic_acceptance_length`, SGLang `SGLANG_SIMULATE_ACC_LEN`, TensorRT-LLM `TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS`, etc). With a fair, engine-independent acceptance target in place, spec-decode results are directly comparable on their own and a separate non-spec-decode track is redundant. Agentic coding recipes are therefore run and published with speculative decoding enabled only — MTP, EAGLE/EAGLE3, DSpark, or whatever draft method the model ships — and the non-spec-decode arm is neither run nor published. New models are onboarded that way from day 0, as Kimi-K3 is. ### Thursday, August 6, 2026 @@ -51,13 +55,13 @@ Speculative-decoding A/B retirements — in each pair below the spec-decode arm |---|---|---|---|---| | Qwen3.8 2.4T | `qwen3.8` | TBD | Agentic coding | | | Kimi-K3 | `kimik3` | 2026-07-27 ([#2391](https://github.com/SemiAnalysisAI/InferenceX/pull/2391)) | Agentic coding (DSpark only) | Agentic coding non-DSpark arm (deprecated from day 0) | -| GLM-5.2 | `glm5.2` | 2026-07-18 ([#2268](https://github.com/SemiAnalysisAI/InferenceX/pull/2268)) | Agentic coding (MTP only from 2026-08-03) | | -| MiniMax-M3 | `minimaxm3` | 2026-06-12 ([#1724](https://github.com/SemiAnalysisAI/InferenceX/pull/1724)) | Single-turn 8k1k (until 2026-08-03), Agentic coding (EAGLE3 only from 2026-08-03) | Single-turn 1k1k | -| DeepSeek-V4-Pro | `dsv4` | 2026-04-24 ([#1130](https://github.com/SemiAnalysisAI/InferenceX/pull/1130)) | Single-turn 8k1k, Agentic coding (MTP only from 2026-08-03) | Single-turn 1k1k | +| GLM-5.2 | `glm5.2` | 2026-07-18 ([#2268](https://github.com/SemiAnalysisAI/InferenceX/pull/2268)) | Agentic coding (MTP-only pending — the non-MTP arm still runs; see the Deprecation Notice) | | +| MiniMax-M3 | `minimaxm3` | 2026-06-12 ([#1724](https://github.com/SemiAnalysisAI/InferenceX/pull/1724)) | Agentic coding | Single-turn 1k1k, Single-turn 8k1k (removed 2026-08-04, [#2493](https://github.com/SemiAnalysisAI/InferenceX/pull/2493)) | +| DeepSeek-V4-Pro | `dsv4` | 2026-04-24 ([#1130](https://github.com/SemiAnalysisAI/InferenceX/pull/1130)) | Single-turn 8k1k, Agentic coding (MTP-only pending — the non-MTP arm still runs; see the Deprecation Notice) | Single-turn 1k1k | | GLM-5 / GLM-5.1 | `glm5`, `glm5.1` | 2026-03-06 ([#762](https://github.com/SemiAnalysisAI/InferenceX/pull/762)); GLM-5.1 added 2026-04-21 ([#1098](https://github.com/SemiAnalysisAI/InferenceX/pull/1098)) | — (retired 2026-07-18, [#2276](https://github.com/SemiAnalysisAI/InferenceX/pull/2276)) | Single-turn 1k1k, Single-turn 1k8k (GLM-5 only), Single-turn 8k1k | | MiniMax-M2.5/2.7 | `minimaxm2.5` | 2026-02-18 ([#755](https://github.com/SemiAnalysisAI/InferenceX/pull/755)) | — (retired 2026-06-20, [#1874](https://github.com/SemiAnalysisAI/InferenceX/pull/1874)) | Single-turn 1k1k, Single-turn 1k8k, Single-turn 8k1k | -| Kimi-K2.5/2.6/2.7-Code | `kimik2.5` | 2026-02-17 ([#734](https://github.com/SemiAnalysisAI/InferenceX/pull/734)) | Single-turn 8k1k (until 2026-08-06), Agentic coding (until 2026-08-03) — fully retired after 2026-08-06 | Single-turn 1k1k, Single-turn 1k8k | -| Qwen3.5-397B-A17B | `qwen3.5` | 2026-02-16 ([#704](https://github.com/SemiAnalysisAI/InferenceX/pull/704)) | Single-turn 8k1k, Agentic coding (MTP only from 2026-08-03); fp8/fp4 only — bf16 recipes retired 2026-08-03 | Single-turn 1k1k, Single-turn 1k8k | +| Kimi-K2.5/2.6/2.7-Code | `kimik2.5` | 2026-02-17 ([#734](https://github.com/SemiAnalysisAI/InferenceX/pull/734)) | Single-turn 8k1k (until 2026-08-06) — fully retired after 2026-08-06 | Single-turn 1k1k, Single-turn 1k8k, Agentic coding (removed 2026-08-04, [#2493](https://github.com/SemiAnalysisAI/InferenceX/pull/2493)) | +| Qwen3.5-397B-A17B | `qwen3.5` | 2026-02-16 ([#704](https://github.com/SemiAnalysisAI/InferenceX/pull/704)) | Single-turn 8k1k, Agentic coding; fp8/fp4 only | Single-turn 1k1k, Single-turn 1k8k, all bf16 recipes (removed 2026-08-04, [#2493](https://github.com/SemiAnalysisAI/InferenceX/pull/2493)) | | gpt-oss-120b | `gptoss` | 2025-09-09 | — (retired 2026-07-06, [#2101](https://github.com/SemiAnalysisAI/InferenceX/pull/2101)) | Single-turn 1k1k, Single-turn 1k8k, Single-turn 8k1k | | DeepSeek-R1-0528 | `dsr1` | 2025-08-13 | Single-turn 8k1k | Single-turn 1k1k, Single-turn 1k8k | | Llama-3.1-70B-Instruct | `llama70b` | 2025-08-12 | — (retired 2025-10-29, [#149](https://github.com/SemiAnalysisAI/InferenceX/pull/149)) | Single-turn 1k1k, Single-turn 1k8k, Single-turn 8k1k [^1] | diff --git a/MODELS_zh.md b/MODELS_zh.md index 26c4bf7732..71dca05b1e 100644 --- a/MODELS_zh.md +++ b/MODELS_zh.md @@ -12,6 +12,8 @@ InferenceX-e2e 运行在数量固定且有限的 GPU 资源池上，并由一支 **2026 年 8 月 3 日（星期一）**为下列场景、精度与配方变体的最后运行日，此后即告弃用。 +**已于 2026 年 8 月 4 日部分执行**（[#2493](https://github.com/SemiAnalysisAI/InferenceX/pull/2493)）：第一张表中的场景与精度下线已完成 —— 54 个配置键从启用的 master 配置中移除并归档至 [`configs/deprecated/`](configs/deprecated/)，其基准测试脚本亦移入同级 `deprecated/` 目录。第二张表中的投机解码 A/B 下线**尚未执行**，详见该表下方说明。 + 场景与精度下线： | 模型 | 弃用内容 | 保留内容 | @@ -30,6 +32,8 @@ InferenceX-e2e 运行在数量固定且有限的 GPU 资源池上，并由一支 | GLM-5.2（`glm5.2`） | 智能体编码，非 MTP | 智能体编码，MTP | | Kimi-K3（`kimik3`） | 智能体编码，非 DSpark —— 自第 0 天（day 0）起即弃用 | 智能体编码，DSpark | +**状态：尚未执行。**上表中所有非投机解码智能体分支仍在运行。若此刻移除，MiniMax-M3 与 GLM-5.2 将没有任何启用配置（其 EAGLE3 与 MTP 智能体分支尚未合入），并会使 DeepSeek-V4-Pro 与 Qwen3.5 在 AMD 及 SGLang 上的智能体覆盖全部消失 —— 这些平台/引擎组合均无对应的 MTP 分支。待替代分支就绪后再执行本轮下线。 + **今后我们不再以 A/B 对照的方式基准测试「非投机解码 vs 投机解码」。**当初保留非投机解码分支，是把它当作中立基线：那时接受长度（AL）完全取决于提交方草稿头（draft head）的实际水平，导致各家投机解码数据之间无法横向比较。这一问题现已解决：[`golden_al_distribution/`](golden_al_distribution/) 为每个模型、thinking 模式与草稿长度各提交了一条黄金 AL 曲线，均在 SPEED-Bench `coding` 类别上测得；AgentX 通过合成接受（synthetic acceptance）将所有提交锁定到该曲线（vLLM 用 `synthetic_acceptance_length`，SGLang 用 `SGLANG_SIMULATE_ACC_LEN`，TensorRT-LLM 用 `TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS`，等等）。既然已有公平且与引擎无关的接受目标，投机解码结果本身即可直接横向比较，单独保留一条非投机解码赛道已属冗余。因此，智能体编码配方一律仅在启用投机解码的条件下运行与发布 —— 具体为 MTP、EAGLE/EAGLE3、DSpark，或该模型自带的任何草稿方法 —— 非投机解码分支既不运行也不发布。新模型自第 0 天起即按此方式接入，Kimi-K3 即为一例。 ### 2026 年 8 月 6 日（星期四） @@ -51,13 +55,13 @@ InferenceX-e2e 运行在数量固定且有限的 GPU 资源池上，并由一支 |---|---|---|---|---| | Qwen3.8 2.4T | `qwen3.8` | 待定 | 智能体编码 | | | Kimi-K3 | `kimik3` | 2026-07-27 ([#2391](https://github.com/SemiAnalysisAI/InferenceX/pull/2391)) | 智能体编码（仅 DSpark） | 智能体编码非 DSpark 分支（自第 0 天起弃用） | -| GLM-5.2 | `glm5.2` | 2026-07-18（[#2268](https://github.com/SemiAnalysisAI/InferenceX/pull/2268)） | 智能体编码（自 2026-08-03 起仅 MTP） | | -| MiniMax-M3 | `minimaxm3` | 2026-06-12（[#1724](https://github.com/SemiAnalysisAI/InferenceX/pull/1724)） | 单轮 8k1k（至 2026-08-03）、智能体编码（自 2026-08-03 起仅 EAGLE3） | 单轮 1k1k | -| DeepSeek-V4-Pro | `dsv4` | 2026-04-24（[#1130](https://github.com/SemiAnalysisAI/InferenceX/pull/1130)） | 单轮 8k1k、智能体编码（自 2026-08-03 起仅 MTP） | 单轮 1k1k | +| GLM-5.2 | `glm5.2` | 2026-07-18（[#2268](https://github.com/SemiAnalysisAI/InferenceX/pull/2268)） | 智能体编码（「仅 MTP」待执行 —— 非 MTP 分支仍在运行，见弃用公告） | | +| MiniMax-M3 | `minimaxm3` | 2026-06-12（[#1724](https://github.com/SemiAnalysisAI/InferenceX/pull/1724)） | 智能体编码 | 单轮 1k1k、单轮 8k1k（2026-08-04 移除，[#2493](https://github.com/SemiAnalysisAI/InferenceX/pull/2493)） | +| DeepSeek-V4-Pro | `dsv4` | 2026-04-24（[#1130](https://github.com/SemiAnalysisAI/InferenceX/pull/1130)） | 单轮 8k1k、智能体编码（「仅 MTP」待执行 —— 非 MTP 分支仍在运行，见弃用公告） | 单轮 1k1k | | GLM-5 / GLM-5.1 | `glm5`、`glm5.1` | 2026-03-06（[#762](https://github.com/SemiAnalysisAI/InferenceX/pull/762)）；GLM-5.1 于 2026-04-21 加入（[#1098](https://github.com/SemiAnalysisAI/InferenceX/pull/1098)） | —（2026-07-18 退役，[#2276](https://github.com/SemiAnalysisAI/InferenceX/pull/2276)） | 单轮 1k1k、单轮 1k8k（仅 GLM-5）、单轮 8k1k | | MiniMax-M2.5/2.7 | `minimaxm2.5` | 2026-02-18（[#755](https://github.com/SemiAnalysisAI/InferenceX/pull/755)） | —（2026-06-20 退役，[#1874](https://github.com/SemiAnalysisAI/InferenceX/pull/1874)） | 单轮 1k1k、单轮 1k8k、单轮 8k1k | -| Kimi-K2.5/2.6/2.7-Code | `kimik2.5` | 2026-02-17（[#734](https://github.com/SemiAnalysisAI/InferenceX/pull/734)） | 单轮 8k1k（至 2026-08-06）、智能体编码（至 2026-08-03）—— 2026-08-06 后完全退役 | 单轮 1k1k、单轮 1k8k | -| Qwen3.5-397B-A17B | `qwen3.5` | 2026-02-16（[#704](https://github.com/SemiAnalysisAI/InferenceX/pull/704)） | 单轮 8k1k、智能体编码（自 2026-08-03 起仅 MTP）；仅 fp8/fp4 —— bf16 配方于 2026-08-03 下线 | 单轮 1k1k、单轮 1k8k | +| Kimi-K2.5/2.6/2.7-Code | `kimik2.5` | 2026-02-17（[#734](https://github.com/SemiAnalysisAI/InferenceX/pull/734)） | 单轮 8k1k（至 2026-08-06）—— 2026-08-06 后完全退役 | 单轮 1k1k、单轮 1k8k、智能体编码（2026-08-04 移除，[#2493](https://github.com/SemiAnalysisAI/InferenceX/pull/2493)） | +| Qwen3.5-397B-A17B | `qwen3.5` | 2026-02-16（[#704](https://github.com/SemiAnalysisAI/InferenceX/pull/704)） | 单轮 8k1k、智能体编码；仅 fp8/fp4 | 单轮 1k1k、单轮 1k8k、全部 bf16 配方（2026-08-04 移除，[#2493](https://github.com/SemiAnalysisAI/InferenceX/pull/2493)） | | gpt-oss-120b | `gptoss` | 2025-09-09 | —（2026-07-06 退役，[#2101](https://github.com/SemiAnalysisAI/InferenceX/pull/2101)） | 单轮 1k1k、单轮 1k8k、单轮 8k1k | | DeepSeek-R1-0528 | `dsr1` | 2025-08-13 | 单轮 8k1k | 单轮 1k1k、单轮 1k8k | | Llama-3.1-70B-Instruct | `llama70b` | 2025-08-12 | —（2025-10-29 退役，[#149](https://github.com/SemiAnalysisAI/InferenceX/pull/149)） | 单轮 1k1k、单轮 1k8k、单轮 8k1k [^1] | diff --git a/benchmarks/multi_node/amd_utils/models_vllm.yaml b/benchmarks/multi_node/amd_utils/models_vllm.yaml index 642fa1e3ed..79d4f4d8f8 100644 --- a/benchmarks/multi_node/amd_utils/models_vllm.yaml +++ b/benchmarks/multi_node/amd_utils/models_vllm.yaml @@ -58,7 +58,7 @@ MiniMax-M3-MXFP8: # --block-size 128 is mandatory (MSA sparse/index cache); text-only benchmark # so --language-model-only frees the vision encoder. gfx950 uses FP8 KV cache. # Serve flags + env kept in sync with the single-node recipe - # benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_mi355x.sh: + # benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp8_mi355x.sh: # --moe-backend aiter (AITER fused MoE), --linear-backend emulation, and the # larger --max-num-batched-tokens 32768 prefill budget; plus the AITER # router-append shared-experts fusion (self-disables under EP; the sweep is diff --git a/benchmarks/multi_node/minimaxm3_fp4_mi355x_atom-disagg.sh b/benchmarks/multi_node/deprecated/minimaxm3_fp4_mi355x_atom-disagg.sh similarity index 97% rename from benchmarks/multi_node/minimaxm3_fp4_mi355x_atom-disagg.sh rename to benchmarks/multi_node/deprecated/minimaxm3_fp4_mi355x_atom-disagg.sh index 1505b905de..e86e044a45 100644 --- a/benchmarks/multi_node/minimaxm3_fp4_mi355x_atom-disagg.sh +++ b/benchmarks/multi_node/deprecated/minimaxm3_fp4_mi355x_atom-disagg.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash -source "$(dirname "$0")/../benchmark_lib.sh" +source "$(dirname "$0")/../../benchmark_lib.sh" check_env_vars \ CONC_LIST \ diff --git a/benchmarks/multi_node/minimaxm3_fp4_mi355x_vllm-disagg.sh b/benchmarks/multi_node/deprecated/minimaxm3_fp4_mi355x_vllm-disagg.sh similarity index 96% rename from benchmarks/multi_node/minimaxm3_fp4_mi355x_vllm-disagg.sh rename to benchmarks/multi_node/deprecated/minimaxm3_fp4_mi355x_vllm-disagg.sh index 2658b8615c..ac0f0a27d7 100755 --- a/benchmarks/multi_node/minimaxm3_fp4_mi355x_vllm-disagg.sh +++ b/benchmarks/multi_node/deprecated/minimaxm3_fp4_mi355x_vllm-disagg.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash -source "$(dirname "$0")/../benchmark_lib.sh" +source "$(dirname "$0")/../../benchmark_lib.sh" check_env_vars \ CONC_LIST \ diff --git a/benchmarks/multi_node/minimaxm3_fp8_mi355x_atom-disagg.sh b/benchmarks/multi_node/deprecated/minimaxm3_fp8_mi355x_atom-disagg.sh similarity index 97% rename from benchmarks/multi_node/minimaxm3_fp8_mi355x_atom-disagg.sh rename to benchmarks/multi_node/deprecated/minimaxm3_fp8_mi355x_atom-disagg.sh index 1505b905de..e86e044a45 100644 --- a/benchmarks/multi_node/minimaxm3_fp8_mi355x_atom-disagg.sh +++ b/benchmarks/multi_node/deprecated/minimaxm3_fp8_mi355x_atom-disagg.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash -source "$(dirname "$0")/../benchmark_lib.sh" +source "$(dirname "$0")/../../benchmark_lib.sh" check_env_vars \ CONC_LIST \ diff --git a/benchmarks/multi_node/minimaxm3_fp8_mi355x_vllm-disagg.sh b/benchmarks/multi_node/deprecated/minimaxm3_fp8_mi355x_vllm-disagg.sh similarity index 97% rename from benchmarks/multi_node/minimaxm3_fp8_mi355x_vllm-disagg.sh rename to benchmarks/multi_node/deprecated/minimaxm3_fp8_mi355x_vllm-disagg.sh index f54940e294..91628d632f 100644 --- a/benchmarks/multi_node/minimaxm3_fp8_mi355x_vllm-disagg.sh +++ b/benchmarks/multi_node/deprecated/minimaxm3_fp8_mi355x_vllm-disagg.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash -source "$(dirname "$0")/../benchmark_lib.sh" +source "$(dirname "$0")/../../benchmark_lib.sh" check_env_vars \ CONC_LIST \ diff --git a/benchmarks/single_node/agentic/kimik2.5_fp4_b200.sh b/benchmarks/single_node/agentic/deprecated/kimik2.5_fp4_b200.sh similarity index 99% rename from benchmarks/single_node/agentic/kimik2.5_fp4_b200.sh rename to benchmarks/single_node/agentic/deprecated/kimik2.5_fp4_b200.sh index c4fd4a8903..e75e3bff2f 100755 --- a/benchmarks/single_node/agentic/kimik2.5_fp4_b200.sh +++ b/benchmarks/single_node/agentic/deprecated/kimik2.5_fp4_b200.sh @@ -9,7 +9,7 @@ set -x # # KV_OFFLOADING=dram requires KV_OFFLOAD_BACKEND=lmcache. -source "$(dirname "$0")/../../benchmark_lib.sh" +source "$(dirname "$0")/../../../benchmark_lib.sh" check_env_vars MODEL TP CONC KV_OFFLOADING TOTAL_CPU_DRAM_GB RESULT_DIR DURATION diff --git a/benchmarks/single_node/agentic/kimik2.5_fp4_b300.sh b/benchmarks/single_node/agentic/deprecated/kimik2.5_fp4_b300.sh similarity index 98% rename from benchmarks/single_node/agentic/kimik2.5_fp4_b300.sh rename to benchmarks/single_node/agentic/deprecated/kimik2.5_fp4_b300.sh index e2731f7503..94636fad41 100755 --- a/benchmarks/single_node/agentic/kimik2.5_fp4_b300.sh +++ b/benchmarks/single_node/agentic/deprecated/kimik2.5_fp4_b300.sh @@ -9,7 +9,7 @@ set -x # # KV_OFFLOADING=dram requires KV_OFFLOAD_BACKEND=vllm-simple. -source "$(dirname "$0")/../../benchmark_lib.sh" +source "$(dirname "$0")/../../../benchmark_lib.sh" check_env_vars MODEL TP CONC KV_OFFLOADING TOTAL_CPU_DRAM_GB RESULT_DIR DURATION diff --git a/benchmarks/single_node/agentic/kimik2.5_fp4_b300_mtp.sh b/benchmarks/single_node/agentic/deprecated/kimik2.5_fp4_b300_mtp.sh similarity index 98% rename from benchmarks/single_node/agentic/kimik2.5_fp4_b300_mtp.sh rename to benchmarks/single_node/agentic/deprecated/kimik2.5_fp4_b300_mtp.sh index 484a8f55f1..20f6ad25d9 100755 --- a/benchmarks/single_node/agentic/kimik2.5_fp4_b300_mtp.sh +++ b/benchmarks/single_node/agentic/deprecated/kimik2.5_fp4_b300_mtp.sh @@ -2,7 +2,7 @@ set -euo pipefail set -x -source "$(dirname "$0")/../../benchmark_lib.sh" +source "$(dirname "$0")/../../../benchmark_lib.sh" check_env_vars MODEL TP CONC KV_OFFLOADING TOTAL_CPU_DRAM_GB RESULT_DIR DURATION diff --git a/benchmarks/single_node/agentic/kimik2.5_fp4_mi355x.sh b/benchmarks/single_node/agentic/deprecated/kimik2.5_fp4_mi355x.sh similarity index 98% rename from benchmarks/single_node/agentic/kimik2.5_fp4_mi355x.sh rename to benchmarks/single_node/agentic/deprecated/kimik2.5_fp4_mi355x.sh index 75e952c4e3..b0d5f801c8 100755 --- a/benchmarks/single_node/agentic/kimik2.5_fp4_mi355x.sh +++ b/benchmarks/single_node/agentic/deprecated/kimik2.5_fp4_mi355x.sh @@ -10,7 +10,7 @@ set -x # KV_OFFLOADING=dram requires KV_OFFLOAD_BACKEND=vllm-native. -source "$(dirname "$0")/../../benchmark_lib.sh" +source "$(dirname "$0")/../../../benchmark_lib.sh" check_env_vars MODEL TP CONC KV_OFFLOADING TOTAL_CPU_DRAM_GB RESULT_DIR DURATION EP_SIZE diff --git a/benchmarks/single_node/agentic/kimik2.5_int4_b200.sh b/benchmarks/single_node/agentic/deprecated/kimik2.5_int4_b200.sh similarity index 97% rename from benchmarks/single_node/agentic/kimik2.5_int4_b200.sh rename to benchmarks/single_node/agentic/deprecated/kimik2.5_int4_b200.sh index f7e50077ab..b216ec7b4c 100755 --- a/benchmarks/single_node/agentic/kimik2.5_int4_b200.sh +++ b/benchmarks/single_node/agentic/deprecated/kimik2.5_int4_b200.sh @@ -7,7 +7,7 @@ set -x # Required env vars: # MODEL, TP, CONC, RESULT_DIR -source "$(dirname "$0")/../../benchmark_lib.sh" +source "$(dirname "$0")/../../../benchmark_lib.sh" check_env_vars MODEL TP CONC KV_OFFLOADING TOTAL_CPU_DRAM_GB RESULT_DIR DURATION diff --git a/benchmarks/single_node/agentic/kimik2.5_int4_h100.sh b/benchmarks/single_node/agentic/deprecated/kimik2.5_int4_h100.sh similarity index 97% rename from benchmarks/single_node/agentic/kimik2.5_int4_h100.sh rename to benchmarks/single_node/agentic/deprecated/kimik2.5_int4_h100.sh index f90dba9ed9..9f6ab9066b 100755 --- a/benchmarks/single_node/agentic/kimik2.5_int4_h100.sh +++ b/benchmarks/single_node/agentic/deprecated/kimik2.5_int4_h100.sh @@ -7,7 +7,7 @@ set -x # Required env vars: # MODEL, TP, CONC, KV_OFFLOADING, TOTAL_CPU_DRAM_GB, RESULT_DIR -source "$(dirname "$0")/../../benchmark_lib.sh" +source "$(dirname "$0")/../../../benchmark_lib.sh" check_env_vars MODEL TP CONC KV_OFFLOADING TOTAL_CPU_DRAM_GB RESULT_DIR DURATION diff --git a/benchmarks/single_node/agentic/kimik2.5_int4_h200.sh b/benchmarks/single_node/agentic/deprecated/kimik2.5_int4_h200.sh similarity index 98% rename from benchmarks/single_node/agentic/kimik2.5_int4_h200.sh rename to benchmarks/single_node/agentic/deprecated/kimik2.5_int4_h200.sh index f73aaac17f..683dcd254d 100755 --- a/benchmarks/single_node/agentic/kimik2.5_int4_h200.sh +++ b/benchmarks/single_node/agentic/deprecated/kimik2.5_int4_h200.sh @@ -7,7 +7,7 @@ set -x # Required env vars: # MODEL, TP, CONC, KV_OFFLOADING, TOTAL_CPU_DRAM_GB, RESULT_DIR -source "$(dirname "$0")/../../benchmark_lib.sh" +source "$(dirname "$0")/../../../benchmark_lib.sh" check_env_vars MODEL TP CONC KV_OFFLOADING TOTAL_CPU_DRAM_GB RESULT_DIR DURATION diff --git a/benchmarks/single_node/agentic/qwen3.5_bf16_b200.sh b/benchmarks/single_node/agentic/deprecated/qwen3.5_bf16_b200.sh similarity index 97% rename from benchmarks/single_node/agentic/qwen3.5_bf16_b200.sh rename to benchmarks/single_node/agentic/deprecated/qwen3.5_bf16_b200.sh index 7fd9d40034..36f47e9147 100755 --- a/benchmarks/single_node/agentic/qwen3.5_bf16_b200.sh +++ b/benchmarks/single_node/agentic/deprecated/qwen3.5_bf16_b200.sh @@ -7,7 +7,7 @@ set -x # Required env vars: # MODEL, TP, CONC, RESULT_DIR -source "$(dirname "$0")/../../benchmark_lib.sh" +source "$(dirname "$0")/../../../benchmark_lib.sh" check_env_vars MODEL TP CONC RESULT_DIR DURATION EP_SIZE diff --git a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp4_b200.sh b/benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp4_b200.sh similarity index 97% rename from benchmarks/single_node/fixed_seq_len/minimaxm3_fp4_b200.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp4_b200.sh index 0b36b5ec26..c28a3f1f63 100755 --- a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp4_b200.sh +++ b/benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp4_b200.sh @@ -5,7 +5,7 @@ # checkpoint. MiniMax-M3 modelopt NVFP4 support (vllm-project/vllm PR #46380) is # baked into the perf container image, so no runtime patch is needed. -source "$(dirname "$0")/../../benchmark_lib.sh" +source "$(dirname "$0")/../../../benchmark_lib.sh" check_env_vars \ MODEL \ diff --git a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp4_b200_mtp.sh b/benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp4_b200_mtp.sh similarity index 98% rename from benchmarks/single_node/fixed_seq_len/minimaxm3_fp4_b200_mtp.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp4_b200_mtp.sh index 8f767954bf..a60f478cb2 100755 --- a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp4_b200_mtp.sh +++ b/benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp4_b200_mtp.sh @@ -6,7 +6,7 @@ # (vllm-project/vllm PR #46380) is baked into the perf container image, so no # runtime patch is needed. -source "$(dirname "$0")/../../benchmark_lib.sh" +source "$(dirname "$0")/../../../benchmark_lib.sh" check_env_vars \ MODEL \ diff --git a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp4_b300.sh b/benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp4_b300.sh similarity index 98% rename from benchmarks/single_node/fixed_seq_len/minimaxm3_fp4_b300.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp4_b300.sh index 17155e1992..435fc25369 100755 --- a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp4_b300.sh +++ b/benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp4_b300.sh @@ -5,7 +5,7 @@ # checkpoint. MiniMax-M3 modelopt NVFP4 support (vllm-project/vllm PR #46380) is # baked into the perf container image, so no runtime patch is needed. -source "$(dirname "$0")/../../benchmark_lib.sh" +source "$(dirname "$0")/../../../benchmark_lib.sh" check_env_vars \ MODEL \ diff --git a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp4_b300_mtp.sh b/benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp4_b300_mtp.sh similarity index 98% rename from benchmarks/single_node/fixed_seq_len/minimaxm3_fp4_b300_mtp.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp4_b300_mtp.sh index 2c515917c1..3a89f14b82 100755 --- a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp4_b300_mtp.sh +++ b/benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp4_b300_mtp.sh @@ -6,7 +6,7 @@ # (vllm-project/vllm PR #46380) is baked into the perf container image, so no # runtime patch is needed. -source "$(dirname "$0")/../../benchmark_lib.sh" +source "$(dirname "$0")/../../../benchmark_lib.sh" check_env_vars \ MODEL \ diff --git a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp4_mi355x_atom.sh b/benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp4_mi355x_atom.sh similarity index 97% rename from benchmarks/single_node/fixed_seq_len/minimaxm3_fp4_mi355x_atom.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp4_mi355x_atom.sh index dfb1e7a9fc..c35ec86d63 100644 --- a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp4_mi355x_atom.sh +++ b/benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp4_mi355x_atom.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash -source "$(dirname "$0")/../../benchmark_lib.sh" +source "$(dirname "$0")/../../../benchmark_lib.sh" check_env_vars \ MODEL \ diff --git a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp4_mi355x_atom_mtp.sh b/benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp4_mi355x_atom_mtp.sh similarity index 98% rename from benchmarks/single_node/fixed_seq_len/minimaxm3_fp4_mi355x_atom_mtp.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp4_mi355x_atom_mtp.sh index 4ef60e71ef..1c60c8d2f9 100644 --- a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp4_mi355x_atom_mtp.sh +++ b/benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp4_mi355x_atom_mtp.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash -source "$(dirname "$0")/../../benchmark_lib.sh" +source "$(dirname "$0")/../../../benchmark_lib.sh" check_env_vars \ MODEL \ diff --git a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp4_mi355x_vllm.sh b/benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp4_mi355x_vllm.sh similarity index 98% rename from benchmarks/single_node/fixed_seq_len/minimaxm3_fp4_mi355x_vllm.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp4_mi355x_vllm.sh index 6254674ecd..f5ac074078 100755 --- a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp4_mi355x_vllm.sh +++ b/benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp4_mi355x_vllm.sh @@ -11,7 +11,7 @@ # 64/128/256. # * fp8 KV cache (--kv-cache-dtype fp8). -source "$(dirname "$0")/../../benchmark_lib.sh" +source "$(dirname "$0")/../../../benchmark_lib.sh" check_env_vars \ MODEL \ diff --git a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp4_mi355x_vllm_mtp.sh b/benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp4_mi355x_vllm_mtp.sh similarity index 98% rename from benchmarks/single_node/fixed_seq_len/minimaxm3_fp4_mi355x_vllm_mtp.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp4_mi355x_vllm_mtp.sh index 8a15b8c892..69f1d71fa7 100755 --- a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp4_mi355x_vllm_mtp.sh +++ b/benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp4_mi355x_vllm_mtp.sh @@ -6,7 +6,7 @@ # Inferact/MiniMax-M3-EAGLE3. The pinned nightly includes upstream AMD # MiniMax-M3 SupportsEagle3 support, so no runtime model patch is needed. -source "$(dirname "$0")/../../benchmark_lib.sh" +source "$(dirname "$0")/../../../benchmark_lib.sh" check_env_vars \ MODEL \ diff --git a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_b200.sh b/benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp8_b200.sh similarity index 98% rename from benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_b200.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp8_b200.sh index 1b4e70fd9f..795333cbfa 100755 --- a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_b200.sh +++ b/benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp8_b200.sh @@ -8,7 +8,7 @@ # dp-attn=true maps to DP×EP (DEP) per the recipe's "DP8 + Expert Parallel" # layout; ep>1 maps to TP+EP (TEP). -source "$(dirname "$0")/../../benchmark_lib.sh" +source "$(dirname "$0")/../../../benchmark_lib.sh" check_env_vars \ MODEL \ diff --git a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_b200_mtp.sh b/benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp8_b200_mtp.sh similarity index 98% rename from benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_b200_mtp.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp8_b200_mtp.sh index 9bb1dd777d..c8844a35b9 100644 --- a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_b200_mtp.sh +++ b/benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp8_b200_mtp.sh @@ -13,7 +13,7 @@ # The target uses the FlashInfer TRT-LLM attention path. The EAGLE3-GQA drafter # is pinned separately to FLASH_ATTN. -source "$(dirname "$0")/../../benchmark_lib.sh" +source "$(dirname "$0")/../../../benchmark_lib.sh" check_env_vars \ MODEL \ diff --git a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_b300.sh b/benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp8_b300.sh similarity index 98% rename from benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_b300.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp8_b300.sh index e37d30c930..e146fe4fb3 100755 --- a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_b300.sh +++ b/benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp8_b300.sh @@ -9,7 +9,7 @@ # mandatory (MSA sparse/index cache); the benchmark is text-only, so # --language-model-only frees the vision encoder's VRAM. -source "$(dirname "$0")/../../benchmark_lib.sh" +source "$(dirname "$0")/../../../benchmark_lib.sh" check_env_vars \ MODEL \ diff --git a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_b300_mtp.sh b/benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp8_b300_mtp.sh similarity index 98% rename from benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_b300_mtp.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp8_b300_mtp.sh index e5df40e5fc..7439387f9f 100644 --- a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_b300_mtp.sh +++ b/benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp8_b300_mtp.sh @@ -11,7 +11,7 @@ # The target uses the FlashInfer TRT-LLM attention path. The EAGLE3-GQA drafter # is pinned separately to FLASH_ATTN. -source "$(dirname "$0")/../../benchmark_lib.sh" +source "$(dirname "$0")/../../../benchmark_lib.sh" check_env_vars \ MODEL \ diff --git a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_h100.sh b/benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp8_h100.sh similarity index 98% rename from benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_h100.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp8_h100.sh index d8a795987a..9d9c5802b1 100755 --- a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_h100.sh +++ b/benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp8_h100.sh @@ -10,7 +10,7 @@ # the MXFP8 MoE runs through vLLM's Hopper-compatible backends (Marlin / # DeepGEMM) selected by the mxfp8 oracle in the minimax-m3 image. -source "$(dirname "$0")/../../benchmark_lib.sh" +source "$(dirname "$0")/../../../benchmark_lib.sh" check_env_vars \ MODEL \ diff --git a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_h100_mtp.sh b/benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp8_h100_mtp.sh similarity index 99% rename from benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_h100_mtp.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp8_h100_mtp.sh index 8d9f6333f7..caa57e432b 100644 --- a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_h100_mtp.sh +++ b/benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp8_h100_mtp.sh @@ -16,7 +16,7 @@ # the B300 MTP canary). FLASH_ATTN takes any multiple-of-16 block size, so # the mandatory 128 is fine for the draft. -source "$(dirname "$0")/../../benchmark_lib.sh" +source "$(dirname "$0")/../../../benchmark_lib.sh" check_env_vars \ MODEL \ diff --git a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_h200.sh b/benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp8_h200.sh similarity index 98% rename from benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_h200.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp8_h200.sh index 057c0c230b..bf6ddb2c21 100755 --- a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_h200.sh +++ b/benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp8_h200.sh @@ -10,7 +10,7 @@ # the MXFP8 MoE runs through vLLM's Hopper-compatible backends (Marlin / # DeepGEMM) selected by the mxfp8 oracle in the minimax-m3 image. -source "$(dirname "$0")/../../benchmark_lib.sh" +source "$(dirname "$0")/../../../benchmark_lib.sh" check_env_vars \ MODEL \ diff --git a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_h200_mtp.sh b/benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp8_h200_mtp.sh similarity index 98% rename from benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_h200_mtp.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp8_h200_mtp.sh index 2bc98d4b8f..d94d5cc014 100644 --- a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_h200_mtp.sh +++ b/benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp8_h200_mtp.sh @@ -15,7 +15,7 @@ # the B300 MTP canary). FLASH_ATTN takes any multiple-of-16 block size, so # the mandatory 128 is fine for the draft. -source "$(dirname "$0")/../../benchmark_lib.sh" +source "$(dirname "$0")/../../../benchmark_lib.sh" check_env_vars \ MODEL \ diff --git a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_mi300x.sh b/benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp8_mi300x.sh similarity index 98% rename from benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_mi300x.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp8_mi300x.sh index 4a6ce42e71..917cf78352 100755 --- a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_mi300x.sh +++ b/benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp8_mi300x.sh @@ -9,7 +9,7 @@ # scheduling levers: the larger prefill batch amortizes the per-step # BF16-emulated MoE weight read, lifting high-concurrency throughput. -source "$(dirname "$0")/../../benchmark_lib.sh" +source "$(dirname "$0")/../../../benchmark_lib.sh" check_env_vars \ MODEL \ diff --git a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_mi300x_mtp.sh b/benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp8_mi300x_mtp.sh similarity index 98% rename from benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_mi300x_mtp.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp8_mi300x_mtp.sh index 64f4b1fa13..6de6422ed5 100644 --- a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_mi300x_mtp.sh +++ b/benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp8_mi300x_mtp.sh @@ -16,7 +16,7 @@ # Here the whole server runs on TRITON_ATTN (set globally below), which serves # the MHA draft fine. -source "$(dirname "$0")/../../benchmark_lib.sh" +source "$(dirname "$0")/../../../benchmark_lib.sh" check_env_vars \ MODEL \ diff --git a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_mi325x.sh b/benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp8_mi325x.sh similarity index 97% rename from benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_mi325x.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp8_mi325x.sh index 6290722feb..48e83cfa73 100755 --- a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_mi325x.sh +++ b/benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp8_mi325x.sh @@ -4,7 +4,7 @@ # https://recipes.vllm.ai/MiniMaxAI/MiniMax-M3?hardware=mi325x&variant=mxfp8 # MXFP8 runs from TP=4 on gfx942; block size 128 is mandatory for MSA. -source "$(dirname "$0")/../../benchmark_lib.sh" +source "$(dirname "$0")/../../../benchmark_lib.sh" check_env_vars \ MODEL \ diff --git a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_mi325x_mtp.sh b/benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp8_mi325x_mtp.sh similarity index 99% rename from benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_mi325x_mtp.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp8_mi325x_mtp.sh index 4ba15e7610..24d4eae974 100644 --- a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_mi325x_mtp.sh +++ b/benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp8_mi325x_mtp.sh @@ -28,7 +28,7 @@ # same patch is validated green on MI355X. It is idempotent and fails the job # loudly if the installed amd/model.py has drifted from the expected base. -source "$(dirname "$0")/../../benchmark_lib.sh" +source "$(dirname "$0")/../../../benchmark_lib.sh" check_env_vars \ MODEL \ diff --git a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_mi355x.sh b/benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp8_mi355x.sh similarity index 99% rename from benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_mi355x.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp8_mi355x.sh index 61004c0b52..7df628b70b 100755 --- a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_mi355x.sh +++ b/benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp8_mi355x.sh @@ -15,7 +15,7 @@ # indexer-skip override: it reduces model-architecture FLOPs, which is # disallowed by docs/PR_REVIEW_CHECKLIST.md. -source "$(dirname "$0")/../../benchmark_lib.sh" +source "$(dirname "$0")/../../../benchmark_lib.sh" check_env_vars \ MODEL \ diff --git a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_mi355x_atom.sh b/benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp8_mi355x_atom.sh similarity index 97% rename from benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_mi355x_atom.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp8_mi355x_atom.sh index c5ce072fce..6bbe70d5d0 100644 --- a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_mi355x_atom.sh +++ b/benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp8_mi355x_atom.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash -source "$(dirname "$0")/../../benchmark_lib.sh" +source "$(dirname "$0")/../../../benchmark_lib.sh" check_env_vars \ MODEL \ diff --git a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_mi355x_atom_mtp.sh b/benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp8_mi355x_atom_mtp.sh similarity index 98% rename from benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_mi355x_atom_mtp.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp8_mi355x_atom_mtp.sh index 66320d03d5..a6bb3321d2 100644 --- a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_mi355x_atom_mtp.sh +++ b/benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp8_mi355x_atom_mtp.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash -source "$(dirname "$0")/../../benchmark_lib.sh" +source "$(dirname "$0")/../../../benchmark_lib.sh" check_env_vars \ MODEL \ diff --git a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_mi355x_mtp.sh b/benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp8_mi355x_mtp.sh similarity index 99% rename from benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_mi355x_mtp.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp8_mi355x_mtp.sh index 7d51f46118..6871e50fc0 100644 --- a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_mi355x_mtp.sh +++ b/benchmarks/single_node/fixed_seq_len/deprecated/minimaxm3_fp8_mi355x_mtp.sh @@ -19,7 +19,7 @@ # hardware ahead of an image rebuild. The patch is idempotent and fails the # job loudly if the installed amd/model.py has drifted from the expected base. -source "$(dirname "$0")/../../benchmark_lib.sh" +source "$(dirname "$0")/../../../benchmark_lib.sh" check_env_vars \ MODEL \ diff --git a/benchmarks/single_node/fixed_seq_len/qwen3.5_bf16_b200.sh b/benchmarks/single_node/fixed_seq_len/deprecated/qwen3.5_bf16_b200.sh similarity index 98% rename from benchmarks/single_node/fixed_seq_len/qwen3.5_bf16_b200.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/qwen3.5_bf16_b200.sh index 3f7c6a314a..422c2cdf38 100755 --- a/benchmarks/single_node/fixed_seq_len/qwen3.5_bf16_b200.sh +++ b/benchmarks/single_node/fixed_seq_len/deprecated/qwen3.5_bf16_b200.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash -source "$(dirname "$0")/../../benchmark_lib.sh" +source "$(dirname "$0")/../../../benchmark_lib.sh" check_env_vars \ MODEL \ diff --git a/benchmarks/single_node/fixed_seq_len/qwen3.5_bf16_b200_mtp.sh b/benchmarks/single_node/fixed_seq_len/deprecated/qwen3.5_bf16_b200_mtp.sh similarity index 98% rename from benchmarks/single_node/fixed_seq_len/qwen3.5_bf16_b200_mtp.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/qwen3.5_bf16_b200_mtp.sh index be314c872d..4c48d28fc5 100755 --- a/benchmarks/single_node/fixed_seq_len/qwen3.5_bf16_b200_mtp.sh +++ b/benchmarks/single_node/fixed_seq_len/deprecated/qwen3.5_bf16_b200_mtp.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash -source "$(dirname "$0")/../../benchmark_lib.sh" +source "$(dirname "$0")/../../../benchmark_lib.sh" check_env_vars \ MODEL \ diff --git a/benchmarks/single_node/fixed_seq_len/qwen3.5_bf16_b300.sh b/benchmarks/single_node/fixed_seq_len/deprecated/qwen3.5_bf16_b300.sh similarity index 98% rename from benchmarks/single_node/fixed_seq_len/qwen3.5_bf16_b300.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/qwen3.5_bf16_b300.sh index 32e4197cb9..33a1532554 100755 --- a/benchmarks/single_node/fixed_seq_len/qwen3.5_bf16_b300.sh +++ b/benchmarks/single_node/fixed_seq_len/deprecated/qwen3.5_bf16_b300.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash -source "$(dirname "$0")/../../benchmark_lib.sh" +source "$(dirname "$0")/../../../benchmark_lib.sh" check_env_vars \ MODEL \ diff --git a/benchmarks/single_node/fixed_seq_len/qwen3.5_bf16_b300_mtp.sh b/benchmarks/single_node/fixed_seq_len/deprecated/qwen3.5_bf16_b300_mtp.sh similarity index 98% rename from benchmarks/single_node/fixed_seq_len/qwen3.5_bf16_b300_mtp.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/qwen3.5_bf16_b300_mtp.sh index 4e12d1284a..c4a5185b2c 100755 --- a/benchmarks/single_node/fixed_seq_len/qwen3.5_bf16_b300_mtp.sh +++ b/benchmarks/single_node/fixed_seq_len/deprecated/qwen3.5_bf16_b300_mtp.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash -source "$(dirname "$0")/../../benchmark_lib.sh" +source "$(dirname "$0")/../../../benchmark_lib.sh" check_env_vars \ MODEL \ diff --git a/benchmarks/single_node/fixed_seq_len/qwen3.5_bf16_mi300x.sh b/benchmarks/single_node/fixed_seq_len/deprecated/qwen3.5_bf16_mi300x.sh similarity index 97% rename from benchmarks/single_node/fixed_seq_len/qwen3.5_bf16_mi300x.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/qwen3.5_bf16_mi300x.sh index 32fe60a735..b803c55514 100755 --- a/benchmarks/single_node/fixed_seq_len/qwen3.5_bf16_mi300x.sh +++ b/benchmarks/single_node/fixed_seq_len/deprecated/qwen3.5_bf16_mi300x.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash -source "$(dirname "$0")/../../benchmark_lib.sh" +source "$(dirname "$0")/../../../benchmark_lib.sh" check_env_vars \ MODEL \ diff --git a/benchmarks/single_node/fixed_seq_len/qwen3.5_bf16_mi325x.sh b/benchmarks/single_node/fixed_seq_len/deprecated/qwen3.5_bf16_mi325x.sh similarity index 97% rename from benchmarks/single_node/fixed_seq_len/qwen3.5_bf16_mi325x.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/qwen3.5_bf16_mi325x.sh index 32fe60a735..b803c55514 100644 --- a/benchmarks/single_node/fixed_seq_len/qwen3.5_bf16_mi325x.sh +++ b/benchmarks/single_node/fixed_seq_len/deprecated/qwen3.5_bf16_mi325x.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash -source "$(dirname "$0")/../../benchmark_lib.sh" +source "$(dirname "$0")/../../../benchmark_lib.sh" check_env_vars \ MODEL \ diff --git a/benchmarks/single_node/fixed_seq_len/qwen3.5_bf16_mi325x_mtp.sh b/benchmarks/single_node/fixed_seq_len/deprecated/qwen3.5_bf16_mi325x_mtp.sh similarity index 97% rename from benchmarks/single_node/fixed_seq_len/qwen3.5_bf16_mi325x_mtp.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/qwen3.5_bf16_mi325x_mtp.sh index e9df93c7da..88624f7407 100755 --- a/benchmarks/single_node/fixed_seq_len/qwen3.5_bf16_mi325x_mtp.sh +++ b/benchmarks/single_node/fixed_seq_len/deprecated/qwen3.5_bf16_mi325x_mtp.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash -source "$(dirname "$0")/../../benchmark_lib.sh" +source "$(dirname "$0")/../../../benchmark_lib.sh" check_env_vars \ MODEL \ diff --git a/benchmarks/single_node/fixed_seq_len/qwen3.5_bf16_mi355x.sh b/benchmarks/single_node/fixed_seq_len/deprecated/qwen3.5_bf16_mi355x.sh similarity index 97% rename from benchmarks/single_node/fixed_seq_len/qwen3.5_bf16_mi355x.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/qwen3.5_bf16_mi355x.sh index 1661df4658..590205c62d 100755 --- a/benchmarks/single_node/fixed_seq_len/qwen3.5_bf16_mi355x.sh +++ b/benchmarks/single_node/fixed_seq_len/deprecated/qwen3.5_bf16_mi355x.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash -source "$(dirname "$0")/../../benchmark_lib.sh" +source "$(dirname "$0")/../../../benchmark_lib.sh" check_env_vars \ MODEL \ diff --git a/benchmarks/single_node/fixed_seq_len/qwen3.5_bf16_mi355x_mtp.sh b/benchmarks/single_node/fixed_seq_len/deprecated/qwen3.5_bf16_mi355x_mtp.sh similarity index 97% rename from benchmarks/single_node/fixed_seq_len/qwen3.5_bf16_mi355x_mtp.sh rename to benchmarks/single_node/fixed_seq_len/deprecated/qwen3.5_bf16_mi355x_mtp.sh index 38230cc884..72c8abec45 100755 --- a/benchmarks/single_node/fixed_seq_len/qwen3.5_bf16_mi355x_mtp.sh +++ b/benchmarks/single_node/fixed_seq_len/deprecated/qwen3.5_bf16_mi355x_mtp.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash -source "$(dirname "$0")/../../benchmark_lib.sh" +source "$(dirname "$0")/../../../benchmark_lib.sh" check_env_vars \ MODEL \ diff --git a/benchmarks/single_node/fixed_seq_len/kimik2.5_fp4_b200.sh b/benchmarks/single_node/fixed_seq_len/kimik2.5_fp4_b200.sh index 59b55c90c6..6c413137a0 100644 --- a/benchmarks/single_node/fixed_seq_len/kimik2.5_fp4_b200.sh +++ b/benchmarks/single_node/fixed_seq_len/kimik2.5_fp4_b200.sh @@ -36,7 +36,7 @@ start_gpu_monitor # (~32% of total), which collides with --gpu-memory-utilization=0.90 and # leaves negative space for the KV cache. Disable the profiler — our 0.90 # already leaves ~18 GB/GPU as safety net (same pattern as -# benchmarks/single_node/agentic/kimik2.5_fp4_b200.sh). +# benchmarks/single_node/agentic/deprecated/kimik2.5_fp4_b200.sh). export VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 set -x diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index 6e8e6253ab..358da1b337 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -170,66 +170,6 @@ dsr1-fp8-mi355x-sglang-mtp: search-space: - { tp: 8, ep: 1, conc-start: 4, conc-end: 64, spec-decoding: mtp } -qwen3.5-bf16-mi355x-sglang: - image: lmsysorg/sglang-rocm:v0.5.12-rocm720-mi35x-20260517 - model: Qwen/Qwen3.5-397B-A17B - model-prefix: qwen3.5 - runner: mi355x - precision: bf16 - framework: sglang - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - { tp: 8, ep: 1, conc-start: 4, conc-end: 256 } - -qwen3.5-bf16-mi355x-sglang-mtp: - image: lmsysorg/sglang-rocm:v0.5.12-rocm720-mi35x-20260517 - model: Qwen/Qwen3.5-397B-A17B - model-prefix: qwen3.5 - runner: mi355x - precision: bf16 - framework: sglang - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - { tp: 8, ep: 1, conc-start: 4, conc-end: 256, spec-decoding: mtp } - -qwen3.5-bf16-mi300x-sglang: - image: lmsysorg/sglang:v0.5.12-rocm720-mi30x - model: Qwen/Qwen3.5-397B-A17B - model-prefix: qwen3.5 - runner: mi300x - precision: bf16 - framework: sglang - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - { tp: 8, conc-start: 4, conc-end: 64 } - -qwen3.5-bf16-mi325x-sglang: - image: lmsysorg/sglang:v0.5.12-rocm720-mi30x - model: Qwen/Qwen3.5-397B-A17B - model-prefix: qwen3.5 - runner: mi325x - precision: bf16 - framework: sglang - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - { tp: 8, conc-start: 4, conc-end: 64 } - qwen3.5-fp8-mi325x-sglang: image: lmsysorg/sglang:v0.5.12-rocm720-mi30x model: Qwen/Qwen3.5-397B-A17B-FP8 @@ -524,30 +464,6 @@ kimik2.5-fp4-mi355x-vllm: - { tp: 8, conc-start: 4, conc-end: 128 } - { tp: 4, conc-start: 4, conc-end: 128 } -kimik2.5-fp4-mi355x-vllm-agentic: - image: vllm/vllm-openai-rocm:v0.22.0 - model: amd/Kimi-K2.5-MXFP4 - model-prefix: kimik2.5 - runner: cluster:mi355x-amds - precision: fp4 - framework: vllm - multinode: false - scenarios: - agentic-coding: - - dram-utilization: 0.80 - search-space: - - { tp: 8, kv-offloading: none, conc-list: [1, 2, 4, 8, 16, 24, 32, 40, 48] } - # DRAM offload only above the KV cliff. Lower concurrencies fit - # entirely on-GPU, so paying the offload-path overhead there would - # just slow them down without measuring anything new. - - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: vllm-native }, conc-list: [32, 40, 48, 56] } - # TP=4 probe: half-node layout doubles per-GPU weight footprint - # (~62 GB on MI355X's 288 GB HBM, plenty of headroom). Restrict to - # cliff-region concurrencies on both offload modes so we can directly - # compare TP=4 vs TP=8 at the same conc points. - - { tp: 4, kv-offloading: none, conc-list: [16, 24, 32, 40] } - - { tp: 4, kv-offloading: dram, kv-offload-backend: { name: vllm-native }, conc-list: [16, 24, 32, 40] } - kimik2.5-fp4-mi355x-atom: image: rocm/atom:rocm7.2.4_ubuntu24.04_py3.12_pytorch_release_2.10.0_atom0.1.4_202607091539 model: amd/Kimi-K2.5-MXFP4 @@ -1450,21 +1366,6 @@ dsv4-fp4-mi355x-atom-mtp: # NEGATIVE (-10.98GB) even at gpu-mem-util 0.90 (run 30277221626). #2327 tp4 # works only because it's non-MTP (no draft KV layer). MTP stays tp8-only. -qwen3.5-bf16-mi325x-sglang-mtp: - image: lmsysorg/sglang:v0.5.12-rocm720-mi30x - model: Qwen/Qwen3.5-397B-A17B - model-prefix: qwen3.5 - runner: mi325x - precision: bf16 - framework: sglang - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - { tp: 8, ep: 1, conc-start: 4, conc-end: 64, spec-decoding: mtp } - dsr1-fp8-mi325x-sglang-mtp: image: lmsysorg/sglang:v0.5.12-rocm700-mi30x model: deepseek-ai/DeepSeek-R1-0528 @@ -1735,438 +1636,6 @@ dsv4-fp4-mi355x-atom-disagg: additional-settings: - "DECODE_NODES=1" # 1P1D TP8 -minimaxm3-fp8-mi355x-vllm: - image: vllm/vllm-openai-rocm:nightly-9e57de7197f234f9d9187715d96e07e007048c0f - model: MiniMaxAI/MiniMax-M3-MXFP8 - model-prefix: minimaxm3 - runner: mi355x - precision: fp8 - framework: vllm - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - { tp: 4, conc-start: 1, conc-end: 512 } - -# EAGLE3 speculative-decoding (spec-decoding: mtp) variant of -# minimaxm3-fp8-mi355x-vllm, pairing MiniMaxAI/MiniMax-M3-MXFP8 with the -# Inferact/MiniMax-M3-EAGLE3 draft head (3 speculative tokens). No -# attention_backend override is needed — the server runs on TRITON_ATTN, so -# the FlashInfer page-128/MHA limitation that forced FLASH_ATTN on Blackwell -# does not apply here. Search space mirrors the non-MTP entry trimmed at the -# extreme-concurrency end, identical to the minimaxm3-fp8-b300-vllm-mtp / -# b200-vllm-mtp precedent: spec decode pays off at low/mid concurrency while -# acceptance dilutes in big batches, and the draft weights + draft KV shave -# headroom — tp2-ep2 is dropped since its KV headroom was already thin. -minimaxm3-fp8-mi355x-vllm-mtp: - image: vllm/vllm-openai-rocm:nightly-09663abde0f50944a8d5ea30120666024b503faa - model: MiniMaxAI/MiniMax-M3-MXFP8 - model-prefix: minimaxm3 - runner: mi355x - precision: fp8 - framework: vllm - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - { tp: 4, conc-start: 1, conc-end: 512, spec-decoding: mtp } - -# MiniMax-M3 MXFP4 MI355X vLLM disaggregated (prefill/decode) config. -minimaxm3-fp4-mi355x-vllm-disagg: - image: vllm/vllm-openai-rocm:nightly-2dfaae752b4db0d43cfc0715c780e33be030d0f1 - model: amd/MiniMax-M3-MXFP4 - model-prefix: minimaxm3 - runner: mi355x-disagg - precision: fp4 - framework: vllm-disagg - router: { name: vllm-router, version: "0.1.14" } - kv-p2p-transfer: moriio - multinode: true - disagg: true - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - # 1P TP4 + 1D TP4 (2 nodes total), conc sweep 1..256 (single job, looped) - - spec-decoding: "none" - conc-list: [ 1, 2, 4, 8, 16, 32, 64, 128, 256 ] - prefill: - num-worker: 1 - tp: 4 - ep: 1 - dp-attn: false - additional-settings: - - "PREFILL_NODES=1" - decode: - num-worker: 1 - tp: 4 - ep: 1 - dp-attn: false - additional-settings: - - "DECODE_NODES=1" -# MiniMax-M3 MXFP4 MI355X vLLM recipe. The pinned nightly includes upstream -# MiniMax-M3 Quark MXFP4 support (vllm-project/vllm#45794). Use the text-only -# language-model path and mirror the MXFP8 MI355X search space for a direct -# precision comparison. -minimaxm3-fp4-mi355x-vllm: - image: vllm/vllm-openai-rocm:nightly-69715823df89b11ee684b84066390cbb9092d5c1 - model: amd/MiniMax-M3-MXFP4 - model-prefix: minimaxm3 - runner: mi355x - precision: fp4 - framework: vllm - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - { tp: 2, conc-start: 1, conc-end: 512 } - - { tp: 4, conc-start: 1, conc-end: 64 } - -# EAGLE3 speculative-decoding variant of minimaxm3-fp4-mi355x-vllm. Pair the -# amd/MiniMax-M3-MXFP4 target with Inferact/MiniMax-M3-EAGLE3 and three draft -# tokens. Search space mirrors the MI355X MXFP8 MTP entry, trimming the base -# FP4 sweep at extreme concurrency where speculative decoding loses value. -minimaxm3-fp4-mi355x-vllm-mtp: - image: vllm/vllm-openai-rocm:nightly-3f5a1e1733200760169ff31ebe60a271072b199e - model: amd/MiniMax-M3-MXFP4 - model-prefix: minimaxm3 - runner: mi355x - precision: fp4 - framework: vllm - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - { tp: 8, conc-start: 1, conc-end: 64, spec-decoding: mtp } - - { tp: 8, ep: 8, conc-start: 1, conc-end: 256, spec-decoding: mtp } - - { tp: 4, conc-start: 1, conc-end: 64, spec-decoding: mtp } - - { tp: 8, ep: 8, dp-attn: true, conc-start: 128, conc-end: 256, spec-decoding: mtp } - -# MiniMax-M3 MXFP4 MI355X atom recipe: -# https://github.com/ROCm/ATOM/blob/5d42d49f9e4292e5b61475917e92e7ec1b1dacb7/recipes/MiniMax-M3.md -# block size 128 is mandatory for MSA. TP4 on a single gfx950 node, per the recipe. -minimaxm3-fp4-mi355x-atom: - image: rocm/atom-dev:nightly_202607011530 - model: amd/MiniMax-M3-MXFP4 - model-prefix: minimaxm3 - runner: mi355x - precision: fp4 - framework: atom - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - { tp: 4, conc-start: 1, conc-end: 256 } - -minimaxm3-fp4-mi355x-atom-mtp: - image: rocm/atom-dev:nightly_202607011530 - model: amd/MiniMax-M3-MXFP4 - model-prefix: minimaxm3 - runner: mi355x - precision: fp4 - framework: atom - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - { tp: 4, conc-start: 1, conc-end: 256, spec-decoding: mtp } - -minimaxm3-fp8-mi355x-atom: - image: rocm/atom-dev:nightly_202607011530 - model: MiniMaxAI/MiniMax-M3-MXFP8 - model-prefix: minimaxm3 - runner: mi355x - precision: fp8 - framework: atom - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - { tp: 4, conc-start: 1, conc-end: 256 } - -minimaxm3-fp8-mi355x-atom-mtp: - image: rocm/atom-dev:nightly_202607011530 - model: MiniMaxAI/MiniMax-M3-MXFP8 - model-prefix: minimaxm3 - runner: mi355x - precision: fp8 - framework: atom - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - { tp: 4, conc-start: 1, conc-end: 256, spec-decoding: mtp } - -minimaxm3-fp8-mi355x-atom-disagg: - image: rocm/atom-dev:nightly_202607011530 - model: MiniMaxAI/MiniMax-M3-MXFP8 - model-prefix: minimaxm3 - runner: mi355x-disagg - precision: fp8 - framework: atom-disagg - router: { name: atomesh, version: "04b120d3040e0dd1a6915e88a3a3c1f588e6684a" } - kv-p2p-transfer: mooncake - multinode: true - disagg: true - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - # 1P1D TP4 - - conc-list: [ 1, 2, 4, 8, 16, 32, 64, 128, 256 ] - prefill: - num-worker: 1 - tp: 4 - ep: 1 - dp-attn: false - additional-settings: - - "PREFILL_NODES=1" - decode: - num-worker: 1 - tp: 4 - ep: 1 - dp-attn: false - additional-settings: - - "DECODE_NODES=1" - - "DECODE_MTP_SIZE=0" - # 2P1D, DPA TP4 - - conc-list: [ 256, 512, 768, 1024 ] - prefill: - num-worker: 2 - tp: 4 - ep: 1 - dp-attn: true - additional-settings: - - "PREFILL_NODES=2" - decode: - num-worker: 1 - tp: 4 - ep: 1 - dp-attn: true - additional-settings: - - "DECODE_NODES=1" - - "DECODE_MTP_SIZE=0" - # 1P1D TP4 -minimaxm3-fp4-mi355x-atom-disagg: - image: rocm/atom-dev:nightly_202607011530 - model: amd/MiniMax-M3-MXFP4 - model-prefix: minimaxm3 - runner: mi355x-disagg - precision: fp4 - framework: atom-disagg - router: { name: atomesh, version: "04b120d3040e0dd1a6915e88a3a3c1f588e6684a" } - kv-p2p-transfer: mooncake - multinode: true - disagg: true - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - # 1P1D TP4 - - conc-list: [ 1, 2, 4, 8, 16, 32, 64, 128, 256 ] - prefill: - num-worker: 1 - tp: 4 - ep: 1 - dp-attn: false - additional-settings: - - "PREFILL_NODES=1" - decode: - num-worker: 1 - tp: 4 - ep: 1 - dp-attn: false - additional-settings: - - "DECODE_NODES=1" - - "DECODE_MTP_SIZE=0" - # 2P1D, DPA TP4 - - conc-list: [ 256, 512, 768, 1024 ] - prefill: - num-worker: 2 - tp: 4 - ep: 1 - dp-attn: true - additional-settings: - - "PREFILL_NODES=2" - decode: - num-worker: 1 - tp: 4 - ep: 1 - dp-attn: true - additional-settings: - - "DECODE_NODES=1" - - "DECODE_MTP_SIZE=0" - # 1P1D TP4 -minimaxm3-fp8-mi300x-vllm: - image: vllm/vllm-openai-rocm:nightly-4559c43a9526597c00cbcc4f59979496500268d1 - model: MiniMaxAI/MiniMax-M3-MXFP8 - model-prefix: minimaxm3 - runner: mi300x - precision: fp8 - framework: vllm - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - { tp: 8, conc-start: 1, conc-end: 256 } - -# EAGLE3 speculative-decoding (spec-decoding: mtp) variant of -# minimaxm3-fp8-mi300x-vllm, pairing MiniMaxAI/MiniMax-M3-MXFP8 with the -# Inferact/MiniMax-M3-EAGLE3 draft head (3 speculative tokens). Same TP8-only -# search space as the non-MTP MI300X entry (gfx942 192 GB is memory-tight, like -# H100), with the TP8 latency rows started at conc 1 to capture single-request -# latency — matching the H100/MI355X MTP recipes. The pinned ROCm nightly -# includes upstream SupportsEagle3 support for the AMD MiniMax-M3 model. -minimaxm3-fp8-mi300x-vllm-mtp: - image: vllm/vllm-openai-rocm:nightly-b53b1c7ffe7aebdafd0876350f30e51d1226c92a - model: MiniMaxAI/MiniMax-M3-MXFP8 - model-prefix: minimaxm3 - runner: mi300x - precision: fp8 - framework: vllm - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - { tp: 8, conc-start: 1, conc-end: 64, spec-decoding: mtp } - - { tp: 8, ep: 8, conc-start: 128, conc-end: 256, spec-decoding: mtp } - -# MiniMax-M3 MXFP8 MI325X day-zero recipe. Reuse the dedicated ROCm image -# and serving flags validated on MI355X, with the H200 search space: TP4 and -# TP8 latency, TP4/TP8 expert parallelism, and TP8 data-parallel attention. -minimaxm3-fp8-mi325x-vllm: - image: vllm/vllm-openai-rocm:minimax-m3 - model: MiniMaxAI/MiniMax-M3-MXFP8 - model-prefix: minimaxm3 - runner: mi325x - precision: fp8 - framework: vllm - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - { tp: 4, conc-start: 1, conc-end: 32 } - - { tp: 8, conc-start: 1, conc-end: 128 } - - { tp: 8, ep: 8, conc-start: 256, conc-end: 256 } - - { tp: 8, ep: 8, dp-attn: true, conc-start: 256, conc-end: 512 } - -# EAGLE3 speculative-decoding (spec-decoding: mtp) variant of -# minimaxm3-fp8-mi325x-vllm, pairing MiniMaxAI/MiniMax-M3-MXFP8 with the -# Inferact/MiniMax-M3-EAGLE3 draft head (3 speculative tokens). Same H200-style -# search space as the non-MTP MI325X entry, trimmed at the extreme-concurrency -# end with TP-only latency rows started at conc 1 (matching the H200/MI355X MTP -# recipes). Runs with CUDA graphs (no --enforce-eager, VLLM_USE_BREAKABLE_CUDAGRAPH=0, -# BF16 KV on gfx942). The shipped ROCm image lacks SupportsEagle3 on the AMD -# MiniMax-M3 model, so the recipe applies that fix in-place at runtime -# (functionstackx/vllm#1, upstream vllm-project/vllm#45546; validated green on -# MI355X/MI300X) before serving. -minimaxm3-fp8-mi325x-vllm-mtp: - image: vllm/vllm-openai-rocm:minimax-m3 - model: MiniMaxAI/MiniMax-M3-MXFP8 - model-prefix: minimaxm3 - runner: mi325x - precision: fp8 - framework: vllm - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - { tp: 4, conc-start: 1, conc-end: 32, spec-decoding: mtp } - - { tp: 8, conc-start: 1, conc-end: 128, spec-decoding: mtp } - - { tp: 8, ep: 8, conc-start: 256, conc-end: 256, spec-decoding: mtp } - - { tp: 8, ep: 8, dp-attn: true, conc-start: 256, conc-end: 256, spec-decoding: mtp } - -# MiniMax-M3 MXFP8 MI355X vLLM disaggregated (prefill/decode) sweep on the -# day-zero ROCm image, over the MoRI-IO KV-transfer pipeline (MoRI-patch-removal -# infra #1585). All workers are TP4, no EP: the single-node M3 MXFP8 recipe -# (minimaxm3-fp8-mi355x-vllm, PR #2003) found plain TP4 beats both TP8 and -# TP4/EP4 on tok/s/GPU for this model on gfx950, so prefill and decode both use -# TP4 and we tune the prefill:decode worker ratio (xP:yD) instead of TP. The -# mi355x-disagg pool has 3 nodes and the launcher places one worker per node -# (NUM_NODES = xP + yD), so every layout keeps xP + yD <= 3: -# - 1P-TP4 / 1D-TP4 (2 nodes): balanced, full concurrency curve. -# - 1P-TP4 / 2D-TP4 (3 nodes): decode-heavy, for the decode-bound 1k1k tail. -# - 2P-TP4 / 1D-TP4 (3 nodes): prefill-heavy, for the prefill-bound 8k1k tail. -# Per-worker serve flags live in -# benchmarks/multi_node/amd_utils/models_vllm.yaml (MiniMax-M3-MXFP8). -minimaxm3-fp8-mi355x-vllm-disagg: - image: vllm/vllm-openai-rocm:nightly-2afa3f7e950264bb179d030c23a1ed1f46558fd9 - model: MiniMaxAI/MiniMax-M3-MXFP8 - model-prefix: minimaxm3 - runner: mi355x-disagg - precision: fp8 - framework: vllm-disagg - router: { name: vllm-router, version: "0.1.14" } - kv-p2p-transfer: moriio - multinode: true - disagg: true - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - # Balanced 1P TP4 + 1D TP4 (2 nodes) across the full curve. - - spec-decoding: "none" - conc-list: [ 1, 2, 4, 8, 16, 32, 64, 128, 256, 512 ] - prefill: - num-worker: 1 - tp: 4 - ep: 1 - dp-attn: false - additional-settings: - - "PREFILL_NODES=1" - decode: - num-worker: 1 - tp: 4 - ep: 1 - dp-attn: false - additional-settings: - - "DECODE_NODES=1" - # Prefill-heavy 2P TP4 + 1D TP4 (3 nodes): two half-node TP4 prefill workers - # keep the single TP4 decode engine fed for the prefill-bound 8k1k tail. - - spec-decoding: "none" - conc-list: [ 128, 256, 512 ] - prefill: - num-worker: 2 - tp: 4 - ep: 1 - dp-attn: false - additional-settings: - - "PREFILL_NODES=2" - decode: - num-worker: 1 - tp: 4 - ep: 1 - dp-attn: false - additional-settings: - - "DECODE_NODES=1" minimaxm3-fp8-mi300x-vllm-agentic: image: vllm/vllm-openai-rocm:nightly-04c2a8deac44fdb1ca3e2b5ec3e6bf16f3f6a914 model: MiniMaxAI/MiniMax-M3-MXFP8 diff --git a/configs/deprecated/amd-kimik2.5-agentic-master.yaml b/configs/deprecated/amd-kimik2.5-agentic-master.yaml new file mode 100644 index 0000000000..641bb41f1b --- /dev/null +++ b/configs/deprecated/amd-kimik2.5-agentic-master.yaml @@ -0,0 +1,30 @@ +# Deprecated Kimi-K2.5/2.6/2.7-Code agentic-coding entries archived from +# amd-master.yaml. Deprecated after Monday, August 3, 2026 (see the +# Deprecation Notice in MODELS.md). Single-turn 8k1k stays active until +# August 6, 2026. +# Removed from the active master config so sweep generation no longer selects them. + + +kimik2.5-fp4-mi355x-vllm-agentic: + image: vllm/vllm-openai-rocm:v0.22.0 + model: amd/Kimi-K2.5-MXFP4 + model-prefix: kimik2.5 + runner: cluster:mi355x-amds + precision: fp4 + framework: vllm + multinode: false + scenarios: + agentic-coding: + - dram-utilization: 0.80 + search-space: + - { tp: 8, kv-offloading: none, conc-list: [1, 2, 4, 8, 16, 24, 32, 40, 48] } + # DRAM offload only above the KV cliff. Lower concurrencies fit + # entirely on-GPU, so paying the offload-path overhead there would + # just slow them down without measuring anything new. + - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: vllm-native }, conc-list: [32, 40, 48, 56] } + # TP=4 probe: half-node layout doubles per-GPU weight footprint + # (~62 GB on MI355X's 288 GB HBM, plenty of headroom). Restrict to + # cliff-region concurrencies on both offload modes so we can directly + # compare TP=4 vs TP=8 at the same conc points. + - { tp: 4, kv-offloading: none, conc-list: [16, 24, 32, 40] } + - { tp: 4, kv-offloading: dram, kv-offload-backend: { name: vllm-native }, conc-list: [16, 24, 32, 40] } diff --git a/configs/deprecated/amd-minimaxm3-8k1k-master.yaml b/configs/deprecated/amd-minimaxm3-8k1k-master.yaml new file mode 100644 index 0000000000..09b669ac6a --- /dev/null +++ b/configs/deprecated/amd-minimaxm3-8k1k-master.yaml @@ -0,0 +1,437 @@ +# Deprecated MiniMax-M3 Single-turn 8k1k entries archived from amd-master.yaml. +# The scenario was deprecated after Monday, August 3, 2026 (see the Deprecation +# Notice in MODELS.md). Agentic coding remains active for this model. +# Removed from the active master config so sweep generation no longer selects them. + +minimaxm3-fp8-mi355x-vllm: + image: vllm/vllm-openai-rocm:nightly-9e57de7197f234f9d9187715d96e07e007048c0f + model: MiniMaxAI/MiniMax-M3-MXFP8 + model-prefix: minimaxm3 + runner: mi355x + precision: fp8 + framework: vllm + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - { tp: 4, conc-start: 1, conc-end: 512 } + +# EAGLE3 speculative-decoding (spec-decoding: mtp) variant of +# minimaxm3-fp8-mi355x-vllm, pairing MiniMaxAI/MiniMax-M3-MXFP8 with the +# Inferact/MiniMax-M3-EAGLE3 draft head (3 speculative tokens). No +# attention_backend override is needed — the server runs on TRITON_ATTN, so +# the FlashInfer page-128/MHA limitation that forced FLASH_ATTN on Blackwell +# does not apply here. Search space mirrors the non-MTP entry trimmed at the +# extreme-concurrency end, identical to the minimaxm3-fp8-b300-vllm-mtp / +# b200-vllm-mtp precedent: spec decode pays off at low/mid concurrency while +# acceptance dilutes in big batches, and the draft weights + draft KV shave +# headroom — tp2-ep2 is dropped since its KV headroom was already thin. +minimaxm3-fp8-mi355x-vllm-mtp: + image: vllm/vllm-openai-rocm:nightly-09663abde0f50944a8d5ea30120666024b503faa + model: MiniMaxAI/MiniMax-M3-MXFP8 + model-prefix: minimaxm3 + runner: mi355x + precision: fp8 + framework: vllm + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - { tp: 4, conc-start: 1, conc-end: 512, spec-decoding: mtp } + +# MiniMax-M3 MXFP4 MI355X vLLM disaggregated (prefill/decode) config. +minimaxm3-fp4-mi355x-vllm-disagg: + image: vllm/vllm-openai-rocm:nightly-2dfaae752b4db0d43cfc0715c780e33be030d0f1 + model: amd/MiniMax-M3-MXFP4 + model-prefix: minimaxm3 + runner: mi355x-disagg + precision: fp4 + framework: vllm-disagg + router: { name: vllm-router, version: "0.1.14" } + kv-p2p-transfer: moriio + multinode: true + disagg: true + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + # 1P TP4 + 1D TP4 (2 nodes total), conc sweep 1..256 (single job, looped) + - spec-decoding: "none" + conc-list: [ 1, 2, 4, 8, 16, 32, 64, 128, 256 ] + prefill: + num-worker: 1 + tp: 4 + ep: 1 + dp-attn: false + additional-settings: + - "PREFILL_NODES=1" + decode: + num-worker: 1 + tp: 4 + ep: 1 + dp-attn: false + additional-settings: + - "DECODE_NODES=1" +# MiniMax-M3 MXFP4 MI355X vLLM recipe. The pinned nightly includes upstream +# MiniMax-M3 Quark MXFP4 support (vllm-project/vllm#45794). Use the text-only +# language-model path and mirror the MXFP8 MI355X search space for a direct +# precision comparison. +minimaxm3-fp4-mi355x-vllm: + image: vllm/vllm-openai-rocm:nightly-69715823df89b11ee684b84066390cbb9092d5c1 + model: amd/MiniMax-M3-MXFP4 + model-prefix: minimaxm3 + runner: mi355x + precision: fp4 + framework: vllm + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - { tp: 2, conc-start: 1, conc-end: 512 } + - { tp: 4, conc-start: 1, conc-end: 64 } + +# EAGLE3 speculative-decoding variant of minimaxm3-fp4-mi355x-vllm. Pair the +# amd/MiniMax-M3-MXFP4 target with Inferact/MiniMax-M3-EAGLE3 and three draft +# tokens. Search space mirrors the MI355X MXFP8 MTP entry, trimming the base +# FP4 sweep at extreme concurrency where speculative decoding loses value. +minimaxm3-fp4-mi355x-vllm-mtp: + image: vllm/vllm-openai-rocm:nightly-3f5a1e1733200760169ff31ebe60a271072b199e + model: amd/MiniMax-M3-MXFP4 + model-prefix: minimaxm3 + runner: mi355x + precision: fp4 + framework: vllm + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - { tp: 8, conc-start: 1, conc-end: 64, spec-decoding: mtp } + - { tp: 8, ep: 8, conc-start: 1, conc-end: 256, spec-decoding: mtp } + - { tp: 4, conc-start: 1, conc-end: 64, spec-decoding: mtp } + - { tp: 8, ep: 8, dp-attn: true, conc-start: 128, conc-end: 256, spec-decoding: mtp } + +# MiniMax-M3 MXFP4 MI355X atom recipe: +# https://github.com/ROCm/ATOM/blob/5d42d49f9e4292e5b61475917e92e7ec1b1dacb7/recipes/MiniMax-M3.md +# block size 128 is mandatory for MSA. TP4 on a single gfx950 node, per the recipe. +minimaxm3-fp4-mi355x-atom: + image: rocm/atom-dev:nightly_202607011530 + model: amd/MiniMax-M3-MXFP4 + model-prefix: minimaxm3 + runner: mi355x + precision: fp4 + framework: atom + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - { tp: 4, conc-start: 1, conc-end: 256 } + +minimaxm3-fp4-mi355x-atom-mtp: + image: rocm/atom-dev:nightly_202607011530 + model: amd/MiniMax-M3-MXFP4 + model-prefix: minimaxm3 + runner: mi355x + precision: fp4 + framework: atom + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - { tp: 4, conc-start: 1, conc-end: 256, spec-decoding: mtp } + +minimaxm3-fp8-mi355x-atom: + image: rocm/atom-dev:nightly_202607011530 + model: MiniMaxAI/MiniMax-M3-MXFP8 + model-prefix: minimaxm3 + runner: mi355x + precision: fp8 + framework: atom + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - { tp: 4, conc-start: 1, conc-end: 256 } + +minimaxm3-fp8-mi355x-atom-mtp: + image: rocm/atom-dev:nightly_202607011530 + model: MiniMaxAI/MiniMax-M3-MXFP8 + model-prefix: minimaxm3 + runner: mi355x + precision: fp8 + framework: atom + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - { tp: 4, conc-start: 1, conc-end: 256, spec-decoding: mtp } + +minimaxm3-fp8-mi355x-atom-disagg: + image: rocm/atom-dev:nightly_202607011530 + model: MiniMaxAI/MiniMax-M3-MXFP8 + model-prefix: minimaxm3 + runner: mi355x-disagg + precision: fp8 + framework: atom-disagg + router: { name: atomesh, version: "04b120d3040e0dd1a6915e88a3a3c1f588e6684a" } + kv-p2p-transfer: mooncake + multinode: true + disagg: true + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + # 1P1D TP4 + - conc-list: [ 1, 2, 4, 8, 16, 32, 64, 128, 256 ] + prefill: + num-worker: 1 + tp: 4 + ep: 1 + dp-attn: false + additional-settings: + - "PREFILL_NODES=1" + decode: + num-worker: 1 + tp: 4 + ep: 1 + dp-attn: false + additional-settings: + - "DECODE_NODES=1" + - "DECODE_MTP_SIZE=0" + # 2P1D, DPA TP4 + - conc-list: [ 256, 512, 768, 1024 ] + prefill: + num-worker: 2 + tp: 4 + ep: 1 + dp-attn: true + additional-settings: + - "PREFILL_NODES=2" + decode: + num-worker: 1 + tp: 4 + ep: 1 + dp-attn: true + additional-settings: + - "DECODE_NODES=1" + - "DECODE_MTP_SIZE=0" + # 1P1D TP4 +minimaxm3-fp4-mi355x-atom-disagg: + image: rocm/atom-dev:nightly_202607011530 + model: amd/MiniMax-M3-MXFP4 + model-prefix: minimaxm3 + runner: mi355x-disagg + precision: fp4 + framework: atom-disagg + router: { name: atomesh, version: "04b120d3040e0dd1a6915e88a3a3c1f588e6684a" } + kv-p2p-transfer: mooncake + multinode: true + disagg: true + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + # 1P1D TP4 + - conc-list: [ 1, 2, 4, 8, 16, 32, 64, 128, 256 ] + prefill: + num-worker: 1 + tp: 4 + ep: 1 + dp-attn: false + additional-settings: + - "PREFILL_NODES=1" + decode: + num-worker: 1 + tp: 4 + ep: 1 + dp-attn: false + additional-settings: + - "DECODE_NODES=1" + - "DECODE_MTP_SIZE=0" + # 2P1D, DPA TP4 + - conc-list: [ 256, 512, 768, 1024 ] + prefill: + num-worker: 2 + tp: 4 + ep: 1 + dp-attn: true + additional-settings: + - "PREFILL_NODES=2" + decode: + num-worker: 1 + tp: 4 + ep: 1 + dp-attn: true + additional-settings: + - "DECODE_NODES=1" + - "DECODE_MTP_SIZE=0" + # 1P1D TP4 +minimaxm3-fp8-mi300x-vllm: + image: vllm/vllm-openai-rocm:nightly-4559c43a9526597c00cbcc4f59979496500268d1 + model: MiniMaxAI/MiniMax-M3-MXFP8 + model-prefix: minimaxm3 + runner: mi300x + precision: fp8 + framework: vllm + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - { tp: 8, conc-start: 1, conc-end: 256 } + +# EAGLE3 speculative-decoding (spec-decoding: mtp) variant of +# minimaxm3-fp8-mi300x-vllm, pairing MiniMaxAI/MiniMax-M3-MXFP8 with the +# Inferact/MiniMax-M3-EAGLE3 draft head (3 speculative tokens). Same TP8-only +# search space as the non-MTP MI300X entry (gfx942 192 GB is memory-tight, like +# H100), with the TP8 latency rows started at conc 1 to capture single-request +# latency — matching the H100/MI355X MTP recipes. The pinned ROCm nightly +# includes upstream SupportsEagle3 support for the AMD MiniMax-M3 model. +minimaxm3-fp8-mi300x-vllm-mtp: + image: vllm/vllm-openai-rocm:nightly-b53b1c7ffe7aebdafd0876350f30e51d1226c92a + model: MiniMaxAI/MiniMax-M3-MXFP8 + model-prefix: minimaxm3 + runner: mi300x + precision: fp8 + framework: vllm + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - { tp: 8, conc-start: 1, conc-end: 64, spec-decoding: mtp } + - { tp: 8, ep: 8, conc-start: 128, conc-end: 256, spec-decoding: mtp } + +# MiniMax-M3 MXFP8 MI325X day-zero recipe. Reuse the dedicated ROCm image +# and serving flags validated on MI355X, with the H200 search space: TP4 and +# TP8 latency, TP4/TP8 expert parallelism, and TP8 data-parallel attention. +minimaxm3-fp8-mi325x-vllm: + image: vllm/vllm-openai-rocm:minimax-m3 + model: MiniMaxAI/MiniMax-M3-MXFP8 + model-prefix: minimaxm3 + runner: mi325x + precision: fp8 + framework: vllm + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - { tp: 4, conc-start: 1, conc-end: 32 } + - { tp: 8, conc-start: 1, conc-end: 128 } + - { tp: 8, ep: 8, conc-start: 256, conc-end: 256 } + - { tp: 8, ep: 8, dp-attn: true, conc-start: 256, conc-end: 512 } + +# EAGLE3 speculative-decoding (spec-decoding: mtp) variant of +# minimaxm3-fp8-mi325x-vllm, pairing MiniMaxAI/MiniMax-M3-MXFP8 with the +# Inferact/MiniMax-M3-EAGLE3 draft head (3 speculative tokens). Same H200-style +# search space as the non-MTP MI325X entry, trimmed at the extreme-concurrency +# end with TP-only latency rows started at conc 1 (matching the H200/MI355X MTP +# recipes). Runs with CUDA graphs (no --enforce-eager, VLLM_USE_BREAKABLE_CUDAGRAPH=0, +# BF16 KV on gfx942). The shipped ROCm image lacks SupportsEagle3 on the AMD +# MiniMax-M3 model, so the recipe applies that fix in-place at runtime +# (functionstackx/vllm#1, upstream vllm-project/vllm#45546; validated green on +# MI355X/MI300X) before serving. +minimaxm3-fp8-mi325x-vllm-mtp: + image: vllm/vllm-openai-rocm:minimax-m3 + model: MiniMaxAI/MiniMax-M3-MXFP8 + model-prefix: minimaxm3 + runner: mi325x + precision: fp8 + framework: vllm + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - { tp: 4, conc-start: 1, conc-end: 32, spec-decoding: mtp } + - { tp: 8, conc-start: 1, conc-end: 128, spec-decoding: mtp } + - { tp: 8, ep: 8, conc-start: 256, conc-end: 256, spec-decoding: mtp } + - { tp: 8, ep: 8, dp-attn: true, conc-start: 256, conc-end: 256, spec-decoding: mtp } + +# MiniMax-M3 MXFP8 MI355X vLLM disaggregated (prefill/decode) sweep on the +# day-zero ROCm image, over the MoRI-IO KV-transfer pipeline (MoRI-patch-removal +# infra #1585). All workers are TP4, no EP: the single-node M3 MXFP8 recipe +# (minimaxm3-fp8-mi355x-vllm, PR #2003) found plain TP4 beats both TP8 and +# TP4/EP4 on tok/s/GPU for this model on gfx950, so prefill and decode both use +# TP4 and we tune the prefill:decode worker ratio (xP:yD) instead of TP. The +# mi355x-disagg pool has 3 nodes and the launcher places one worker per node +# (NUM_NODES = xP + yD), so every layout keeps xP + yD <= 3: +# - 1P-TP4 / 1D-TP4 (2 nodes): balanced, full concurrency curve. +# - 1P-TP4 / 2D-TP4 (3 nodes): decode-heavy, for the decode-bound 1k1k tail. +# - 2P-TP4 / 1D-TP4 (3 nodes): prefill-heavy, for the prefill-bound 8k1k tail. +# Per-worker serve flags live in +# benchmarks/multi_node/amd_utils/models_vllm.yaml (MiniMax-M3-MXFP8). +minimaxm3-fp8-mi355x-vllm-disagg: + image: vllm/vllm-openai-rocm:nightly-2afa3f7e950264bb179d030c23a1ed1f46558fd9 + model: MiniMaxAI/MiniMax-M3-MXFP8 + model-prefix: minimaxm3 + runner: mi355x-disagg + precision: fp8 + framework: vllm-disagg + router: { name: vllm-router, version: "0.1.14" } + kv-p2p-transfer: moriio + multinode: true + disagg: true + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + # Balanced 1P TP4 + 1D TP4 (2 nodes) across the full curve. + - spec-decoding: "none" + conc-list: [ 1, 2, 4, 8, 16, 32, 64, 128, 256, 512 ] + prefill: + num-worker: 1 + tp: 4 + ep: 1 + dp-attn: false + additional-settings: + - "PREFILL_NODES=1" + decode: + num-worker: 1 + tp: 4 + ep: 1 + dp-attn: false + additional-settings: + - "DECODE_NODES=1" + # Prefill-heavy 2P TP4 + 1D TP4 (3 nodes): two half-node TP4 prefill workers + # keep the single TP4 decode engine fed for the prefill-bound 8k1k tail. + - spec-decoding: "none" + conc-list: [ 128, 256, 512 ] + prefill: + num-worker: 2 + tp: 4 + ep: 1 + dp-attn: false + additional-settings: + - "PREFILL_NODES=2" + decode: + num-worker: 1 + tp: 4 + ep: 1 + dp-attn: false + additional-settings: + - "DECODE_NODES=1" diff --git a/configs/deprecated/amd-qwen3.5-bf16-master.yaml b/configs/deprecated/amd-qwen3.5-bf16-master.yaml new file mode 100644 index 0000000000..945786c57f --- /dev/null +++ b/configs/deprecated/amd-qwen3.5-bf16-master.yaml @@ -0,0 +1,81 @@ +# Deprecated Qwen3.5-397B-A17B bf16 entries archived from amd-master.yaml. +# All bf16 recipes, in every scenario, were deprecated after Monday, +# August 3, 2026 (see the Deprecation Notice in MODELS.md). fp8 and fp4 +# recipes remain active. +# Removed from the active master config so sweep generation no longer selects them. + + +qwen3.5-bf16-mi355x-sglang: + image: lmsysorg/sglang-rocm:v0.5.12-rocm720-mi35x-20260517 + model: Qwen/Qwen3.5-397B-A17B + model-prefix: qwen3.5 + runner: mi355x + precision: bf16 + framework: sglang + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - { tp: 8, ep: 1, conc-start: 4, conc-end: 256 } + +qwen3.5-bf16-mi355x-sglang-mtp: + image: lmsysorg/sglang-rocm:v0.5.12-rocm720-mi35x-20260517 + model: Qwen/Qwen3.5-397B-A17B + model-prefix: qwen3.5 + runner: mi355x + precision: bf16 + framework: sglang + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - { tp: 8, ep: 1, conc-start: 4, conc-end: 256, spec-decoding: mtp } + +qwen3.5-bf16-mi300x-sglang: + image: lmsysorg/sglang:v0.5.12-rocm720-mi30x + model: Qwen/Qwen3.5-397B-A17B + model-prefix: qwen3.5 + runner: mi300x + precision: bf16 + framework: sglang + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - { tp: 8, conc-start: 4, conc-end: 64 } + +qwen3.5-bf16-mi325x-sglang: + image: lmsysorg/sglang:v0.5.12-rocm720-mi30x + model: Qwen/Qwen3.5-397B-A17B + model-prefix: qwen3.5 + runner: mi325x + precision: bf16 + framework: sglang + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - { tp: 8, conc-start: 4, conc-end: 64 } + +qwen3.5-bf16-mi325x-sglang-mtp: + image: lmsysorg/sglang:v0.5.12-rocm720-mi30x + model: Qwen/Qwen3.5-397B-A17B + model-prefix: qwen3.5 + runner: mi325x + precision: bf16 + framework: sglang + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - { tp: 8, ep: 1, conc-start: 4, conc-end: 64, spec-decoding: mtp } diff --git a/configs/deprecated/nvidia-kimik2.5-agentic-master.yaml b/configs/deprecated/nvidia-kimik2.5-agentic-master.yaml new file mode 100644 index 0000000000..ea21aacf91 --- /dev/null +++ b/configs/deprecated/nvidia-kimik2.5-agentic-master.yaml @@ -0,0 +1,91 @@ +# Deprecated Kimi-K2.5/2.6/2.7-Code agentic-coding entries archived from +# nvidia-master.yaml. Deprecated after Monday, August 3, 2026 (see the +# Deprecation Notice in MODELS.md). Single-turn 8k1k stays active until +# August 6, 2026. +# Removed from the active master config so sweep generation no longer selects them. + + +kimik2.5-int4-b200-vllm-agentic: + image: vllm/vllm-openai:v0.22.0 + model: moonshotai/Kimi-K2.5 + model-prefix: kimik2.5 + runner: cluster:b200-dgxc + precision: int4 + framework: vllm + multinode: false + scenarios: + agentic-coding: + - dram-utilization: 0.80 + search-space: + - { tp: 8, kv-offloading: none, conc-list: [1, 2, 4, 8, 16, 32] } + - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, conc-list: [32, 64, 96, 128] } + +kimik2.5-int4-h200-vllm-agentic: + image: vllm/vllm-openai:v0.22.0 + model: moonshotai/Kimi-K2.5 + model-prefix: kimik2.5 + runner: cluster:h200-dgxc + precision: int4 + framework: vllm + multinode: false + scenarios: + agentic-coding: + - dram-utilization: 0.80 + search-space: + - { tp: 8, kv-offloading: none, conc-list: [1, 2, 3, 4, 5, 6, 7] } + - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, conc-list: [6, 7, 8, 9, 10, 11, 12, 13, 14] } + +kimik2.5-fp4-b300-vllm-agentic: + # v0.20.2 (cu129) lacks the flashinfer kernels for B300's reported SM + # (sm_12x); workers hit "Only SM 10.x and 11.x are supported" in the + # trtllm_fp4_block_scale_moe path. v0.20.0-cu130 is the Blackwell-targeted + # build that has the full sm_10x/sm_11x/sm_12x kernel set and is what the + # INT4 B300 sister already uses successfully. + image: vllm/vllm-openai:v0.22.0 + model: nvidia/Kimi-K2.5-NVFP4 + model-prefix: kimik2.5 + runner: cluster:b300-nv + precision: fp4 + framework: vllm + multinode: false + scenarios: + agentic-coding: + - dram-utilization: 0.80 + search-space: + - { tp: 8, ep: 1, kv-offloading: none, conc-list: [1, 2, 4, 8, 16, 32, 40, 48, 56, 64] } + - { tp: 8, ep: 1, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, conc-list: [1, 2, 4, 8, 16, 32, 40, 48, 56, 64] } + +kimik2.5-fp4-b300-vllm-agentic-mtp: + image: vllm/vllm-openai:nightly-94c0ef300180f8fd1071d9cbe7270a8348155f94 + model: Kimi-K2.6-NVFP4 + model-prefix: kimik2.5 + runner: cluster:b300-nv + precision: fp4 + framework: vllm + multinode: false + scenarios: + agentic-coding: + - dram-utilization: 0.80 + search-space: + - { tp: 8, ep: 1, spec-decoding: mtp, kv-offloading: none, conc-list: [1] } + - { tp: 4, ep: 1, spec-decoding: mtp, kv-offloading: none, conc-list: [2, 4, 8] } + - { tp: 4, ep: 1, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: { name: native }, conc-list: [8, 16, 32] } + - { tp: 4, ep: 1, spec-decoding: mtp, dcp-size: 4, kv-offloading: none, conc-list: [32, 64, 80, 96, 112, 128] } + - { tp: 4, ep: 1, spec-decoding: mtp, dcp-size: 4, kv-offloading: dram, kv-offload-backend: { name: native }, conc-list: [64, 80, 96, 112, 128, 144, 160] } + +kimik2.5-fp4-b200-vllm-agentic-lmcache: + image: vllm/vllm-openai:v0.22.0 + model: nvidia/Kimi-K2.5-NVFP4 + model-prefix: kimik2.5 + runner: cluster:b200-dgxc + precision: fp4 + framework: vllm + multinode: false + scenarios: + agentic-coding: + - dram-utilization: 0.80 + search-space: + - { tp: 8, ep: 1, kv-offloading: none, conc-list: [1, 2, 4, 8, 16, 24] } + - { tp: 8, ep: 1, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "0.5.1" }, conc-list: [16, 24, 32, 36] } + - { tp: 4, ep: 1, kv-offloading: none, conc-list: [8, 12, 14, 16, 18, 20] } + - { tp: 4, ep: 1, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "0.5.1" }, conc-list: [12, 14, 16, 18, 20, 22, 24, 32] } diff --git a/configs/deprecated/nvidia-minimaxm3-8k1k-master.yaml b/configs/deprecated/nvidia-minimaxm3-8k1k-master.yaml new file mode 100644 index 0000000000..05461cc43f --- /dev/null +++ b/configs/deprecated/nvidia-minimaxm3-8k1k-master.yaml @@ -0,0 +1,1341 @@ +# Deprecated MiniMax-M3 Single-turn 8k1k entries archived from nvidia-master.yaml. +# The scenario was deprecated after Monday, August 3, 2026 (see the Deprecation +# Notice in MODELS.md). Agentic coding remains active for this model. +# Removed from the active master config so sweep generation no longer selects them. + + +minimaxm3-fp8-h100-vllm: + image: vllm/vllm-openai:minimax-m3 + model: MiniMaxAI/MiniMax-M3-MXFP8 + model-prefix: minimaxm3 + runner: h100 + precision: fp8 + framework: vllm + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - { tp: 8, conc-start: 1, conc-end: 64 } + - { tp: 8, ep: 8, conc-start: 1, conc-end: 256 } + +minimaxm3-fp8-h200-vllm: + image: vllm/vllm-openai:minimax-m3 + model: MiniMaxAI/MiniMax-M3-MXFP8 + model-prefix: minimaxm3 + runner: h200 + precision: fp8 + framework: vllm + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - { tp: 4, conc-start: 1, conc-end: 32 } + - { tp: 4, ep: 4, conc-start: 1, conc-end: 256 } + - { tp: 8, conc-start: 1, conc-end: 128 } + - { tp: 8, ep: 8, conc-start: 1, conc-end: 256 } + - { tp: 8, ep: 8, dp-attn: true, conc-start: 256, conc-end: 512 } + +minimaxm3-fp4-b200-dynamo-vllm: + image: vllm/vllm-openai:nightly-5e35a6f4f9bbc217c599692157ca985c894373f7 + model: nvidia/MiniMax-M3-NVFP4 + model-prefix: minimaxm3 + runner: b200-multinode + precision: fp4 + framework: dynamo-vllm + router: { name: dynamo-router, version: "1.3.0.dev20260710" } + kv-p2p-transfer: nixl + multinode: true + disagg: true + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - conc-list: [1, 4, 8, 16] + prefill: + num-worker: 1 + tp: 2 + ep: 2 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3/b200-fp4/8k1k/1p1d-dep2-tp4-c1-c16-8k1k.yaml" + decode: + num-worker: 1 + tp: 4 + ep: 1 + dp-attn: false + - conc-list: [64] + prefill: + num-worker: 1 + tp: 2 + ep: 2 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3/b200-fp4/8k1k/1p2d-dep2-tp4-c64-8k1k.yaml" + decode: + num-worker: 2 + tp: 4 + ep: 1 + dp-attn: false + - conc-list: [128] + prefill: + num-worker: 2 + tp: 2 + ep: 2 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3/b200-fp4/8k1k/2p2d-dep2-tp4-c128-8k1k.yaml" + decode: + num-worker: 2 + tp: 4 + ep: 1 + dp-attn: false + - conc-list: [256] + prefill: + num-worker: 3 + tp: 2 + ep: 2 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3/b200-fp4/8k1k/3p2d-dep2-tp4-c256-8k1k.yaml" + decode: + num-worker: 2 + tp: 4 + ep: 1 + dp-attn: false + - conc-list: [512] + prefill: + num-worker: 3 + tp: 2 + ep: 2 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3/b200-fp4/8k1k/3p2d-dep2-tp4-c512-8k1k.yaml" + decode: + num-worker: 2 + tp: 4 + ep: 1 + dp-attn: false + - conc-list: [1024] + prefill: + num-worker: 4 + tp: 2 + ep: 2 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3/b200-fp4/8k1k/4p2d-dep2-tp4-c1024-8k1k.yaml" + decode: + num-worker: 2 + tp: 4 + ep: 1 + dp-attn: false + +# MiniMax-M3 NVFP4 disagg sweep on the same B300 topology matrix as the MXFP8 +# baseline above. The image includes vLLM PR #46380, so no runtime patch is +# needed. +minimaxm3-fp8-b300-dynamo-vllm: + image: vllm/vllm-openai:nightly-4080263bb2c5d10deac17aaeb88e0823bc35bca9 + model: MiniMaxAI/MiniMax-M3-MXFP8 + model-prefix: minimaxm3 + runner: b300 + precision: fp8 + framework: dynamo-vllm + router: { name: dynamo-router, version: "1.3.0.dev20260614" } + kv-p2p-transfer: nixl + multinode: true + disagg: true + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - conc-list: [256, 512] + prefill: + num-worker: 2 + tp: 2 + ep: 2 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3/b300-fp8/8k1k/2p2d-dep2-dep8-8k1k.yaml" + decode: + num-worker: 2 + tp: 8 + ep: 8 + dp-attn: true + - conc-list: [16] + prefill: + num-worker: 2 + tp: 2 + ep: 2 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3/b300-fp8/8k1k/2p2d-dep2-tep8-8k1k.yaml" + decode: + num-worker: 2 + tp: 8 + ep: 8 + dp-attn: false + - conc-list: [4096] + prefill: + num-worker: 4 + tp: 2 + ep: 2 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3/b300-fp8/8k1k/4p2d-dep2-dep8-8k1k.yaml" + decode: + num-worker: 2 + tp: 8 + ep: 8 + dp-attn: true + - conc-list: [1, 4, 8, 16] + prefill: + num-worker: 1 + tp: 2 + ep: 2 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3/b300-fp8/8k1k/1p1d-dep2-tp4-marlin-8k1k.yaml" + decode: + num-worker: 1 + tp: 4 + ep: 1 + dp-attn: false + - conc-list: [1024, 2048] + prefill: + num-worker: 3 + tp: 2 + ep: 2 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3/b300-fp8/8k1k/3p2d-dep2-dep8-8k1k.yaml" + decode: + num-worker: 2 + tp: 8 + ep: 8 + dp-attn: true + - conc-list: [16, 32, 64, 128] + prefill: + num-worker: 1 + tp: 2 + ep: 2 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3/b300-fp8/8k1k/1p4d-dep2-tep4-8k1k.yaml" + decode: + num-worker: 4 + tp: 4 + ep: 4 + dp-attn: false + - conc-list: [16] + prefill: + num-worker: 1 + tp: 2 + ep: 2 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3/b300-fp8/8k1k/1p2d-dep2-tep4-8k1k.yaml" + decode: + num-worker: 2 + tp: 4 + ep: 4 + dp-attn: false + - conc-list: [4] + prefill: + num-worker: 1 + tp: 2 + ep: 2 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3/b300-fp8/8k1k/1p4d-dep2-tep8-8k1k.yaml" + decode: + num-worker: 4 + tp: 8 + ep: 8 + dp-attn: false + +# MiniMax-M3 NVFP4 disagg sweep on the same B300 topology matrix as the MXFP8 +# baseline above. The image includes vLLM PR #46380, so no runtime patch is +# needed. +minimaxm3-fp4-b300-dynamo-vllm-8k1k-tp1: + image: vllm/vllm-openai:nightly-5e35a6f4f9bbc217c599692157ca985c894373f7 + model: nvidia/MiniMax-M3-NVFP4 + model-prefix: minimaxm3 + runner: b300 + precision: fp4 + framework: dynamo-vllm + router: { name: dynamo-router, version: "1.3.0.dev20260713" } + kv-p2p-transfer: nixl + multinode: true + disagg: true + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - conc-list: [8, 16, 24, 32, 48, 64] + prefill: + num-worker: 1 + tp: 1 + ep: 1 + dp-attn: false + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3/b300-fp4/8k1k/1p1d-tp1-tp4-8k1k.yaml" + decode: + num-worker: 1 + tp: 4 + ep: 1 + dp-attn: false + - conc-list: [1, 2, 4, 8, 16] + prefill: + num-worker: 1 + tp: 1 + ep: 1 + dp-attn: false + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3/b300-fp4/8k1k/1p1d-tp1-tp8-8k1k.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 1 + dp-attn: false + - conc-list: [128] + prefill: + num-worker: 2 + tp: 1 + ep: 1 + dp-attn: false + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3/b300-fp4/8k1k/2p1d-tp1-tp4-8k1k.yaml" + decode: + num-worker: 1 + tp: 4 + ep: 1 + dp-attn: false + - conc-list: [256] + prefill: + num-worker: 3 + tp: 1 + ep: 1 + dp-attn: false + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3/b300-fp4/8k1k/3p1d-tp1-tp4-8k1k.yaml" + decode: + num-worker: 1 + tp: 4 + ep: 1 + dp-attn: false + - conc-list: [512] + prefill: + num-worker: 5 + tp: 1 + ep: 1 + dp-attn: false + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3/b300-fp4/8k1k/5p2d-tp1-dep8-8k1k.yaml" + decode: + num-worker: 2 + tp: 8 + ep: 8 + dp-attn: true + - conc-list: [768, 1024] + prefill: + num-worker: 8 + tp: 1 + ep: 1 + dp-attn: false + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3/b300-fp4/8k1k/8p2d-tp1-dep8-8k1k.yaml" + decode: + num-worker: 2 + tp: 8 + ep: 8 + dp-attn: true + +# Compare the refreshed image on a one-node max-throughput topology: four +# TP1 prefills colocated with one TEP4 decode worker. +minimaxm3-fp4-b300-dynamo-vllm-8k1k-tp1-tep4: + image: vllm/vllm-openai:nightly-5e35a6f4f9bbc217c599692157ca985c894373f7 + model: nvidia/MiniMax-M3-NVFP4 + model-prefix: minimaxm3 + runner: b300 + precision: fp4 + framework: dynamo-vllm + router: { name: dynamo-router, version: "1.3.0.dev20260713" } + kv-p2p-transfer: nixl + multinode: true + disagg: true + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - conc-list: [4096] + prefill: + num-worker: 4 + tp: 1 + ep: 1 + dp-attn: false + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3/b300-fp4/8k1k/4p1d-tp1-tep4-8k1k.yaml" + decode: + num-worker: 1 + tp: 4 + ep: 4 + dp-attn: false + +# Preserve the previously proven max-throughput point exactly, while keeping +# it separate from the legacy mixed 1k1k/8k1k entry so 1k1k is not rerun. +minimaxm3-fp4-b300-dynamo-vllm-8k1k-legacy-max-tput: + image: vllm/vllm-openai:vllm-minimax-m3-perf-x86_64-13.0.1-8b00f41 + model: nvidia/MiniMax-M3-NVFP4 + model-prefix: minimaxm3 + runner: b300 + precision: fp4 + framework: dynamo-vllm + router: { name: dynamo-router, version: "1.3.0.dev20260614" } + kv-p2p-transfer: nixl + multinode: true + disagg: true + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - conc-list: [4096] + prefill: + num-worker: 4 + tp: 2 + ep: 2 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3/b300-fp4/8k1k/4p2d-dep2-tep4-8k1k.yaml" + decode: + num-worker: 2 + tp: 4 + ep: 4 + dp-attn: false + +minimaxm3-fp4-b200-dynamo-vllm-mtp: + image: vllm/vllm-openai:nightly-5e35a6f4f9bbc217c599692157ca985c894373f7 + model: nvidia/MiniMax-M3-NVFP4 + model-prefix: minimaxm3 + runner: b200-multinode + precision: fp4 + framework: dynamo-vllm + router: { name: dynamo-router, version: "1.3.0.dev20260710" } + kv-p2p-transfer: nixl + multinode: true + disagg: true + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - spec-decoding: "mtp" + conc-list: [12, 24, 48] + prefill: + num-worker: 1 + tp: 4 + ep: 1 + dp-attn: false + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3/b200-fp4/8k1k/mtp/1p6d-tp4-tp4-eagle3-8k1k.yaml" + decode: + num-worker: 6 + tp: 4 + ep: 1 + dp-attn: false + - spec-decoding: "mtp" + conc-list: [4, 8, 16, 32, 64] + prefill: + num-worker: 1 + tp: 4 + ep: 1 + dp-attn: false + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3/b200-fp4/8k1k/mtp/1p4d-tp4-tp4-eagle3-8k1k.yaml" + decode: + num-worker: 4 + tp: 4 + ep: 1 + dp-attn: false + - spec-decoding: "mtp" + conc-list: [32, 64, 96] + prefill: + num-worker: 1 + tp: 4 + ep: 1 + dp-attn: false + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3/b200-fp4/8k1k/mtp/1p2d-tp4-tp4-eagle3-8k1k.yaml" + decode: + num-worker: 2 + tp: 4 + ep: 1 + dp-attn: false + - spec-decoding: "mtp" + conc-list: [192, 256, 512] + prefill: + num-worker: 1 + tp: 4 + ep: 1 + dp-attn: false + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3/b200-fp4/8k1k/mtp/1p1d-tp4-tp4-eagle3-8k1k.yaml" + decode: + num-worker: 1 + tp: 4 + ep: 1 + dp-attn: false + + - spec-decoding: "mtp" + conc-list: [12, 24, 48] + prefill: + num-worker: 1 + tp: 4 + ep: 1 + dp-attn: false + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3/b200-fp4/8k1k/mtp/1p3d-tp4-tp4-eagle3-8k1k.yaml" + decode: + num-worker: 3 + tp: 4 + ep: 1 + dp-attn: false + + + - spec-decoding: "mtp" + conc-list: [256, 512] + prefill: + num-worker: 1 + tp: 4 + ep: 1 + dp-attn: false + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3/b200-fp4/8k1k/mtp/1p1d-tp4-dep4-eagle3-8k1k.yaml" + decode: + num-worker: 1 + tp: 4 + ep: 4 + dp-attn: true + +minimaxm3-fp4-b300-dynamo-vllm-mtp: + image: vllm/vllm-openai:nightly-5e35a6f4f9bbc217c599692157ca985c894373f7 + model: nvidia/MiniMax-M3-NVFP4 + model-prefix: minimaxm3 + runner: b300 + precision: fp4 + framework: dynamo-vllm + router: { name: dynamo-router, version: "1.3.0.dev20260710" } + kv-p2p-transfer: nixl + multinode: true + disagg: true + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - spec-decoding: "mtp" + conc-list: [24, 48] + prefill: + num-worker: 1 + tp: 2 + ep: 2 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3/b300-fp4/8k1k/mtp/1p6d-dep2-tp4-eagle3-8k1k.yaml" + decode: + num-worker: 6 + tp: 4 + ep: 1 + dp-attn: false + - spec-decoding: "mtp" + conc-list: [4, 8, 64] + prefill: + num-worker: 1 + tp: 2 + ep: 2 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3/b300-fp4/8k1k/mtp/1p4d-dep2-tp4-eagle3-8k1k.yaml" + decode: + num-worker: 4 + tp: 4 + ep: 1 + dp-attn: false + - spec-decoding: "mtp" + conc-list: [4, 32, 64] + prefill: + num-worker: 1 + tp: 2 + ep: 2 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3/b300-fp4/8k1k/mtp/1p2d-dep2-tp4-eagle3-8k1k.yaml" + decode: + num-worker: 2 + tp: 4 + ep: 1 + dp-attn: false + - spec-decoding: "mtp" + conc-list: [64] + prefill: + num-worker: 1 + tp: 2 + ep: 2 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3/b300-fp4/8k1k/mtp/1p1d-dep2-tp4-eagle3-8k1k.yaml" + decode: + num-worker: 1 + tp: 4 + ep: 1 + dp-attn: false + + - spec-decoding: "mtp" + conc-list: [192] + prefill: + num-worker: 2 + tp: 2 + ep: 2 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3/b300-fp4/8k1k/mtp/2p3d-dep2-tp4-eagle3-8k1k.yaml" + decode: + num-worker: 3 + tp: 4 + ep: 1 + dp-attn: false + +minimaxm3-fp4-b300-dynamo-vllm-mtp-legacy-dep4: + image: vllm/vllm-openai:nightly-5e35a6f4f9bbc217c599692157ca985c894373f7 + model: nvidia/MiniMax-M3-NVFP4 + model-prefix: minimaxm3 + runner: b300 + precision: fp4 + framework: dynamo-vllm + router: { name: dynamo-router, version: "1.3.0.dev20260710" } + kv-p2p-transfer: nixl + multinode: true + disagg: true + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - spec-decoding: "mtp" + conc-list: [256, 512] + prefill: + num-worker: 2 + tp: 2 + ep: 2 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3/b300-fp4/8k1k/mtp/2p1d-dep2-dep4-eagle3-8k1k.yaml" + decode: + num-worker: 1 + tp: 4 + ep: 4 + dp-attn: true +# MiniMax-M3 GB300 disagg sweep — refreshed recipe set (no Marlin variants). +# All prefill DEP2 (TP1 DP2 EP, 2 GPU/worker). Decode: DEP4, TEP8, DEP8, TEP4. +# 4 GPU/node (GB300 NVL72). kv-cache-dtype=fp8. srun_options mem=0 required. +minimaxm3-fp8-gb300-dynamo-vllm-mtp: + image: vllm/vllm-openai:nightly-5e35a6f4f9bbc217c599692157ca985c894373f7 + model: MiniMaxAI/MiniMax-M3-MXFP8 + model-prefix: minimaxm3 + runner: gb300-nv + precision: fp8 + framework: dynamo-vllm + router: { name: dynamo-router, version: "1.4.0.dev20260730" } + kv-p2p-transfer: nixl + multinode: true + disagg: true + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - spec-decoding: "mtp" + conc-list: [1] + prefill: + num-worker: 1 + tp: 2 + ep: 2 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3-gb300-fp8/8k1k/mtp/1p1d-dep2-tp4-eagle3-c1-8k1k.yaml" + decode: + num-worker: 1 + tp: 4 + ep: 1 + dp-attn: false + - spec-decoding: "mtp" + conc-list: [8] + prefill: + num-worker: 1 + tp: 2 + ep: 2 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3-gb300-fp8/8k1k/mtp/1p1d-dep2-tp4-eagle3-c8-8k1k.yaml" + decode: + num-worker: 1 + tp: 4 + ep: 1 + dp-attn: false + - spec-decoding: "mtp" + conc-list: [1] + prefill: + num-worker: 1 + tp: 2 + ep: 2 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3-gb300-fp8/8k1k/mtp/1p1d-dep2-tp8-eagle3-c1-8k1k.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 1 + dp-attn: false + - spec-decoding: "mtp" + conc-list: [4] + prefill: + num-worker: 1 + tp: 2 + ep: 2 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3-gb300-fp8/8k1k/mtp/1p1d-dep2-tp8-eagle3-c4-8k1k.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 1 + dp-attn: false + - spec-decoding: "mtp" + conc-list: [8] + prefill: + num-worker: 1 + tp: 2 + ep: 2 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3-gb300-fp8/8k1k/mtp/1p1d-dep2-tp8-eagle3-c8-8k1k.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 1 + dp-attn: false + - spec-decoding: "mtp" + conc-list: [64] + prefill: + num-worker: 1 + tp: 2 + ep: 2 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3-gb300-fp8/8k1k/mtp/1p1d-dep2-dep8-eagle3-c64-8k1k.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + - spec-decoding: "mtp" + conc-list: [512] + prefill: + num-worker: 2 + tp: 2 + ep: 2 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3-gb300-fp8/8k1k/mtp/2p1d-dep2-dep8-eagle3-c512-8k1k.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + - spec-decoding: "mtp" + conc-list: [256] + prefill: + num-worker: 3 + tp: 2 + ep: 2 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3-gb300-fp8/8k1k/mtp/3p1d-dep2-dep8-eagle3-c256-8k1k.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + - spec-decoding: "mtp" + conc-list: [1024] + prefill: + num-worker: 4 + tp: 2 + ep: 2 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3-gb300-fp8/8k1k/mtp/4p1d-dep2-dep8-eagle3-c1024-8k1k.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + - spec-decoding: "mtp" + conc-list: [2048] + prefill: + num-worker: 4 + tp: 2 + ep: 2 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3-gb300-fp8/8k1k/mtp/4p1d-dep2-dep8-eagle3-c2048-8k1k.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + - spec-decoding: "mtp" + conc-list: [2048] + prefill: + num-worker: 6 + tp: 2 + ep: 2 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3-gb300-fp8/8k1k/mtp/6p1d-dep2-dep8-eagle3-c2048-8k1k.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + + +minimaxm3-fp8-gb300-dynamo-vllm: + image: vllm/vllm-openai:nightly-4080263bb2c5d10deac17aaeb88e0823bc35bca9 + model: MiniMaxAI/MiniMax-M3-MXFP8 + model-prefix: minimaxm3 + runner: gb300-nv + precision: fp8 + framework: dynamo-vllm + router: { name: dynamo-router, version: "1.3.0.dev20260614" } + kv-p2p-transfer: nixl + multinode: true + disagg: true + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + # 1p1d DEP2+DEP8, 3n: conc 256 + - conc-list: [256] + prefill: + num-worker: 1 + tp: 2 + ep: 2 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3-gb300-fp8/8k1k/1p1d-dep2-dep8-8k1k.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + + # 1p1d DEP2+TEP8, 3n: conc 128 + - conc-list: [128] + prefill: + num-worker: 1 + tp: 2 + ep: 2 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3-gb300-fp8/8k1k/1p1d-dep2-tep8-8k1k.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: false + + # 1p2d DEP2+TEP8, 5n: conc 32,64,128 + - conc-list: [32, 64, 128] + prefill: + num-worker: 1 + tp: 2 + ep: 2 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3-gb300-fp8/8k1k/1p2d-dep2-tep8-8k1k.yaml" + decode: + num-worker: 2 + tp: 8 + ep: 8 + dp-attn: false + + # 2p1d DEP2+DEP8, 3n: conc 512 + - conc-list: [512] + prefill: + num-worker: 2 + tp: 2 + ep: 2 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3-gb300-fp8/8k1k/2p1d-dep2-dep8-8k1k.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + + # 2p2d DEP2+TEP8, 5n: conc 16 + - conc-list: [16] + prefill: + num-worker: 2 + tp: 2 + ep: 2 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3-gb300-fp8/8k1k/2p2d-dep2-tep8-8k1k.yaml" + decode: + num-worker: 2 + tp: 8 + ep: 8 + dp-attn: false + + # 2p4d DEP2+TEP4, 5n: conc 4 + - conc-list: [4] + prefill: + num-worker: 2 + tp: 2 + ep: 2 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3-gb300-fp8/8k1k/2p4d-dep2-tep4-8k1k.yaml" + decode: + num-worker: 4 + tp: 4 + ep: 4 + dp-attn: false + + # 3p1d DEP2+DEP8, 4n: conc 1024 + - conc-list: [1024] + prefill: + num-worker: 3 + tp: 2 + ep: 2 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3-gb300-fp8/8k1k/3p1d-dep2-dep8-8k1k.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + + # 3p1d DEP2+DEP16, 6n: conc 512 + - conc-list: [512] + prefill: + num-worker: 3 + tp: 2 + ep: 2 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3-gb300-fp8/8k1k/3p1d-dep2-dep16-8k1k.yaml" + decode: + num-worker: 1 + tp: 16 + ep: 16 + dp-attn: true + + # 6p1d DEP2+DEP8, 5n: conc 2048 + - conc-list: [2048] + prefill: + num-worker: 6 + tp: 2 + ep: 2 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3-gb300-fp8/8k1k/6p1d-dep2-dep8-8k1k.yaml" + decode: + num-worker: 1 + tp: 8 + ep: 8 + dp-attn: true + +# MiniMax-M3 GB200 disagg sweep — adapted from NV B300 PR #1863. +# All prefill DEP4 (TP1 DP4 EP, 4 GPU/worker). Decode: Marlin, TEP8, DEP8, TEP4. +# 4 GPU/node (GB200 NVL72). FLASHINFER attention with FP8 KV cache. +minimaxm3-fp8-gb200-dynamo-vllm: + image: vllm/vllm-openai:nightly-4080263bb2c5d10deac17aaeb88e0823bc35bca9 + model: MiniMaxAI/MiniMax-M3-MXFP8 + model-prefix: minimaxm3 + runner: gb200 + precision: fp8 + framework: dynamo-vllm + router: { name: dynamo-router, version: "1.3.0.dev20260614" } + kv-p2p-transfer: nixl + multinode: true + disagg: true + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + # 1p2d DEP4+DEP8, 5n: conc 512 + - conc-list: [512] + prefill: + num-worker: 1 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3-gb200-fp8/8k1k/1p1d-dep4-dep16-8k1k.yaml" + decode: + num-worker: 1 + tp: 16 + ep: 16 + dp-attn: true + + # 1p2d DEP4+TEP4, 3n: conc 4 + - conc-list: [4] + prefill: + num-worker: 1 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3-gb200-fp8/8k1k/1p2d-dep4-tep4-8k1k.yaml" + decode: + num-worker: 2 + tp: 4 + ep: 4 + dp-attn: false + + # 1p2d DEP4+TEP8, 5n: conc 4,16,32,64,128 + - conc-list: [4, 16, 32, 64, 128] + prefill: + num-worker: 1 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3-gb200-fp8/8k1k/1p2d-dep4-tep8-8k1k.yaml" + decode: + num-worker: 2 + tp: 8 + ep: 8 + dp-attn: false + + # 2p2d DEP4+DEP8, 6n: conc 512,1024 + - conc-list: [512, 1024] + prefill: + num-worker: 2 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3-gb200-fp8/8k1k/2p1d-dep4-dep16-8k1k.yaml" + decode: + num-worker: 1 + tp: 16 + ep: 16 + dp-attn: true + + # 3p2d DEP4+DEP8, 7n: conc 4096 + - conc-list: [4096] + prefill: + num-worker: 3 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3-gb200-fp8/8k1k/3p1d-dep4-dep16-8k1k.yaml" + decode: + num-worker: 1 + tp: 16 + ep: 16 + dp-attn: true + + # 5p2d DEP4+DEP8, 9n: conc 4096 + - conc-list: [4096] + prefill: + num-worker: 5 + tp: 4 + ep: 4 + dp-attn: true + additional-settings: + - "CONFIG_FILE=recipes/vllm/minimax-m3-gb200-fp8/8k1k/5p1d-dep4-dep16-8k1k.yaml" + decode: + num-worker: 1 + tp: 16 + ep: 16 + dp-attn: true + +# MiniMax-M3 day-zero (https://recipes.vllm.ai/MiniMaxAI/MiniMax-M3). +# 427B total / 26B active MoE with MSA sparse attention; MXFP8 checkpoint +# (MiniMaxAI/MiniMax-M3-MXFP8, ~444 GB) quantized by NVIDIA — native MX tensor +# cores on Blackwell. M3 support has not shipped in a stable vLLM release; +# the dedicated vllm/vllm-openai:minimax-m3-0618-x86_64-cu130 image is built +# from the m3_release branch (vllm-project/vllm#45381). --block-size 128 is mandatory (MSA +# sparse/index cache alignment). Weights are NOT SRE-staged; b300 falls back +# to writable /data/models (see launch_b300-nv.sh MODEL_PATH split). +minimaxm3-fp8-b300-vllm: + image: vllm/vllm-openai:nightly-4080263bb2c5d10deac17aaeb88e0823bc35bca9 + model: MiniMaxAI/MiniMax-M3-MXFP8 + model-prefix: minimaxm3 + runner: b300 + precision: fp8 + framework: vllm + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - { tp: 8, conc-start: 1, conc-end: 64 } + - { tp: 8, ep: 8, conc-start: 1, conc-end: 512 } + - { tp: 4, conc-start: 1, conc-end: 128 } + - { tp: 4, ep: 4, conc-start: 64, conc-end: 256 } + - { tp: 4, ep: 4, dp-attn: true, conc-start: 64, conc-end: 128 } + - { tp: 8, ep: 8, dp-attn: true, conc-start: 128, conc-end: 512 } + +# MiniMax-M3 NVFP4 (nvidia/MiniMax-M3-NVFP4) B300 single-node vLLM — FP4 variant +# of minimaxm3-fp8-b300-vllm. MiniMax-M3 modelopt NVFP4 support (vllm-project/vllm +# PR #46380) is baked into the perf container image, so no runtime patch is +# needed. --block-size 128 is mandatory (MSA sparse/index cache); +# weights are pre-staged read-only at /scratch/models/MiniMax-M3-NVFP4 (added to +# the STAGED_MODELS allow-list in launch_b300-nv.sh). +minimaxm3-fp4-b300-vllm: + image: vllm/vllm-openai:nightly-5e35a6f4f9bbc217c599692157ca985c894373f7 + model: nvidia/MiniMax-M3-NVFP4 + model-prefix: minimaxm3 + runner: b300 + precision: fp4 + framework: vllm + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - { tp: 8, conc-start: 1, conc-end: 2 } + - { tp: 4, conc-start: 1, conc-end: 2 } + - { tp: 2, conc-start: 4, conc-end: 256 } + - { tp: 2, ep: 2, dp-attn: true, conc-start: 512, conc-end: 512 } + +# EAGLE3 speculative-decoding (spec-decoding: mtp) variant of MiniMax-M3 NVFP4 +# (nvidia/MiniMax-M3-NVFP4) B300 single-node vLLM, pairing the target with the +# Inferact/MiniMax-M3-EAGLE3 draft head (3 speculative tokens). MiniMax-M3 +# modelopt NVFP4 support (vllm-project/vllm PR #46380) is baked into the perf +# container image, so no runtime patch is needed; prompts are routed through the +# chat template. Target weights are pre-staged read-only at +# /scratch/models/MiniMax-M3-NVFP4 (added to the STAGED_MODELS allow-list in +# launch_b300-nv.sh); the EAGLE3 draft is downloaded to the writable models dir. +minimaxm3-fp4-b300-vllm-mtp: + image: vllm/vllm-openai:nightly-4080263bb2c5d10deac17aaeb88e0823bc35bca9 + model: nvidia/MiniMax-M3-NVFP4 + model-prefix: minimaxm3 + runner: b300 + precision: fp4 + framework: vllm + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - { tp: 2, conc-list: [1, 2, 4, 8, 16, 32, 128, 256], spec-decoding: mtp } + - { tp: 2, ep: 2, conc-list: [1, 2, 4, 8, 32, 64, 128], spec-decoding: mtp } + - { tp: 2, ep: 2, dp-attn: true, conc-list: [128], spec-decoding: mtp } + - { tp: 4, conc-list: [1, 2, 4, 8], spec-decoding: mtp } + - { tp: 4, ep: 4, conc-list: [1, 2, 4, 8], spec-decoding: mtp } + - { tp: 8, conc-list: [1, 2, 4, 8], spec-decoding: mtp } + +# MiniMax-M3 day-zero (https://recipes.vllm.ai/MiniMaxAI/MiniMax-M3). +# 427B total / 26B active MoE with MSA sparse attention; MXFP8 checkpoint +# (MiniMaxAI/MiniMax-M3-MXFP8, ~444 GB) quantized by NVIDIA — native MX tensor +# cores on Blackwell. M3 support has not shipped in a stable vLLM release; +# the dedicated vllm/vllm-openai:minimax-m3-0618-x86_64-cu130 image is built +# from the m3_release branch (vllm-project/vllm#45381). --block-size 128 is mandatory (MSA +# sparse/index cache alignment). Weights are NOT SRE-staged: b200-dgxc reads +# /lustre/fsw/gharunners/models/MiniMax-M3-MXFP8 (pre-downloaded, see +# launch_b200-dgxc.sh). +minimaxm3-fp8-b200-vllm: + image: vllm/vllm-openai:nightly-4080263bb2c5d10deac17aaeb88e0823bc35bca9 + model: MiniMaxAI/MiniMax-M3-MXFP8 + model-prefix: minimaxm3 + runner: b200-dgxc + precision: fp8 + framework: vllm + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - { tp: 8, conc-start: 1, conc-end: 64 } + - { tp: 8, ep: 8, conc-start: 1, conc-end: 256 } + - { tp: 4, conc-start: 1, conc-end: 64 } + - { tp: 4, ep: 4, conc-start: 64, conc-end: 256 } + - { tp: 4, ep: 4, dp-attn: true, conc-start: 64, conc-end: 128 } + - { tp: 8, ep: 8, dp-attn: true, conc-start: 128, conc-end: 256 } + +# MiniMax-M3 NVFP4 (nvidia/MiniMax-M3-NVFP4) B200 single-node vLLM — FP4 variant +# of minimaxm3-fp8-b200-vllm, running on the b200-dgxc cluster. MiniMax-M3 +# modelopt NVFP4 support (vllm-project/vllm PR #46380) is baked into the perf +# container image, so no runtime patch is needed. --block-size 128 is mandatory +# (MSA sparse/index cache); weights are pre-staged at /scratch/fsw/models/MiniMax-M3-NVFP4 +# (launch_b200-dgxc.sh resolves MODEL_PATH for minimaxm3-fp4). +minimaxm3-fp4-b200-vllm: + image: vllm/vllm-openai:nightly-5e35a6f4f9bbc217c599692157ca985c894373f7 + model: nvidia/MiniMax-M3-NVFP4 + model-prefix: minimaxm3 + runner: b200-dgxc + precision: fp4 + framework: vllm + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - { tp: 8, conc-start: 1, conc-end: 4 } + - { tp: 4, conc-start: 1, conc-end: 4 } + - { tp: 2, conc-start: 1, conc-end: 128 } + - { tp: 4, conc-start: 256, conc-end: 1024 } + - { tp: 4, ep: 4, dp-attn: true, conc-start: 256, conc-end: 1024 } + +# EAGLE3 speculative-decoding (spec-decoding: mtp) variant of +# minimaxm3-fp8-b200-vllm, pairing MiniMaxAI/MiniMax-M3-MXFP8 with the +# Inferact/MiniMax-M3-EAGLE3 draft head (3 speculative tokens, drafter pinned +# to TRITON_ATTN). Search space mirrors the non-MTP entry trimmed at the +# extreme-concurrency end, identical to the minimaxm3-fp8-b300-vllm-mtp +# precedent: spec decode pays off at low/mid concurrency while acceptance +# dilutes in big batches, and the draft weights + draft KV shave headroom. +minimaxm3-fp8-b200-vllm-mtp: + image: vllm/vllm-openai:nightly-4080263bb2c5d10deac17aaeb88e0823bc35bca9 + model: MiniMaxAI/MiniMax-M3-MXFP8 + model-prefix: minimaxm3 + runner: b200-dgxc + precision: fp8 + framework: vllm + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - { tp: 8, conc-start: 1, conc-end: 64, spec-decoding: mtp } + - { tp: 8, ep: 8, conc-start: 1, conc-end: 256, spec-decoding: mtp } + - { tp: 4, conc-start: 1, conc-end: 64, spec-decoding: mtp } + - { tp: 4, ep: 4, conc-start: 64, conc-end: 256, spec-decoding: mtp } + - { tp: 8, ep: 8, dp-attn: true, conc-start: 128, conc-end: 256, spec-decoding: mtp } + +# EAGLE3 speculative-decoding (spec-decoding: mtp) variant of MiniMax-M3 NVFP4 +# (nvidia/MiniMax-M3-NVFP4) B200 single-node vLLM, pairing the target with the +# Inferact/MiniMax-M3-EAGLE3 draft head (3 speculative tokens). Runs on the +# b200-dgxc cluster. MiniMax-M3 modelopt NVFP4 support (vllm-project/vllm +# PR #46380) is baked into the perf container image, so no runtime patch is +# needed; prompts are routed through the chat template. Target weights are +# pre-staged at /scratch/fsw/models/MiniMax-M3-NVFP4 (launch_b200-dgxc.sh +# resolves MODEL_PATH for minimaxm3-fp4); the EAGLE3 draft is fetched next to +# the target weights. +minimaxm3-fp4-b200-vllm-mtp: + image: vllm/vllm-openai:nightly-4080263bb2c5d10deac17aaeb88e0823bc35bca9 + model: nvidia/MiniMax-M3-NVFP4 + model-prefix: minimaxm3 + runner: b200-dgxc + precision: fp4 + framework: vllm + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - { tp: 2, conc-list: [1, 8, 16, 32, 256], spec-decoding: mtp } + - { tp: 2, ep: 2, conc-list: [1, 2, 4, 16, 32, 64, 128, 256, 512], spec-decoding: mtp } + - { tp: 4, conc-list: [1, 2, 4, 8, 16, 256], spec-decoding: mtp } + - { tp: 4, ep: 4, conc-list: [1, 2, 4, 8, 256], spec-decoding: mtp } + - { tp: 8, conc-list: [1, 2, 4, 8], spec-decoding: mtp } + +# EAGLE3 speculative-decoding (spec-decoding: mtp) variant of +# minimaxm3-fp8-b300-vllm, pairing MiniMaxAI/MiniMax-M3-MXFP8 with the +# Inferact/MiniMax-M3-EAGLE3 draft head (3 speculative tokens). +# Search space mirrors the non-MTP entry trimmed at +# the extreme-concurrency end, per the dsv4-fp4-b300-vllm-mtp precedent: +# spec decode pays off at low/mid concurrency while acceptance dilutes in +# big batches, and the draft weights + draft KV shave headroom — tp2-ep2 is +# dropped entirely since its KV headroom was already thin without a draft. +minimaxm3-fp8-b300-vllm-mtp: + image: vllm/vllm-openai:nightly-4080263bb2c5d10deac17aaeb88e0823bc35bca9 + model: MiniMaxAI/MiniMax-M3-MXFP8 + model-prefix: minimaxm3 + runner: b300 + precision: fp8 + framework: vllm + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - { tp: 8, conc-start: 1, conc-end: 64, spec-decoding: mtp } + - { tp: 8, ep: 8, conc-start: 1, conc-end: 256, spec-decoding: mtp } + - { tp: 4, conc-start: 1, conc-end: 64, spec-decoding: mtp } + - { tp: 4, ep: 4, conc-start: 64, conc-end: 256, spec-decoding: mtp } + - { tp: 8, ep: 8, dp-attn: true, conc-start: 128, conc-end: 256, spec-decoding: mtp } + +# EAGLE3 speculative-decoding (spec-decoding: mtp) variant of +# minimaxm3-fp8-h200-vllm (PR #1731), pairing MiniMaxAI/MiniMax-M3-MXFP8 with +# the Inferact/MiniMax-M3-EAGLE3 draft head (3 speculative tokens, drafter +# pinned to FLASH_ATTN since the head is MHA and FlashInfer needs GQA/MQA at +# page size 128). Search space mirrors the non-MTP entry trimmed at the +# extreme-concurrency end, per the dsv4-fp4-b300-vllm-mtp / minimaxm3 b300-mtp +# precedent: spec decode pays off at low/mid concurrency while acceptance +# dilutes in big batches, and the draft weights + draft KV shave headroom. +minimaxm3-fp8-h200-vllm-mtp: + image: vllm/vllm-openai:minimax-m3 + model: MiniMaxAI/MiniMax-M3-MXFP8 + model-prefix: minimaxm3 + runner: h200 + precision: fp8 + framework: vllm + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - { tp: 4, conc-start: 1, conc-end: 32, spec-decoding: mtp } + - { tp: 8, conc-start: 1, conc-end: 128, spec-decoding: mtp } + - { tp: 8, ep: 8, conc-start: 256, conc-end: 256, spec-decoding: mtp } + - { tp: 8, ep: 8, dp-attn: true, conc-start: 256, conc-end: 256, spec-decoding: mtp } + +# EAGLE3 speculative-decoding (spec-decoding: mtp) variant of +# minimaxm3-fp8-h100-vllm (PR #1731). Same TP8-only layout (H100 80 GB has no +# room below TP8 once ~56 GB of weights land per GPU) plus the +# Inferact/MiniMax-M3-EAGLE3 draft head (3 speculative tokens, FLASH_ATTN +# drafter). DEP stays omitted as on the non-MTP entry — KV-cache init already +# failed at high conc, and the draft head + draft KV only tighten it further. +minimaxm3-fp8-h100-vllm-mtp: + image: vllm/vllm-openai:minimax-m3 + model: MiniMaxAI/MiniMax-M3-MXFP8 + model-prefix: minimaxm3 + runner: h100 + precision: fp8 + framework: vllm + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - { tp: 8, conc-start: 1, conc-end: 64, spec-decoding: mtp } + - { tp: 8, ep: 8, conc-start: 128, conc-end: 256, spec-decoding: mtp } diff --git a/configs/deprecated/nvidia-qwen3.5-bf16-master.yaml b/configs/deprecated/nvidia-qwen3.5-bf16-master.yaml new file mode 100644 index 0000000000..29121e2e48 --- /dev/null +++ b/configs/deprecated/nvidia-qwen3.5-bf16-master.yaml @@ -0,0 +1,68 @@ +# Deprecated Qwen3.5-397B-A17B bf16 entries archived from nvidia-master.yaml. +# All bf16 recipes, in every scenario, were deprecated after Monday, +# August 3, 2026 (see the Deprecation Notice in MODELS.md). fp8 and fp4 +# recipes remain active. +# Removed from the active master config so sweep generation no longer selects them. + + +qwen3.5-bf16-b200-sglang: + image: lmsysorg/sglang:v0.5.14-cu130 + model: Qwen/Qwen3.5-397B-A17B + model-prefix: qwen3.5 + runner: b200 + precision: bf16 + framework: sglang + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - { tp: 8, ep: 1, conc-start: 4, conc-end: 64 } + +qwen3.5-bf16-b200-sglang-mtp: + image: lmsysorg/sglang:v0.5.14-cu130 + model: Qwen/Qwen3.5-397B-A17B + model-prefix: qwen3.5 + runner: b200 + precision: bf16 + framework: sglang + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - { tp: 8, ep: 1, conc-start: 4, conc-end: 64, spec-decoding: mtp } + +qwen3.5-bf16-b300-sglang: + image: lmsysorg/sglang:v0.5.12-cu130 + model: Qwen/Qwen3.5-397B-A17B + model-prefix: qwen3.5 + runner: b300 + precision: bf16 + framework: sglang + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - { tp: 8, ep: 1, conc-start: 4, conc-end: 64 } + - { tp: 4, ep: 1, conc-start: 4, conc-end: 64 } + +qwen3.5-bf16-b300-sglang-mtp: + image: lmsysorg/sglang:v0.5.12-cu130 + model: Qwen/Qwen3.5-397B-A17B + model-prefix: qwen3.5 + runner: b300 + precision: bf16 + framework: sglang + multinode: false + scenarios: + fixed-seq-len: + - isl: 8192 + osl: 1024 + search-space: + - { tp: 8, ep: 1, conc-start: 4, conc-end: 64, spec-decoding: mtp } + - { tp: 4, ep: 1, conc-start: 4, conc-end: 64, spec-decoding: mtp } diff --git a/configs/nvidia-master.yaml b/configs/nvidia-master.yaml index d6ea947df5..51a650397b 100644 --- a/configs/nvidia-master.yaml +++ b/configs/nvidia-master.yaml @@ -1190,36 +1190,6 @@ dsv4-fp4-b300-sglang-mtp: - { tp: 8, ep: 1, conc-start: 1, conc-end: 8, spec-decoding: mtp } - { tp: 4, ep: 1, conc-start: 4, conc-end: 32, spec-decoding: mtp } -qwen3.5-bf16-b200-sglang: - image: lmsysorg/sglang:v0.5.14-cu130 - model: Qwen/Qwen3.5-397B-A17B - model-prefix: qwen3.5 - runner: b200 - precision: bf16 - framework: sglang - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - { tp: 8, ep: 1, conc-start: 4, conc-end: 64 } - -qwen3.5-bf16-b200-sglang-mtp: - image: lmsysorg/sglang:v0.5.14-cu130 - model: Qwen/Qwen3.5-397B-A17B - model-prefix: qwen3.5 - runner: b200 - precision: bf16 - framework: sglang - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - { tp: 8, ep: 1, conc-start: 4, conc-end: 64, spec-decoding: mtp } - qwen3.5-fp8-b200-sglang: image: lmsysorg/sglang:v0.5.14-cu130 model: Qwen/Qwen3.5-397B-A17B-FP8 @@ -1395,38 +1365,6 @@ qwen3.5-fp4-b300-sglang-mtp: - { tp: 4, ep: 1, conc-start: 4, conc-end: 128, spec-decoding: mtp } - { tp: 2, ep: 2, conc-start: 4, conc-end: 128, spec-decoding: mtp } -qwen3.5-bf16-b300-sglang: - image: lmsysorg/sglang:v0.5.12-cu130 - model: Qwen/Qwen3.5-397B-A17B - model-prefix: qwen3.5 - runner: b300 - precision: bf16 - framework: sglang - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - { tp: 8, ep: 1, conc-start: 4, conc-end: 64 } - - { tp: 4, ep: 1, conc-start: 4, conc-end: 64 } - -qwen3.5-bf16-b300-sglang-mtp: - image: lmsysorg/sglang:v0.5.12-cu130 - model: Qwen/Qwen3.5-397B-A17B - model-prefix: qwen3.5 - runner: b300 - precision: bf16 - framework: sglang - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - { tp: 8, ep: 1, conc-start: 4, conc-end: 64, spec-decoding: mtp } - - { tp: 4, ep: 1, conc-start: 4, conc-end: 64, spec-decoding: mtp } - kimik2.5-int4-b200-vllm: image: vllm/vllm-openai:v0.25.0 model: moonshotai/Kimi-K2.5 @@ -1442,21 +1380,6 @@ kimik2.5-int4-b200-vllm: search-space: - { tp: 8, conc-start: 4, conc-end: 64 } -kimik2.5-int4-b200-vllm-agentic: - image: vllm/vllm-openai:v0.22.0 - model: moonshotai/Kimi-K2.5 - model-prefix: kimik2.5 - runner: cluster:b200-dgxc - precision: int4 - framework: vllm - multinode: false - scenarios: - agentic-coding: - - dram-utilization: 0.80 - search-space: - - { tp: 8, kv-offloading: none, conc-list: [1, 2, 4, 8, 16, 32] } - - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, conc-list: [32, 64, 96, 128] } - kimik2.5-int4-b300-vllm: image: vllm/vllm-openai:v0.25.0 model: moonshotai/Kimi-K2.5 @@ -1488,21 +1411,6 @@ kimik2.5-int4-h200-vllm: search-space: - { tp: 8, conc-start: 4, conc-end: 64 } -kimik2.5-int4-h200-vllm-agentic: - image: vllm/vllm-openai:v0.22.0 - model: moonshotai/Kimi-K2.5 - model-prefix: kimik2.5 - runner: cluster:h200-dgxc - precision: int4 - framework: vllm - multinode: false - scenarios: - agentic-coding: - - dram-utilization: 0.80 - search-space: - - { tp: 8, kv-offloading: none, conc-list: [1, 2, 3, 4, 5, 6, 7] } - - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, conc-list: [6, 7, 8, 9, 10, 11, 12, 13, 14] } - # NOTE: At the time of submission, https://docs.vllm.ai/projects/recipes/en/latest/moonshotai/Kimi-K2.5.html # does not have a B300-specific recipe, so this config reuses the existing # Kimi-K2.5 FP4 B200 vLLM recipe as-is until B300-specific tuning is available. @@ -1576,44 +1484,6 @@ dsr1-fp8-b300-sglang-mtp: search-space: - { tp: 8, ep: 1, conc-start: 1, conc-end: 512, spec-decoding: mtp } -kimik2.5-fp4-b300-vllm-agentic: - # v0.20.2 (cu129) lacks the flashinfer kernels for B300's reported SM - # (sm_12x); workers hit "Only SM 10.x and 11.x are supported" in the - # trtllm_fp4_block_scale_moe path. v0.20.0-cu130 is the Blackwell-targeted - # build that has the full sm_10x/sm_11x/sm_12x kernel set and is what the - # INT4 B300 sister already uses successfully. - image: vllm/vllm-openai:v0.22.0 - model: nvidia/Kimi-K2.5-NVFP4 - model-prefix: kimik2.5 - runner: cluster:b300-nv - precision: fp4 - framework: vllm - multinode: false - scenarios: - agentic-coding: - - dram-utilization: 0.80 - search-space: - - { tp: 8, ep: 1, kv-offloading: none, conc-list: [1, 2, 4, 8, 16, 32, 40, 48, 56, 64] } - - { tp: 8, ep: 1, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, conc-list: [1, 2, 4, 8, 16, 32, 40, 48, 56, 64] } - -kimik2.5-fp4-b300-vllm-agentic-mtp: - image: vllm/vllm-openai:nightly-94c0ef300180f8fd1071d9cbe7270a8348155f94 - model: Kimi-K2.6-NVFP4 - model-prefix: kimik2.5 - runner: cluster:b300-nv - precision: fp4 - framework: vllm - multinode: false - scenarios: - agentic-coding: - - dram-utilization: 0.80 - search-space: - - { tp: 8, ep: 1, spec-decoding: mtp, kv-offloading: none, conc-list: [1] } - - { tp: 4, ep: 1, spec-decoding: mtp, kv-offloading: none, conc-list: [2, 4, 8] } - - { tp: 4, ep: 1, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: { name: native }, conc-list: [8, 16, 32] } - - { tp: 4, ep: 1, spec-decoding: mtp, dcp-size: 4, kv-offloading: none, conc-list: [32, 64, 80, 96, 112, 128] } - - { tp: 4, ep: 1, spec-decoding: mtp, dcp-size: 4, kv-offloading: dram, kv-offload-backend: { name: native }, conc-list: [64, 80, 96, 112, 128, 144, 160] } - kimik3-fp4-b300-vllm-agentic: # Day-zero Kimi-K3 recipe. `vllm/vllm-openai:kimi-k3` is the pre-release # Kimi-K3 build (vLLM 0.1.dev19262+gb6bbf29dd, pushed 2026-07-27); K3 ships as @@ -2500,22 +2370,6 @@ dsr1-fp8-h100-dynamo-trt: ep: 16 dp-attn: true -minimaxm3-fp8-h100-vllm: - image: vllm/vllm-openai:minimax-m3 - model: MiniMaxAI/MiniMax-M3-MXFP8 - model-prefix: minimaxm3 - runner: h100 - precision: fp8 - framework: vllm - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - { tp: 8, conc-start: 1, conc-end: 64 } - - { tp: 8, ep: 8, conc-start: 1, conc-end: 256 } - dsr1-fp8-h100-dynamo-sglang: image: lmsysorg/sglang:v0.5.8-cu130 model: deepseek-ai/DeepSeek-R1-0528 @@ -2591,25 +2445,6 @@ dsr1-fp8-h100-dynamo-sglang: ep: 16 dp-attn: true -minimaxm3-fp8-h200-vllm: - image: vllm/vllm-openai:minimax-m3 - model: MiniMaxAI/MiniMax-M3-MXFP8 - model-prefix: minimaxm3 - runner: h200 - precision: fp8 - framework: vllm - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - { tp: 4, conc-start: 1, conc-end: 32 } - - { tp: 4, ep: 4, conc-start: 1, conc-end: 256 } - - { tp: 8, conc-start: 1, conc-end: 128 } - - { tp: 8, ep: 8, conc-start: 1, conc-end: 256 } - - { tp: 8, ep: 8, dp-attn: true, conc-start: 256, conc-end: 512 } - dsr1-fp4-gb200-dynamo-trt: image: nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:0.8.1.post2 model: nvidia/DeepSeek-R1-0528-NVFP4-v2 @@ -7300,23 +7135,6 @@ qwen3.5-fp4-b300-sglang-agentic-mtp: - { tp: 2, ep: 2, spec-decoding: mtp, kv-offloading: none, conc-list: [1, 4, 8, 12, 16, 20, 24, 28, 32] } - { tp: 2, ep: 2, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [32, 34, 36, 38, 40, 44, 48, 52, 56] } -kimik2.5-fp4-b200-vllm-agentic-lmcache: - image: vllm/vllm-openai:v0.22.0 - model: nvidia/Kimi-K2.5-NVFP4 - model-prefix: kimik2.5 - runner: cluster:b200-dgxc - precision: fp4 - framework: vllm - multinode: false - scenarios: - agentic-coding: - - dram-utilization: 0.80 - search-space: - - { tp: 8, ep: 1, kv-offloading: none, conc-list: [1, 2, 4, 8, 16, 24] } - - { tp: 8, ep: 1, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "0.5.1" }, conc-list: [16, 24, 32, 36] } - - { tp: 4, ep: 1, kv-offloading: none, conc-list: [8, 12, 14, 16, 18, 20] } - - { tp: 4, ep: 1, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "0.5.1" }, conc-list: [12, 14, 16, 18, 20, 22, 24, 32] } - # CONC range conservative for H100's 80 GB HBM3 under the long-ISL with- # subagents corpus. hicache arm capped at conc 16 since high-conc + hicache # tends to flake on first runs and conc 16 covers the cliff. The bench script @@ -7402,1072 +7220,34 @@ qwen3.5-fp8-h100-sglang-agentic: - { tp: 8, ep: 8, kv-offloading: none, conc-list: [1, 2, 4, 8, 12, 14, 16] } - { tp: 8, ep: 8, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [12, 14, 16, 20, 24, 28, 32, 42] } -minimaxm3-fp4-b200-dynamo-vllm: - image: vllm/vllm-openai:nightly-5e35a6f4f9bbc217c599692157ca985c894373f7 - model: nvidia/MiniMax-M3-NVFP4 - model-prefix: minimaxm3 - runner: b200-multinode +qwen3.5-fp4-b200-trt: + image: nvcr.io#nvidia/tensorrt-llm/release:1.3.0rc18 + model: nvidia/Qwen3.5-397B-A17B-NVFP4 + model-prefix: qwen3.5 + runner: b200 precision: fp4 - framework: dynamo-vllm - router: { name: dynamo-router, version: "1.3.0.dev20260710" } - kv-p2p-transfer: nixl - multinode: true - disagg: true + framework: trt + multinode: false scenarios: fixed-seq-len: - isl: 8192 osl: 1024 search-space: - - conc-list: [1, 4, 8, 16] - prefill: - num-worker: 1 - tp: 2 - ep: 2 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3/b200-fp4/8k1k/1p1d-dep2-tp4-c1-c16-8k1k.yaml" - decode: - num-worker: 1 - tp: 4 - ep: 1 - dp-attn: false - - conc-list: [64] - prefill: - num-worker: 1 - tp: 2 - ep: 2 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3/b200-fp4/8k1k/1p2d-dep2-tp4-c64-8k1k.yaml" - decode: - num-worker: 2 - tp: 4 - ep: 1 - dp-attn: false - - conc-list: [128] - prefill: - num-worker: 2 - tp: 2 - ep: 2 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3/b200-fp4/8k1k/2p2d-dep2-tp4-c128-8k1k.yaml" - decode: - num-worker: 2 - tp: 4 - ep: 1 - dp-attn: false - - conc-list: [256] - prefill: - num-worker: 3 - tp: 2 - ep: 2 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3/b200-fp4/8k1k/3p2d-dep2-tp4-c256-8k1k.yaml" - decode: - num-worker: 2 - tp: 4 - ep: 1 - dp-attn: false - - conc-list: [512] - prefill: - num-worker: 3 - tp: 2 - ep: 2 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3/b200-fp4/8k1k/3p2d-dep2-tp4-c512-8k1k.yaml" - decode: - num-worker: 2 - tp: 4 - ep: 1 - dp-attn: false - - conc-list: [1024] - prefill: - num-worker: 4 - tp: 2 - ep: 2 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3/b200-fp4/8k1k/4p2d-dep2-tp4-c1024-8k1k.yaml" - decode: - num-worker: 2 - tp: 4 - ep: 1 - dp-attn: false + - { tp: 2, ep: 1, conc-list: [4, 16] } + - { tp: 4, ep: 1, conc-list: [4] } + - { tp: 2, ep: 2, conc-list: [8, 32] } + - { tp: 8, ep: 8, conc-list: [4] } + - { tp: 4, ep: 4, dp-attn: true, conc-list: [1024] } + - { tp: 8, ep: 8, dp-attn: true, conc-list: [256, 512, 1024] } -# MiniMax-M3 NVFP4 disagg sweep on the same B300 topology matrix as the MXFP8 -# baseline above. The image includes vLLM PR #46380, so no runtime patch is -# needed. -minimaxm3-fp8-b300-dynamo-vllm: - image: vllm/vllm-openai:nightly-4080263bb2c5d10deac17aaeb88e0823bc35bca9 - model: MiniMaxAI/MiniMax-M3-MXFP8 - model-prefix: minimaxm3 - runner: b300 - precision: fp8 - framework: dynamo-vllm - router: { name: dynamo-router, version: "1.3.0.dev20260614" } - kv-p2p-transfer: nixl - multinode: true - disagg: true - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - conc-list: [256, 512] - prefill: - num-worker: 2 - tp: 2 - ep: 2 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3/b300-fp8/8k1k/2p2d-dep2-dep8-8k1k.yaml" - decode: - num-worker: 2 - tp: 8 - ep: 8 - dp-attn: true - - conc-list: [16] - prefill: - num-worker: 2 - tp: 2 - ep: 2 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3/b300-fp8/8k1k/2p2d-dep2-tep8-8k1k.yaml" - decode: - num-worker: 2 - tp: 8 - ep: 8 - dp-attn: false - - conc-list: [4096] - prefill: - num-worker: 4 - tp: 2 - ep: 2 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3/b300-fp8/8k1k/4p2d-dep2-dep8-8k1k.yaml" - decode: - num-worker: 2 - tp: 8 - ep: 8 - dp-attn: true - - conc-list: [1, 4, 8, 16] - prefill: - num-worker: 1 - tp: 2 - ep: 2 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3/b300-fp8/8k1k/1p1d-dep2-tp4-marlin-8k1k.yaml" - decode: - num-worker: 1 - tp: 4 - ep: 1 - dp-attn: false - - conc-list: [1024, 2048] - prefill: - num-worker: 3 - tp: 2 - ep: 2 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3/b300-fp8/8k1k/3p2d-dep2-dep8-8k1k.yaml" - decode: - num-worker: 2 - tp: 8 - ep: 8 - dp-attn: true - - conc-list: [16, 32, 64, 128] - prefill: - num-worker: 1 - tp: 2 - ep: 2 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3/b300-fp8/8k1k/1p4d-dep2-tep4-8k1k.yaml" - decode: - num-worker: 4 - tp: 4 - ep: 4 - dp-attn: false - - conc-list: [16] - prefill: - num-worker: 1 - tp: 2 - ep: 2 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3/b300-fp8/8k1k/1p2d-dep2-tep4-8k1k.yaml" - decode: - num-worker: 2 - tp: 4 - ep: 4 - dp-attn: false - - conc-list: [4] - prefill: - num-worker: 1 - tp: 2 - ep: 2 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3/b300-fp8/8k1k/1p4d-dep2-tep8-8k1k.yaml" - decode: - num-worker: 4 - tp: 8 - ep: 8 - dp-attn: false - -# MiniMax-M3 NVFP4 disagg sweep on the same B300 topology matrix as the MXFP8 -# baseline above. The image includes vLLM PR #46380, so no runtime patch is -# needed. -minimaxm3-fp4-b300-dynamo-vllm-8k1k-tp1: - image: vllm/vllm-openai:nightly-5e35a6f4f9bbc217c599692157ca985c894373f7 - model: nvidia/MiniMax-M3-NVFP4 - model-prefix: minimaxm3 - runner: b300 - precision: fp4 - framework: dynamo-vllm - router: { name: dynamo-router, version: "1.3.0.dev20260713" } - kv-p2p-transfer: nixl - multinode: true - disagg: true - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - conc-list: [8, 16, 24, 32, 48, 64] - prefill: - num-worker: 1 - tp: 1 - ep: 1 - dp-attn: false - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3/b300-fp4/8k1k/1p1d-tp1-tp4-8k1k.yaml" - decode: - num-worker: 1 - tp: 4 - ep: 1 - dp-attn: false - - conc-list: [1, 2, 4, 8, 16] - prefill: - num-worker: 1 - tp: 1 - ep: 1 - dp-attn: false - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3/b300-fp4/8k1k/1p1d-tp1-tp8-8k1k.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 1 - dp-attn: false - - conc-list: [128] - prefill: - num-worker: 2 - tp: 1 - ep: 1 - dp-attn: false - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3/b300-fp4/8k1k/2p1d-tp1-tp4-8k1k.yaml" - decode: - num-worker: 1 - tp: 4 - ep: 1 - dp-attn: false - - conc-list: [256] - prefill: - num-worker: 3 - tp: 1 - ep: 1 - dp-attn: false - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3/b300-fp4/8k1k/3p1d-tp1-tp4-8k1k.yaml" - decode: - num-worker: 1 - tp: 4 - ep: 1 - dp-attn: false - - conc-list: [512] - prefill: - num-worker: 5 - tp: 1 - ep: 1 - dp-attn: false - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3/b300-fp4/8k1k/5p2d-tp1-dep8-8k1k.yaml" - decode: - num-worker: 2 - tp: 8 - ep: 8 - dp-attn: true - - conc-list: [768, 1024] - prefill: - num-worker: 8 - tp: 1 - ep: 1 - dp-attn: false - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3/b300-fp4/8k1k/8p2d-tp1-dep8-8k1k.yaml" - decode: - num-worker: 2 - tp: 8 - ep: 8 - dp-attn: true - -# Compare the refreshed image on a one-node max-throughput topology: four -# TP1 prefills colocated with one TEP4 decode worker. -minimaxm3-fp4-b300-dynamo-vllm-8k1k-tp1-tep4: - image: vllm/vllm-openai:nightly-5e35a6f4f9bbc217c599692157ca985c894373f7 - model: nvidia/MiniMax-M3-NVFP4 - model-prefix: minimaxm3 - runner: b300 - precision: fp4 - framework: dynamo-vllm - router: { name: dynamo-router, version: "1.3.0.dev20260713" } - kv-p2p-transfer: nixl - multinode: true - disagg: true - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - conc-list: [4096] - prefill: - num-worker: 4 - tp: 1 - ep: 1 - dp-attn: false - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3/b300-fp4/8k1k/4p1d-tp1-tep4-8k1k.yaml" - decode: - num-worker: 1 - tp: 4 - ep: 4 - dp-attn: false - -# Preserve the previously proven max-throughput point exactly, while keeping -# it separate from the legacy mixed 1k1k/8k1k entry so 1k1k is not rerun. -minimaxm3-fp4-b300-dynamo-vllm-8k1k-legacy-max-tput: - image: vllm/vllm-openai:vllm-minimax-m3-perf-x86_64-13.0.1-8b00f41 - model: nvidia/MiniMax-M3-NVFP4 - model-prefix: minimaxm3 - runner: b300 - precision: fp4 - framework: dynamo-vllm - router: { name: dynamo-router, version: "1.3.0.dev20260614" } - kv-p2p-transfer: nixl - multinode: true - disagg: true - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - conc-list: [4096] - prefill: - num-worker: 4 - tp: 2 - ep: 2 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3/b300-fp4/8k1k/4p2d-dep2-tep4-8k1k.yaml" - decode: - num-worker: 2 - tp: 4 - ep: 4 - dp-attn: false - -minimaxm3-fp4-b200-dynamo-vllm-mtp: - image: vllm/vllm-openai:nightly-5e35a6f4f9bbc217c599692157ca985c894373f7 - model: nvidia/MiniMax-M3-NVFP4 - model-prefix: minimaxm3 - runner: b200-multinode - precision: fp4 - framework: dynamo-vllm - router: { name: dynamo-router, version: "1.3.0.dev20260710" } - kv-p2p-transfer: nixl - multinode: true - disagg: true - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - spec-decoding: "mtp" - conc-list: [12, 24, 48] - prefill: - num-worker: 1 - tp: 4 - ep: 1 - dp-attn: false - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3/b200-fp4/8k1k/mtp/1p6d-tp4-tp4-eagle3-8k1k.yaml" - decode: - num-worker: 6 - tp: 4 - ep: 1 - dp-attn: false - - spec-decoding: "mtp" - conc-list: [4, 8, 16, 32, 64] - prefill: - num-worker: 1 - tp: 4 - ep: 1 - dp-attn: false - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3/b200-fp4/8k1k/mtp/1p4d-tp4-tp4-eagle3-8k1k.yaml" - decode: - num-worker: 4 - tp: 4 - ep: 1 - dp-attn: false - - spec-decoding: "mtp" - conc-list: [32, 64, 96] - prefill: - num-worker: 1 - tp: 4 - ep: 1 - dp-attn: false - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3/b200-fp4/8k1k/mtp/1p2d-tp4-tp4-eagle3-8k1k.yaml" - decode: - num-worker: 2 - tp: 4 - ep: 1 - dp-attn: false - - spec-decoding: "mtp" - conc-list: [192, 256, 512] - prefill: - num-worker: 1 - tp: 4 - ep: 1 - dp-attn: false - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3/b200-fp4/8k1k/mtp/1p1d-tp4-tp4-eagle3-8k1k.yaml" - decode: - num-worker: 1 - tp: 4 - ep: 1 - dp-attn: false - - - spec-decoding: "mtp" - conc-list: [12, 24, 48] - prefill: - num-worker: 1 - tp: 4 - ep: 1 - dp-attn: false - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3/b200-fp4/8k1k/mtp/1p3d-tp4-tp4-eagle3-8k1k.yaml" - decode: - num-worker: 3 - tp: 4 - ep: 1 - dp-attn: false - - - - spec-decoding: "mtp" - conc-list: [256, 512] - prefill: - num-worker: 1 - tp: 4 - ep: 1 - dp-attn: false - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3/b200-fp4/8k1k/mtp/1p1d-tp4-dep4-eagle3-8k1k.yaml" - decode: - num-worker: 1 - tp: 4 - ep: 4 - dp-attn: true - -minimaxm3-fp4-b300-dynamo-vllm-mtp: - image: vllm/vllm-openai:nightly-5e35a6f4f9bbc217c599692157ca985c894373f7 - model: nvidia/MiniMax-M3-NVFP4 - model-prefix: minimaxm3 - runner: b300 - precision: fp4 - framework: dynamo-vllm - router: { name: dynamo-router, version: "1.3.0.dev20260710" } - kv-p2p-transfer: nixl - multinode: true - disagg: true - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - spec-decoding: "mtp" - conc-list: [24, 48] - prefill: - num-worker: 1 - tp: 2 - ep: 2 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3/b300-fp4/8k1k/mtp/1p6d-dep2-tp4-eagle3-8k1k.yaml" - decode: - num-worker: 6 - tp: 4 - ep: 1 - dp-attn: false - - spec-decoding: "mtp" - conc-list: [4, 8, 64] - prefill: - num-worker: 1 - tp: 2 - ep: 2 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3/b300-fp4/8k1k/mtp/1p4d-dep2-tp4-eagle3-8k1k.yaml" - decode: - num-worker: 4 - tp: 4 - ep: 1 - dp-attn: false - - spec-decoding: "mtp" - conc-list: [4, 32, 64] - prefill: - num-worker: 1 - tp: 2 - ep: 2 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3/b300-fp4/8k1k/mtp/1p2d-dep2-tp4-eagle3-8k1k.yaml" - decode: - num-worker: 2 - tp: 4 - ep: 1 - dp-attn: false - - spec-decoding: "mtp" - conc-list: [64] - prefill: - num-worker: 1 - tp: 2 - ep: 2 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3/b300-fp4/8k1k/mtp/1p1d-dep2-tp4-eagle3-8k1k.yaml" - decode: - num-worker: 1 - tp: 4 - ep: 1 - dp-attn: false - - - spec-decoding: "mtp" - conc-list: [192] - prefill: - num-worker: 2 - tp: 2 - ep: 2 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3/b300-fp4/8k1k/mtp/2p3d-dep2-tp4-eagle3-8k1k.yaml" - decode: - num-worker: 3 - tp: 4 - ep: 1 - dp-attn: false - -minimaxm3-fp4-b300-dynamo-vllm-mtp-legacy-dep4: - image: vllm/vllm-openai:nightly-5e35a6f4f9bbc217c599692157ca985c894373f7 - model: nvidia/MiniMax-M3-NVFP4 - model-prefix: minimaxm3 - runner: b300 - precision: fp4 - framework: dynamo-vllm - router: { name: dynamo-router, version: "1.3.0.dev20260710" } - kv-p2p-transfer: nixl - multinode: true - disagg: true - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - spec-decoding: "mtp" - conc-list: [256, 512] - prefill: - num-worker: 2 - tp: 2 - ep: 2 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3/b300-fp4/8k1k/mtp/2p1d-dep2-dep4-eagle3-8k1k.yaml" - decode: - num-worker: 1 - tp: 4 - ep: 4 - dp-attn: true -# MiniMax-M3 GB300 disagg sweep — refreshed recipe set (no Marlin variants). -# All prefill DEP2 (TP1 DP2 EP, 2 GPU/worker). Decode: DEP4, TEP8, DEP8, TEP4. -# 4 GPU/node (GB300 NVL72). kv-cache-dtype=fp8. srun_options mem=0 required. -minimaxm3-fp8-gb300-dynamo-vllm-mtp: - image: vllm/vllm-openai:nightly-5e35a6f4f9bbc217c599692157ca985c894373f7 - model: MiniMaxAI/MiniMax-M3-MXFP8 - model-prefix: minimaxm3 - runner: gb300-nv - precision: fp8 - framework: dynamo-vllm - router: { name: dynamo-router, version: "1.4.0.dev20260730" } - kv-p2p-transfer: nixl - multinode: true - disagg: true - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - spec-decoding: "mtp" - conc-list: [1] - prefill: - num-worker: 1 - tp: 2 - ep: 2 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3-gb300-fp8/8k1k/mtp/1p1d-dep2-tp4-eagle3-c1-8k1k.yaml" - decode: - num-worker: 1 - tp: 4 - ep: 1 - dp-attn: false - - spec-decoding: "mtp" - conc-list: [8] - prefill: - num-worker: 1 - tp: 2 - ep: 2 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3-gb300-fp8/8k1k/mtp/1p1d-dep2-tp4-eagle3-c8-8k1k.yaml" - decode: - num-worker: 1 - tp: 4 - ep: 1 - dp-attn: false - - spec-decoding: "mtp" - conc-list: [1] - prefill: - num-worker: 1 - tp: 2 - ep: 2 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3-gb300-fp8/8k1k/mtp/1p1d-dep2-tp8-eagle3-c1-8k1k.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 1 - dp-attn: false - - spec-decoding: "mtp" - conc-list: [4] - prefill: - num-worker: 1 - tp: 2 - ep: 2 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3-gb300-fp8/8k1k/mtp/1p1d-dep2-tp8-eagle3-c4-8k1k.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 1 - dp-attn: false - - spec-decoding: "mtp" - conc-list: [8] - prefill: - num-worker: 1 - tp: 2 - ep: 2 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3-gb300-fp8/8k1k/mtp/1p1d-dep2-tp8-eagle3-c8-8k1k.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 1 - dp-attn: false - - spec-decoding: "mtp" - conc-list: [64] - prefill: - num-worker: 1 - tp: 2 - ep: 2 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3-gb300-fp8/8k1k/mtp/1p1d-dep2-dep8-eagle3-c64-8k1k.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - - spec-decoding: "mtp" - conc-list: [512] - prefill: - num-worker: 2 - tp: 2 - ep: 2 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3-gb300-fp8/8k1k/mtp/2p1d-dep2-dep8-eagle3-c512-8k1k.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - - spec-decoding: "mtp" - conc-list: [256] - prefill: - num-worker: 3 - tp: 2 - ep: 2 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3-gb300-fp8/8k1k/mtp/3p1d-dep2-dep8-eagle3-c256-8k1k.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - - spec-decoding: "mtp" - conc-list: [1024] - prefill: - num-worker: 4 - tp: 2 - ep: 2 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3-gb300-fp8/8k1k/mtp/4p1d-dep2-dep8-eagle3-c1024-8k1k.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - - spec-decoding: "mtp" - conc-list: [2048] - prefill: - num-worker: 4 - tp: 2 - ep: 2 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3-gb300-fp8/8k1k/mtp/4p1d-dep2-dep8-eagle3-c2048-8k1k.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - - spec-decoding: "mtp" - conc-list: [2048] - prefill: - num-worker: 6 - tp: 2 - ep: 2 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3-gb300-fp8/8k1k/mtp/6p1d-dep2-dep8-eagle3-c2048-8k1k.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - - -minimaxm3-fp8-gb300-dynamo-vllm: - image: vllm/vllm-openai:nightly-4080263bb2c5d10deac17aaeb88e0823bc35bca9 - model: MiniMaxAI/MiniMax-M3-MXFP8 - model-prefix: minimaxm3 - runner: gb300-nv - precision: fp8 - framework: dynamo-vllm - router: { name: dynamo-router, version: "1.3.0.dev20260614" } - kv-p2p-transfer: nixl - multinode: true - disagg: true - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - # 1p1d DEP2+DEP8, 3n: conc 256 - - conc-list: [256] - prefill: - num-worker: 1 - tp: 2 - ep: 2 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3-gb300-fp8/8k1k/1p1d-dep2-dep8-8k1k.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - - # 1p1d DEP2+TEP8, 3n: conc 128 - - conc-list: [128] - prefill: - num-worker: 1 - tp: 2 - ep: 2 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3-gb300-fp8/8k1k/1p1d-dep2-tep8-8k1k.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: false - - # 1p2d DEP2+TEP8, 5n: conc 32,64,128 - - conc-list: [32, 64, 128] - prefill: - num-worker: 1 - tp: 2 - ep: 2 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3-gb300-fp8/8k1k/1p2d-dep2-tep8-8k1k.yaml" - decode: - num-worker: 2 - tp: 8 - ep: 8 - dp-attn: false - - # 2p1d DEP2+DEP8, 3n: conc 512 - - conc-list: [512] - prefill: - num-worker: 2 - tp: 2 - ep: 2 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3-gb300-fp8/8k1k/2p1d-dep2-dep8-8k1k.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - - # 2p2d DEP2+TEP8, 5n: conc 16 - - conc-list: [16] - prefill: - num-worker: 2 - tp: 2 - ep: 2 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3-gb300-fp8/8k1k/2p2d-dep2-tep8-8k1k.yaml" - decode: - num-worker: 2 - tp: 8 - ep: 8 - dp-attn: false - - # 2p4d DEP2+TEP4, 5n: conc 4 - - conc-list: [4] - prefill: - num-worker: 2 - tp: 2 - ep: 2 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3-gb300-fp8/8k1k/2p4d-dep2-tep4-8k1k.yaml" - decode: - num-worker: 4 - tp: 4 - ep: 4 - dp-attn: false - - # 3p1d DEP2+DEP8, 4n: conc 1024 - - conc-list: [1024] - prefill: - num-worker: 3 - tp: 2 - ep: 2 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3-gb300-fp8/8k1k/3p1d-dep2-dep8-8k1k.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - - # 3p1d DEP2+DEP16, 6n: conc 512 - - conc-list: [512] - prefill: - num-worker: 3 - tp: 2 - ep: 2 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3-gb300-fp8/8k1k/3p1d-dep2-dep16-8k1k.yaml" - decode: - num-worker: 1 - tp: 16 - ep: 16 - dp-attn: true - - # 6p1d DEP2+DEP8, 5n: conc 2048 - - conc-list: [2048] - prefill: - num-worker: 6 - tp: 2 - ep: 2 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3-gb300-fp8/8k1k/6p1d-dep2-dep8-8k1k.yaml" - decode: - num-worker: 1 - tp: 8 - ep: 8 - dp-attn: true - -qwen3.5-fp4-b200-trt: - image: nvcr.io#nvidia/tensorrt-llm/release:1.3.0rc18 - model: nvidia/Qwen3.5-397B-A17B-NVFP4 - model-prefix: qwen3.5 - runner: b200 - precision: fp4 - framework: trt - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - { tp: 2, ep: 1, conc-list: [4, 16] } - - { tp: 4, ep: 1, conc-list: [4] } - - { tp: 2, ep: 2, conc-list: [8, 32] } - - { tp: 8, ep: 8, conc-list: [4] } - - { tp: 4, ep: 4, dp-attn: true, conc-list: [1024] } - - { tp: 8, ep: 8, dp-attn: true, conc-list: [256, 512, 1024] } - -# MiniMax-M3 GB200 disagg sweep — adapted from NV B300 PR #1863. -# All prefill DEP4 (TP1 DP4 EP, 4 GPU/worker). Decode: Marlin, TEP8, DEP8, TEP4. -# 4 GPU/node (GB200 NVL72). FLASHINFER attention with FP8 KV cache. -minimaxm3-fp8-gb200-dynamo-vllm: - image: vllm/vllm-openai:nightly-4080263bb2c5d10deac17aaeb88e0823bc35bca9 - model: MiniMaxAI/MiniMax-M3-MXFP8 - model-prefix: minimaxm3 - runner: gb200 - precision: fp8 - framework: dynamo-vllm - router: { name: dynamo-router, version: "1.3.0.dev20260614" } - kv-p2p-transfer: nixl - multinode: true - disagg: true - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - # 1p2d DEP4+DEP8, 5n: conc 512 - - conc-list: [512] - prefill: - num-worker: 1 - tp: 4 - ep: 4 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3-gb200-fp8/8k1k/1p1d-dep4-dep16-8k1k.yaml" - decode: - num-worker: 1 - tp: 16 - ep: 16 - dp-attn: true - - # 1p2d DEP4+TEP4, 3n: conc 4 - - conc-list: [4] - prefill: - num-worker: 1 - tp: 4 - ep: 4 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3-gb200-fp8/8k1k/1p2d-dep4-tep4-8k1k.yaml" - decode: - num-worker: 2 - tp: 4 - ep: 4 - dp-attn: false - - # 1p2d DEP4+TEP8, 5n: conc 4,16,32,64,128 - - conc-list: [4, 16, 32, 64, 128] - prefill: - num-worker: 1 - tp: 4 - ep: 4 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3-gb200-fp8/8k1k/1p2d-dep4-tep8-8k1k.yaml" - decode: - num-worker: 2 - tp: 8 - ep: 8 - dp-attn: false - - # 2p2d DEP4+DEP8, 6n: conc 512,1024 - - conc-list: [512, 1024] - prefill: - num-worker: 2 - tp: 4 - ep: 4 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3-gb200-fp8/8k1k/2p1d-dep4-dep16-8k1k.yaml" - decode: - num-worker: 1 - tp: 16 - ep: 16 - dp-attn: true - - # 3p2d DEP4+DEP8, 7n: conc 4096 - - conc-list: [4096] - prefill: - num-worker: 3 - tp: 4 - ep: 4 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3-gb200-fp8/8k1k/3p1d-dep4-dep16-8k1k.yaml" - decode: - num-worker: 1 - tp: 16 - ep: 16 - dp-attn: true - - # 5p2d DEP4+DEP8, 9n: conc 4096 - - conc-list: [4096] - prefill: - num-worker: 5 - tp: 4 - ep: 4 - dp-attn: true - additional-settings: - - "CONFIG_FILE=recipes/vllm/minimax-m3-gb200-fp8/8k1k/5p1d-dep4-dep16-8k1k.yaml" - decode: - num-worker: 1 - tp: 16 - ep: 16 - dp-attn: true - -qwen3.5-fp4-b200-trt-mtp: - image: nvcr.io#nvidia/tensorrt-llm/release:1.3.0rc18 - model: nvidia/Qwen3.5-397B-A17B-NVFP4 - model-prefix: qwen3.5 - runner: b200 - precision: fp4 - framework: trt - multinode: false +qwen3.5-fp4-b200-trt-mtp: + image: nvcr.io#nvidia/tensorrt-llm/release:1.3.0rc18 + model: nvidia/Qwen3.5-397B-A17B-NVFP4 + model-prefix: qwen3.5 + runner: b200 + precision: fp4 + framework: trt + multinode: false scenarios: fixed-seq-len: - isl: 8192 @@ -8479,269 +7259,6 @@ qwen3.5-fp4-b200-trt-mtp: - { tp: 8, ep: 8, spec-decoding: "mtp", conc-list: [4] } - { tp: 8, ep: 8, dp-attn: true, spec-decoding: "mtp", conc-list: [128, 256, 1024] } -# MiniMax-M3 day-zero (https://recipes.vllm.ai/MiniMaxAI/MiniMax-M3). -# 427B total / 26B active MoE with MSA sparse attention; MXFP8 checkpoint -# (MiniMaxAI/MiniMax-M3-MXFP8, ~444 GB) quantized by NVIDIA — native MX tensor -# cores on Blackwell. M3 support has not shipped in a stable vLLM release; -# the dedicated vllm/vllm-openai:minimax-m3-0618-x86_64-cu130 image is built -# from the m3_release branch (vllm-project/vllm#45381). --block-size 128 is mandatory (MSA -# sparse/index cache alignment). Weights are NOT SRE-staged; b300 falls back -# to writable /data/models (see launch_b300-nv.sh MODEL_PATH split). -minimaxm3-fp8-b300-vllm: - image: vllm/vllm-openai:nightly-4080263bb2c5d10deac17aaeb88e0823bc35bca9 - model: MiniMaxAI/MiniMax-M3-MXFP8 - model-prefix: minimaxm3 - runner: b300 - precision: fp8 - framework: vllm - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - { tp: 8, conc-start: 1, conc-end: 64 } - - { tp: 8, ep: 8, conc-start: 1, conc-end: 512 } - - { tp: 4, conc-start: 1, conc-end: 128 } - - { tp: 4, ep: 4, conc-start: 64, conc-end: 256 } - - { tp: 4, ep: 4, dp-attn: true, conc-start: 64, conc-end: 128 } - - { tp: 8, ep: 8, dp-attn: true, conc-start: 128, conc-end: 512 } - -# MiniMax-M3 NVFP4 (nvidia/MiniMax-M3-NVFP4) B300 single-node vLLM — FP4 variant -# of minimaxm3-fp8-b300-vllm. MiniMax-M3 modelopt NVFP4 support (vllm-project/vllm -# PR #46380) is baked into the perf container image, so no runtime patch is -# needed. --block-size 128 is mandatory (MSA sparse/index cache); -# weights are pre-staged read-only at /scratch/models/MiniMax-M3-NVFP4 (added to -# the STAGED_MODELS allow-list in launch_b300-nv.sh). -minimaxm3-fp4-b300-vllm: - image: vllm/vllm-openai:nightly-5e35a6f4f9bbc217c599692157ca985c894373f7 - model: nvidia/MiniMax-M3-NVFP4 - model-prefix: minimaxm3 - runner: b300 - precision: fp4 - framework: vllm - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - { tp: 8, conc-start: 1, conc-end: 2 } - - { tp: 4, conc-start: 1, conc-end: 2 } - - { tp: 2, conc-start: 4, conc-end: 256 } - - { tp: 2, ep: 2, dp-attn: true, conc-start: 512, conc-end: 512 } - -# EAGLE3 speculative-decoding (spec-decoding: mtp) variant of MiniMax-M3 NVFP4 -# (nvidia/MiniMax-M3-NVFP4) B300 single-node vLLM, pairing the target with the -# Inferact/MiniMax-M3-EAGLE3 draft head (3 speculative tokens). MiniMax-M3 -# modelopt NVFP4 support (vllm-project/vllm PR #46380) is baked into the perf -# container image, so no runtime patch is needed; prompts are routed through the -# chat template. Target weights are pre-staged read-only at -# /scratch/models/MiniMax-M3-NVFP4 (added to the STAGED_MODELS allow-list in -# launch_b300-nv.sh); the EAGLE3 draft is downloaded to the writable models dir. -minimaxm3-fp4-b300-vllm-mtp: - image: vllm/vllm-openai:nightly-4080263bb2c5d10deac17aaeb88e0823bc35bca9 - model: nvidia/MiniMax-M3-NVFP4 - model-prefix: minimaxm3 - runner: b300 - precision: fp4 - framework: vllm - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - { tp: 2, conc-list: [1, 2, 4, 8, 16, 32, 128, 256], spec-decoding: mtp } - - { tp: 2, ep: 2, conc-list: [1, 2, 4, 8, 32, 64, 128], spec-decoding: mtp } - - { tp: 2, ep: 2, dp-attn: true, conc-list: [128], spec-decoding: mtp } - - { tp: 4, conc-list: [1, 2, 4, 8], spec-decoding: mtp } - - { tp: 4, ep: 4, conc-list: [1, 2, 4, 8], spec-decoding: mtp } - - { tp: 8, conc-list: [1, 2, 4, 8], spec-decoding: mtp } - -# MiniMax-M3 day-zero (https://recipes.vllm.ai/MiniMaxAI/MiniMax-M3). -# 427B total / 26B active MoE with MSA sparse attention; MXFP8 checkpoint -# (MiniMaxAI/MiniMax-M3-MXFP8, ~444 GB) quantized by NVIDIA — native MX tensor -# cores on Blackwell. M3 support has not shipped in a stable vLLM release; -# the dedicated vllm/vllm-openai:minimax-m3-0618-x86_64-cu130 image is built -# from the m3_release branch (vllm-project/vllm#45381). --block-size 128 is mandatory (MSA -# sparse/index cache alignment). Weights are NOT SRE-staged: b200-dgxc reads -# /lustre/fsw/gharunners/models/MiniMax-M3-MXFP8 (pre-downloaded, see -# launch_b200-dgxc.sh). -minimaxm3-fp8-b200-vllm: - image: vllm/vllm-openai:nightly-4080263bb2c5d10deac17aaeb88e0823bc35bca9 - model: MiniMaxAI/MiniMax-M3-MXFP8 - model-prefix: minimaxm3 - runner: b200-dgxc - precision: fp8 - framework: vllm - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - { tp: 8, conc-start: 1, conc-end: 64 } - - { tp: 8, ep: 8, conc-start: 1, conc-end: 256 } - - { tp: 4, conc-start: 1, conc-end: 64 } - - { tp: 4, ep: 4, conc-start: 64, conc-end: 256 } - - { tp: 4, ep: 4, dp-attn: true, conc-start: 64, conc-end: 128 } - - { tp: 8, ep: 8, dp-attn: true, conc-start: 128, conc-end: 256 } - -# MiniMax-M3 NVFP4 (nvidia/MiniMax-M3-NVFP4) B200 single-node vLLM — FP4 variant -# of minimaxm3-fp8-b200-vllm, running on the b200-dgxc cluster. MiniMax-M3 -# modelopt NVFP4 support (vllm-project/vllm PR #46380) is baked into the perf -# container image, so no runtime patch is needed. --block-size 128 is mandatory -# (MSA sparse/index cache); weights are pre-staged at /scratch/fsw/models/MiniMax-M3-NVFP4 -# (launch_b200-dgxc.sh resolves MODEL_PATH for minimaxm3-fp4). -minimaxm3-fp4-b200-vllm: - image: vllm/vllm-openai:nightly-5e35a6f4f9bbc217c599692157ca985c894373f7 - model: nvidia/MiniMax-M3-NVFP4 - model-prefix: minimaxm3 - runner: b200-dgxc - precision: fp4 - framework: vllm - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - { tp: 8, conc-start: 1, conc-end: 4 } - - { tp: 4, conc-start: 1, conc-end: 4 } - - { tp: 2, conc-start: 1, conc-end: 128 } - - { tp: 4, conc-start: 256, conc-end: 1024 } - - { tp: 4, ep: 4, dp-attn: true, conc-start: 256, conc-end: 1024 } - -# EAGLE3 speculative-decoding (spec-decoding: mtp) variant of -# minimaxm3-fp8-b200-vllm, pairing MiniMaxAI/MiniMax-M3-MXFP8 with the -# Inferact/MiniMax-M3-EAGLE3 draft head (3 speculative tokens, drafter pinned -# to TRITON_ATTN). Search space mirrors the non-MTP entry trimmed at the -# extreme-concurrency end, identical to the minimaxm3-fp8-b300-vllm-mtp -# precedent: spec decode pays off at low/mid concurrency while acceptance -# dilutes in big batches, and the draft weights + draft KV shave headroom. -minimaxm3-fp8-b200-vllm-mtp: - image: vllm/vllm-openai:nightly-4080263bb2c5d10deac17aaeb88e0823bc35bca9 - model: MiniMaxAI/MiniMax-M3-MXFP8 - model-prefix: minimaxm3 - runner: b200-dgxc - precision: fp8 - framework: vllm - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - { tp: 8, conc-start: 1, conc-end: 64, spec-decoding: mtp } - - { tp: 8, ep: 8, conc-start: 1, conc-end: 256, spec-decoding: mtp } - - { tp: 4, conc-start: 1, conc-end: 64, spec-decoding: mtp } - - { tp: 4, ep: 4, conc-start: 64, conc-end: 256, spec-decoding: mtp } - - { tp: 8, ep: 8, dp-attn: true, conc-start: 128, conc-end: 256, spec-decoding: mtp } - -# EAGLE3 speculative-decoding (spec-decoding: mtp) variant of MiniMax-M3 NVFP4 -# (nvidia/MiniMax-M3-NVFP4) B200 single-node vLLM, pairing the target with the -# Inferact/MiniMax-M3-EAGLE3 draft head (3 speculative tokens). Runs on the -# b200-dgxc cluster. MiniMax-M3 modelopt NVFP4 support (vllm-project/vllm -# PR #46380) is baked into the perf container image, so no runtime patch is -# needed; prompts are routed through the chat template. Target weights are -# pre-staged at /scratch/fsw/models/MiniMax-M3-NVFP4 (launch_b200-dgxc.sh -# resolves MODEL_PATH for minimaxm3-fp4); the EAGLE3 draft is fetched next to -# the target weights. -minimaxm3-fp4-b200-vllm-mtp: - image: vllm/vllm-openai:nightly-4080263bb2c5d10deac17aaeb88e0823bc35bca9 - model: nvidia/MiniMax-M3-NVFP4 - model-prefix: minimaxm3 - runner: b200-dgxc - precision: fp4 - framework: vllm - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - { tp: 2, conc-list: [1, 8, 16, 32, 256], spec-decoding: mtp } - - { tp: 2, ep: 2, conc-list: [1, 2, 4, 16, 32, 64, 128, 256, 512], spec-decoding: mtp } - - { tp: 4, conc-list: [1, 2, 4, 8, 16, 256], spec-decoding: mtp } - - { tp: 4, ep: 4, conc-list: [1, 2, 4, 8, 256], spec-decoding: mtp } - - { tp: 8, conc-list: [1, 2, 4, 8], spec-decoding: mtp } - -# EAGLE3 speculative-decoding (spec-decoding: mtp) variant of -# minimaxm3-fp8-b300-vllm, pairing MiniMaxAI/MiniMax-M3-MXFP8 with the -# Inferact/MiniMax-M3-EAGLE3 draft head (3 speculative tokens). -# Search space mirrors the non-MTP entry trimmed at -# the extreme-concurrency end, per the dsv4-fp4-b300-vllm-mtp precedent: -# spec decode pays off at low/mid concurrency while acceptance dilutes in -# big batches, and the draft weights + draft KV shave headroom — tp2-ep2 is -# dropped entirely since its KV headroom was already thin without a draft. -minimaxm3-fp8-b300-vllm-mtp: - image: vllm/vllm-openai:nightly-4080263bb2c5d10deac17aaeb88e0823bc35bca9 - model: MiniMaxAI/MiniMax-M3-MXFP8 - model-prefix: minimaxm3 - runner: b300 - precision: fp8 - framework: vllm - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - { tp: 8, conc-start: 1, conc-end: 64, spec-decoding: mtp } - - { tp: 8, ep: 8, conc-start: 1, conc-end: 256, spec-decoding: mtp } - - { tp: 4, conc-start: 1, conc-end: 64, spec-decoding: mtp } - - { tp: 4, ep: 4, conc-start: 64, conc-end: 256, spec-decoding: mtp } - - { tp: 8, ep: 8, dp-attn: true, conc-start: 128, conc-end: 256, spec-decoding: mtp } - -# EAGLE3 speculative-decoding (spec-decoding: mtp) variant of -# minimaxm3-fp8-h200-vllm (PR #1731), pairing MiniMaxAI/MiniMax-M3-MXFP8 with -# the Inferact/MiniMax-M3-EAGLE3 draft head (3 speculative tokens, drafter -# pinned to FLASH_ATTN since the head is MHA and FlashInfer needs GQA/MQA at -# page size 128). Search space mirrors the non-MTP entry trimmed at the -# extreme-concurrency end, per the dsv4-fp4-b300-vllm-mtp / minimaxm3 b300-mtp -# precedent: spec decode pays off at low/mid concurrency while acceptance -# dilutes in big batches, and the draft weights + draft KV shave headroom. -minimaxm3-fp8-h200-vllm-mtp: - image: vllm/vllm-openai:minimax-m3 - model: MiniMaxAI/MiniMax-M3-MXFP8 - model-prefix: minimaxm3 - runner: h200 - precision: fp8 - framework: vllm - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - { tp: 4, conc-start: 1, conc-end: 32, spec-decoding: mtp } - - { tp: 8, conc-start: 1, conc-end: 128, spec-decoding: mtp } - - { tp: 8, ep: 8, conc-start: 256, conc-end: 256, spec-decoding: mtp } - - { tp: 8, ep: 8, dp-attn: true, conc-start: 256, conc-end: 256, spec-decoding: mtp } - -# EAGLE3 speculative-decoding (spec-decoding: mtp) variant of -# minimaxm3-fp8-h100-vllm (PR #1731). Same TP8-only layout (H100 80 GB has no -# room below TP8 once ~56 GB of weights land per GPU) plus the -# Inferact/MiniMax-M3-EAGLE3 draft head (3 speculative tokens, FLASH_ATTN -# drafter). DEP stays omitted as on the non-MTP entry — KV-cache init already -# failed at high conc, and the draft head + draft KV only tighten it further. -minimaxm3-fp8-h100-vllm-mtp: - image: vllm/vllm-openai:minimax-m3 - model: MiniMaxAI/MiniMax-M3-MXFP8 - model-prefix: minimaxm3 - runner: h100 - precision: fp8 - framework: vllm - multinode: false - scenarios: - fixed-seq-len: - - isl: 8192 - osl: 1024 - search-space: - - { tp: 8, conc-start: 1, conc-end: 64, spec-decoding: mtp } - - { tp: 8, ep: 8, conc-start: 128, conc-end: 256, spec-decoding: mtp } - kimik2.5-fp4-gb300-dynamo-vllm: image: vllm/vllm-openai:v0.21.0 model: nvidia/Kimi-K2.5-NVFP4