-
Notifications
You must be signed in to change notification settings - Fork 257
Add GB200 MiniMax-M3 FP4 AgentX sweep #2609
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
cquil11
wants to merge
18
commits into
main
Choose a base branch
from
agent/gb200-minimaxm3-fp4-agentx
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
18 commits
Select commit
Hold shift + click to select a range
b75fb7e
Add GB200 MiniMax M3 AgentX sweep
cquil11 41d7d47
Link MiniMax M3 performance entry
cquil11 915d4a5
Use stable vLLM for MiniMax M3
cquil11 20df687
Scope stable vLLM to GB200 MiniMax
cquil11 8014a84
Align MiniMax routing with Dynamo 1.3
cquil11 27e44d8
Merge remote-tracking branch 'origin/main' into agent/gb200-minimaxm3…
cquil11 63f7a8c
fix: use Dynamo MiniMax response parsers
cquil11 3837e71
fix: reserve MiniMax DEP KV capacity
cquil11 2e8502a
fix: expose MiniMax prompt cache tokens
cquil11 15d868e
Use stable Dynamo and add MiniMax offload probes
cquil11 a37912d
Add MiniMax 3P1D DEP8 probes
cquil11 e3281ae
Remove unsupported MiniMax prompt-details flag
cquil11 141964f
Merge remote-tracking branch 'origin/main' into agent/gb200-minimaxm3…
cquil11 6232539
Merge remote-tracking branch 'origin/main' into agent/gb200-minimaxm3…
cquil11 d707e11
perf(agentx): prune GB200 MiniMax tuning grid
cquil11 104d182
Merge remote-tracking branch 'origin/main' into agent/gb200-minimaxm3…
cquil11 1cf03b2
Stabilize GB200 MiniMax TP4 offload graphs
cquil11 42d221d
Stabilize GB200 MiniMax resident TP4 graphs
cquil11 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
80 changes: 80 additions & 0 deletions
80
...arks/multi_node/srt-slurm-recipes/vllm/minimax-m3/gb200-fp4/agentic/agg-dep4-agentic.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,80 @@ | ||
| name: "minimax-m3-vllm-agg-gb200-dep4-agentic" | ||
|
|
||
| model: { path: "minimax-m3-nvfp4", container: "vllm/vllm-openai:v0.27.1", precision: "fp4" } | ||
| identity: | ||
| model: { repo: "nvidia/MiniMax-M3-NVFP4" } | ||
| container: { image: "vllm/vllm-openai:v0.27.1" } | ||
| frameworks: { dynamo: "1.3.1" } | ||
| dynamo: { version: "1.3.1", install: true } | ||
| environment: { ETCD_LEASE_TTL: "7200" } | ||
| slurm: { time_limit: "12:00:00" } | ||
| health_check: { max_attempts: 2160, interval_seconds: 10 } | ||
| resources: { gpu_type: "gb200", gpus_per_node: 4, agg_nodes: 1, agg_workers: 1, gpus_per_agg: 4 } | ||
| infra: { etcd_nats_dedicated_node: false, nats_max_payload_mb: 32 } | ||
| frontend: | ||
| type: dynamo | ||
| enable_multiple_frontends: false | ||
| env: { DYN_TCP_REQUEST_TIMEOUT: "60" } | ||
| args: { router-mode: "kv", router-kv-events: true, router-reset-states: true, router-temperature: "0", router-session-affinity-ttl-secs: 14400, kv-cache-block-size: 128 } | ||
| backend: | ||
| type: vllm | ||
| connector: null | ||
| dp_launch_mode: per_node | ||
| kv_events_config: { aggregated: true } | ||
| aggregated_environment: | ||
| VLLM_ENGINE_READY_TIMEOUT_S: "7200" | ||
| VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS: "1800" | ||
| VLLM_FLOAT32_MATMUL_PRECISION: "high" | ||
| VLLM_FLASHINFER_ALLREDUCE_BACKEND: "trtllm" | ||
| VLLM_LOG_STATS_INTERVAL: "1" | ||
| NCCL_CUMEM_ENABLE: "1" | ||
| NCCL_MNNVL_ENABLE: "1" | ||
| NCCL_NVLS_ENABLE: "1" | ||
| UCX_MEMTYPE_CACHE: "n" | ||
| UCX_NET_DEVICES: "mlx5_0:1,mlx5_1:1,mlx5_2:1,mlx5_3:1" | ||
| UCX_TLS: "cuda_copy,cuda_ipc,rc" | ||
| NCCL_IB_HCA: "mlx5_0,mlx5_1,mlx5_2,mlx5_3" | ||
| vllm_config: | ||
| aggregated: | ||
| served-model-name: "nvidia/MiniMax-M3-NVFP4" | ||
| tensor-parallel-size: 1 | ||
| pipeline-parallel-size: 1 | ||
| data-parallel-size: 4 | ||
| data-parallel-rpc-port: 13345 | ||
| enable-expert-parallel: true | ||
| trust-remote-code: true | ||
| enable-prefix-caching: true | ||
| kv-cache-metrics: true | ||
| attention-config: '{"backend":"FLASHINFER","use_trtllm_attention":true,"indexer_kv_dtype":"fp8","minimax_m3_msa_decode_backend":"cutlass"}' | ||
| block-size: 128 | ||
| gpu-memory-utilization: 0.95 | ||
| max-model-len: 1048576 | ||
| language-model-only: true | ||
| kv-cache-dtype: "fp8" | ||
| speculative-config: '{"method":"eagle3","model":"Inferact/MiniMax-M3-EAGLE3-GQA","num_speculative_tokens":3,"attention_backend":"FLASH_ATTN"}' | ||
| compilation-config: '{"cudagraph_mode":"FULL_DECODE_ONLY"}' | ||
| stream-interval: 20 | ||
| max-cudagraph-capture-size: 512 | ||
| max-num-batched-tokens: 16384 | ||
| no-enable-flashinfer-autotune: true | ||
| reasoning-parser: "minimax_m3" | ||
| dyn-tool-call-parser: "minimax_m3" | ||
| dyn-reasoning-parser: "minimax_m3" | ||
| sbatch_directives: { cpus-per-task: "144", mem: "0" } | ||
| srun_options: { container-remap-root: "" } | ||
| benchmark: | ||
| type: custom | ||
| command: bash /infmax-workspace/benchmarks/multi_node/agentic_srt.sh | ||
| env: | ||
| INFMAX_CONTAINER_WORKSPACE: "/infmax-workspace" | ||
| RESULT_DIR: "/logs/agentic" | ||
| PORT: "8000" | ||
| IS_MULTINODE: "true" | ||
| AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID: "true" | ||
| AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING: "0" | ||
| AIPERF_DYNAMO_SESSION_TIMEOUT_SECONDS: "14400" | ||
| AIPERF_EXTRA_INPUTS: "thinking:true" | ||
| AIPERF_REQUIRED_SERVER_METRIC_PREFIX: "vllm:" | ||
| AIPERF_DATASET_MMAP_CACHE_DIR: "/aiperf_mmap_cache" | ||
| HF_HUB_CACHE: "/hf_hub_cache" | ||
| WEKA_LOADER_OVERRIDE: "semianalysis_cc_traces_weka_062126" |
82 changes: 82 additions & 0 deletions
82
...ode/srt-slurm-recipes/vllm/minimax-m3/gb200-fp4/agentic/agg-dep4-vllm-simple-agentic.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,82 @@ | ||
| name: "minimax-m3-vllm-agg-gb200-dep4-vllm-simple-agentic" | ||
|
|
||
| model: { path: "minimax-m3-nvfp4", container: "vllm/vllm-openai:v0.27.1", precision: "fp4" } | ||
| identity: | ||
| model: { repo: "nvidia/MiniMax-M3-NVFP4" } | ||
| container: { image: "vllm/vllm-openai:v0.27.1" } | ||
| frameworks: { dynamo: "1.3.1" } | ||
| dynamo: { version: "1.3.1", install: true } | ||
| environment: { ETCD_LEASE_TTL: "7200" } | ||
| slurm: { time_limit: "12:00:00" } | ||
| health_check: { max_attempts: 2160, interval_seconds: 10 } | ||
| resources: { gpu_type: "gb200", gpus_per_node: 4, agg_nodes: 1, agg_workers: 1, gpus_per_agg: 4 } | ||
| infra: { etcd_nats_dedicated_node: false, nats_max_payload_mb: 32 } | ||
| frontend: | ||
| type: dynamo | ||
| enable_multiple_frontends: false | ||
| env: { DYN_TCP_REQUEST_TIMEOUT: "60" } | ||
| args: { router-mode: "kv", router-kv-events: true, router-reset-states: true, router-temperature: "0", router-session-affinity-ttl-secs: 14400, kv-cache-block-size: 128 } | ||
| backend: | ||
| type: vllm | ||
| connector: null | ||
| dp_launch_mode: per_node | ||
| kv_events_config: { aggregated: true } | ||
| aggregated_environment: | ||
| VLLM_ENGINE_READY_TIMEOUT_S: "7200" | ||
| VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS: "1800" | ||
| VLLM_FLOAT32_MATMUL_PRECISION: "high" | ||
| VLLM_FLASHINFER_ALLREDUCE_BACKEND: "trtllm" | ||
| VLLM_LOG_STATS_INTERVAL: "1" | ||
| VLLM_USE_SIMPLE_KV_OFFLOAD: "1" | ||
| NCCL_CUMEM_ENABLE: "1" | ||
| NCCL_MNNVL_ENABLE: "1" | ||
| NCCL_NVLS_ENABLE: "1" | ||
| UCX_MEMTYPE_CACHE: "n" | ||
| UCX_NET_DEVICES: "mlx5_0:1,mlx5_1:1,mlx5_2:1,mlx5_3:1" | ||
| UCX_TLS: "cuda_copy,cuda_ipc,rc" | ||
| NCCL_IB_HCA: "mlx5_0,mlx5_1,mlx5_2,mlx5_3" | ||
| vllm_config: | ||
| aggregated: | ||
| served-model-name: "nvidia/MiniMax-M3-NVFP4" | ||
| tensor-parallel-size: 1 | ||
| pipeline-parallel-size: 1 | ||
| data-parallel-size: 4 | ||
| data-parallel-rpc-port: 13345 | ||
| enable-expert-parallel: true | ||
| trust-remote-code: true | ||
| enable-prefix-caching: true | ||
| kv-cache-metrics: true | ||
| attention-config: '{"backend":"FLASHINFER","use_trtllm_attention":true,"indexer_kv_dtype":"fp8","minimax_m3_msa_decode_backend":"cutlass"}' | ||
| block-size: 128 | ||
| gpu-memory-utilization: 0.95 | ||
| max-model-len: 1048576 | ||
| language-model-only: true | ||
| kv-cache-dtype: "fp8" | ||
| speculative-config: '{"method":"eagle3","model":"Inferact/MiniMax-M3-EAGLE3-GQA","num_speculative_tokens":3,"attention_backend":"FLASH_ATTN"}' | ||
| kv-transfer-config: '{"kv_connector":"SimpleCPUOffloadConnector","kv_role":"kv_both","kv_connector_extra_config":{"cpu_bytes_to_use":549755813888,"cpu_bytes_to_use_per_rank":137438953472,"lazy_offload":false}}' | ||
| compilation-config: '{"cudagraph_mode":"FULL_DECODE_ONLY"}' | ||
| stream-interval: 20 | ||
| max-cudagraph-capture-size: 512 | ||
| max-num-batched-tokens: 16384 | ||
| no-enable-flashinfer-autotune: true | ||
| reasoning-parser: "minimax_m3" | ||
| dyn-tool-call-parser: "minimax_m3" | ||
| dyn-reasoning-parser: "minimax_m3" | ||
| sbatch_directives: { cpus-per-task: "144", mem: "0" } | ||
| srun_options: { container-remap-root: "" } | ||
| benchmark: | ||
| type: custom | ||
| command: bash /infmax-workspace/benchmarks/multi_node/agentic_srt.sh | ||
| env: | ||
| INFMAX_CONTAINER_WORKSPACE: "/infmax-workspace" | ||
| RESULT_DIR: "/logs/agentic" | ||
| PORT: "8000" | ||
| IS_MULTINODE: "true" | ||
| AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID: "true" | ||
| AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING: "0" | ||
| AIPERF_DYNAMO_SESSION_TIMEOUT_SECONDS: "14400" | ||
| AIPERF_EXTRA_INPUTS: "thinking:true" | ||
| AIPERF_REQUIRED_SERVER_METRIC_PREFIX: "vllm:" | ||
| AIPERF_DATASET_MMAP_CACHE_DIR: "/aiperf_mmap_cache" | ||
| HF_HUB_CACHE: "/hf_hub_cache" | ||
| WEKA_LOADER_OVERRIDE: "semianalysis_cc_traces_weka_062126" |
80 changes: 80 additions & 0 deletions
80
...arks/multi_node/srt-slurm-recipes/vllm/minimax-m3/gb200-fp4/agentic/agg-dep8-agentic.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,80 @@ | ||
| name: "minimax-m3-vllm-agg-gb200-dep8-agentic" | ||
|
|
||
| model: { path: "minimax-m3-nvfp4", container: "vllm/vllm-openai:v0.27.1", precision: "fp4" } | ||
| identity: | ||
| model: { repo: "nvidia/MiniMax-M3-NVFP4" } | ||
| container: { image: "vllm/vllm-openai:v0.27.1" } | ||
| frameworks: { dynamo: "1.3.1" } | ||
| dynamo: { version: "1.3.1", install: true } | ||
| environment: { ETCD_LEASE_TTL: "7200" } | ||
| slurm: { time_limit: "12:00:00" } | ||
| health_check: { max_attempts: 2160, interval_seconds: 10 } | ||
| resources: { gpu_type: "gb200", gpus_per_node: 4, agg_nodes: 2, agg_workers: 1, gpus_per_agg: 8 } | ||
| infra: { etcd_nats_dedicated_node: false, nats_max_payload_mb: 32 } | ||
| frontend: | ||
| type: dynamo | ||
| enable_multiple_frontends: false | ||
| env: { DYN_TCP_REQUEST_TIMEOUT: "60" } | ||
| args: { router-mode: "kv", router-kv-events: true, router-reset-states: true, router-temperature: "0", router-session-affinity-ttl-secs: 14400, kv-cache-block-size: 128 } | ||
| backend: | ||
| type: vllm | ||
| connector: null | ||
| dp_launch_mode: per_node | ||
| kv_events_config: { aggregated: true } | ||
| aggregated_environment: | ||
| VLLM_ENGINE_READY_TIMEOUT_S: "7200" | ||
| VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS: "1800" | ||
| VLLM_FLOAT32_MATMUL_PRECISION: "high" | ||
| VLLM_FLASHINFER_ALLREDUCE_BACKEND: "trtllm" | ||
| VLLM_LOG_STATS_INTERVAL: "1" | ||
| NCCL_CUMEM_ENABLE: "1" | ||
| NCCL_MNNVL_ENABLE: "1" | ||
| NCCL_NVLS_ENABLE: "1" | ||
| UCX_MEMTYPE_CACHE: "n" | ||
| UCX_NET_DEVICES: "mlx5_0:1,mlx5_1:1,mlx5_2:1,mlx5_3:1" | ||
| UCX_TLS: "cuda_copy,cuda_ipc,rc" | ||
| NCCL_IB_HCA: "mlx5_0,mlx5_1,mlx5_2,mlx5_3" | ||
| vllm_config: | ||
| aggregated: | ||
| served-model-name: "nvidia/MiniMax-M3-NVFP4" | ||
| tensor-parallel-size: 1 | ||
| pipeline-parallel-size: 1 | ||
| data-parallel-size: 8 | ||
| data-parallel-rpc-port: 13345 | ||
| enable-expert-parallel: true | ||
| trust-remote-code: true | ||
| enable-prefix-caching: true | ||
| kv-cache-metrics: true | ||
| attention-config: '{"backend":"FLASHINFER","use_trtllm_attention":true,"indexer_kv_dtype":"fp8","minimax_m3_msa_decode_backend":"cutlass"}' | ||
| block-size: 128 | ||
| gpu-memory-utilization: 0.95 | ||
| max-model-len: 1048576 | ||
| language-model-only: true | ||
| kv-cache-dtype: "fp8" | ||
| speculative-config: '{"method":"eagle3","model":"Inferact/MiniMax-M3-EAGLE3-GQA","num_speculative_tokens":3,"attention_backend":"FLASH_ATTN"}' | ||
| compilation-config: '{"cudagraph_mode":"FULL_DECODE_ONLY"}' | ||
| stream-interval: 20 | ||
| max-cudagraph-capture-size: 512 | ||
| max-num-batched-tokens: 16384 | ||
| no-enable-flashinfer-autotune: true | ||
| reasoning-parser: "minimax_m3" | ||
| dyn-tool-call-parser: "minimax_m3" | ||
| dyn-reasoning-parser: "minimax_m3" | ||
| sbatch_directives: { cpus-per-task: "144", mem: "0" } | ||
| srun_options: { container-remap-root: "" } | ||
| benchmark: | ||
| type: custom | ||
| command: bash /infmax-workspace/benchmarks/multi_node/agentic_srt.sh | ||
| env: | ||
| INFMAX_CONTAINER_WORKSPACE: "/infmax-workspace" | ||
| RESULT_DIR: "/logs/agentic" | ||
| PORT: "8000" | ||
| IS_MULTINODE: "true" | ||
| AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID: "true" | ||
| AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING: "0" | ||
| AIPERF_DYNAMO_SESSION_TIMEOUT_SECONDS: "14400" | ||
| AIPERF_EXTRA_INPUTS: "thinking:true" | ||
| AIPERF_REQUIRED_SERVER_METRIC_PREFIX: "vllm:" | ||
| AIPERF_DATASET_MMAP_CACHE_DIR: "/aiperf_mmap_cache" | ||
| HF_HUB_CACHE: "/hf_hub_cache" | ||
| WEKA_LOADER_OVERRIDE: "semianalysis_cc_traces_weka_062126" |
99 changes: 99 additions & 0 deletions
99
...marks/multi_node/srt-slurm-recipes/vllm/minimax-m3/gb200-fp4/agentic/agg-tp4-agentic.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,99 @@ | ||
| name: "minimax-m3-vllm-agg-gb200-tp4-agentic" | ||
|
|
||
| model: | ||
| path: "minimax-m3-nvfp4" | ||
| container: "vllm/vllm-openai:v0.27.1" | ||
| precision: "fp4" | ||
|
|
||
| identity: | ||
| model: { repo: "nvidia/MiniMax-M3-NVFP4" } | ||
| container: { image: "vllm/vllm-openai:v0.27.1" } | ||
| frameworks: { dynamo: "1.3.1" } | ||
|
|
||
| dynamo: { version: "1.3.1", install: true } | ||
| environment: { ETCD_LEASE_TTL: "7200" } | ||
|
|
||
| slurm: { time_limit: "12:00:00" } | ||
| health_check: { max_attempts: 2160, interval_seconds: 10 } | ||
|
|
||
| resources: | ||
| gpu_type: "gb200" | ||
| gpus_per_node: 4 | ||
| agg_nodes: 1 | ||
| agg_workers: 1 | ||
| gpus_per_agg: 4 | ||
|
|
||
| infra: { etcd_nats_dedicated_node: false, nats_max_payload_mb: 32 } | ||
|
|
||
| frontend: | ||
| type: dynamo | ||
| enable_multiple_frontends: false | ||
| env: { DYN_TCP_REQUEST_TIMEOUT: "60" } | ||
| args: | ||
| router-mode: "kv" | ||
| router-kv-events: true | ||
| router-reset-states: true | ||
| router-temperature: "0" | ||
| router-session-affinity-ttl-secs: 14400 | ||
| kv-cache-block-size: 128 | ||
|
|
||
| backend: | ||
| type: vllm | ||
| connector: null | ||
| kv_events_config: { aggregated: true } | ||
| aggregated_environment: | ||
| VLLM_ENGINE_READY_TIMEOUT_S: "7200" | ||
| VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS: "1800" | ||
| VLLM_FLOAT32_MATMUL_PRECISION: "high" | ||
| VLLM_FLASHINFER_ALLREDUCE_BACKEND: "trtllm" | ||
| VLLM_LOG_STATS_INTERVAL: "1" | ||
| NCCL_CUMEM_ENABLE: "1" | ||
| NCCL_MNNVL_ENABLE: "1" | ||
| NCCL_NVLS_ENABLE: "1" | ||
| UCX_MEMTYPE_CACHE: "n" | ||
| UCX_NET_DEVICES: "mlx5_0:1,mlx5_1:1,mlx5_2:1,mlx5_3:1" | ||
| UCX_TLS: "cuda_copy,cuda_ipc,rc" | ||
| NCCL_IB_HCA: "mlx5_0,mlx5_1,mlx5_2,mlx5_3" | ||
| vllm_config: | ||
| aggregated: | ||
| served-model-name: "nvidia/MiniMax-M3-NVFP4" | ||
| tensor-parallel-size: 4 | ||
| pipeline-parallel-size: 1 | ||
| trust-remote-code: true | ||
| enable-prefix-caching: true | ||
| kv-cache-metrics: true | ||
| attention-config: '{"backend":"FLASHINFER","use_trtllm_attention":true,"indexer_kv_dtype":"fp8","minimax_m3_msa_decode_backend":"cutlass"}' | ||
| compilation-config: '{"cudagraph_mode":"FULL_DECODE_ONLY"}' | ||
| block-size: 128 | ||
| gpu-memory-utilization: 0.9 | ||
| max-model-len: 1048576 | ||
| language-model-only: true | ||
| kv-cache-dtype: "fp8" | ||
| speculative-config: '{"method":"eagle3","model":"Inferact/MiniMax-M3-EAGLE3-GQA","num_speculative_tokens":3,"attention_backend":"FLASH_ATTN"}' | ||
| stream-interval: 20 | ||
| max-cudagraph-capture-size: 512 | ||
| max-num-batched-tokens: 16384 | ||
| no-enable-flashinfer-autotune: true | ||
| reasoning-parser: "minimax_m3" | ||
| dyn-tool-call-parser: "minimax_m3" | ||
| dyn-reasoning-parser: "minimax_m3" | ||
|
|
||
| sbatch_directives: { cpus-per-task: "144", mem: "0" } | ||
| srun_options: { container-remap-root: "" } | ||
|
|
||
| benchmark: | ||
| type: custom | ||
| command: bash /infmax-workspace/benchmarks/multi_node/agentic_srt.sh | ||
| env: | ||
| INFMAX_CONTAINER_WORKSPACE: "/infmax-workspace" | ||
| RESULT_DIR: "/logs/agentic" | ||
| PORT: "8000" | ||
| IS_MULTINODE: "true" | ||
| AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID: "true" | ||
| AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING: "0" | ||
| AIPERF_DYNAMO_SESSION_TIMEOUT_SECONDS: "14400" | ||
| AIPERF_EXTRA_INPUTS: "thinking:true" | ||
| AIPERF_REQUIRED_SERVER_METRIC_PREFIX: "vllm:" | ||
| AIPERF_DATASET_MMAP_CACHE_DIR: "/aiperf_mmap_cache" | ||
| HF_HUB_CACHE: "/hf_hub_cache" | ||
| WEKA_LOADER_OVERRIDE: "semianalysis_cc_traces_weka_062126" | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🟡 WARNING: Session-affinity TTL is not configured, and
AIPERF_DYNAMO_SESSION_TIMEOUT_SECONDS: "14400"(line 84) is dead on this routing path — this applies to all 14 new recipes.Why it matters: These recipes opt into header-based session routing (
AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID: "true"+AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING: "0"). With that opt-in,benchmark_lib.shskips the conv-aware-routing branch (benchmarks/benchmark_lib.sh:2055-2064) — the only consumer ofAIPERF_DYNAMO_SESSION_TIMEOUT_SECONDS— so the 14400s value is never applied anywhere. Per the comment in that same block, the upstream router default affinity TTL is 300s, which is shorter than an overloaded high-concurrency agentic request. Every existing dsv4 recipe that uses this exact header-routing pattern sets the TTL router-side instead (e.g.disagg-gb200-1p1d-dep8-dep8-mtp-agentic.yaml:46usesrouter-session-affinity-ttl-secs: 14400). Without it, sessions at c128–c256 can lose worker affinity mid-conversation, silently degrading the KV-aware routing this sweep is meant to measure.Fix: Set the TTL on the frontend like the dsv4 recipes (and drop the dead
AIPERF_DYNAMO_SESSION_TIMEOUT_SECONDSenv entries):If srt-slurm v1.0.50 / Dynamo v1.3.1 applies a long affinity TTL by default for
X-Dynamo-Session-IDsessions, please note that in the PR description instead. Fix this →There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The functional half of this is fixed: commit d707e11 ("perf(agentx): prune GB200 MiniMax tuning grid") added
router-session-affinity-ttl-secs: 14400to the frontendargsblock in this recipe (and the other 13), so session affinity is now actually enforced router-side instead of relying on the dead env var. However, the second part of the suggested fix — dropping the now-confirmed-deadAIPERF_DYNAMO_SESSION_TIMEOUT_SECONDS: "14400"entry frombenchmark.env— was not done; it is still present at line 93 in this file (and in all the other 13 recipes). It remains inert cleanup at this point, not a functional issue, sincebenchmark_lib.shskips the conv-aware-routing branch that would have consumed it.