MiniMax-M3 H200 AgentX EAGLE3 tuning on vLLM v0.27.1 #2565
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,283 @@ | ||
| #!/usr/bin/env bash | ||
| set -euo pipefail | ||
| set -x | ||
|
|
||
| # MiniMax-M3 MXFP8 H200 AgentX (agentic-coding) recipe with EAGLE3 speculative | ||
| # decoding — the spec-decoding=mtp variant of agentic/minimaxm3_fp8_h200.sh. | ||
| # Everything outside the speculative block mirrors the non-MTP agentic sibling | ||
| # (Mooncake host-DRAM KV offload, --block-size 128, --language-model-only, | ||
| # --kv-cache-dtype fp8, TRITON_ATTN, gmu 0.92, minimax_m3 parsers, vllm-router | ||
| # for DP-attention), so the spec-decode delta is readable at equal concurrency. | ||
| # | ||
| # Speculative config: Inferact/MiniMax-M3-EAGLE3-GQA, 3 speculative tokens, | ||
| # and the committed thinking-on golden AL. The | ||
| # drafter is pinned to FLASH_ATTN as on the other CUDA M3 MTP recipes: the | ||
| # EAGLE3 head is MHA and FlashInfer only serves page size 128 through its | ||
| # trtllm-gen kernel, which requires GQA/MQA. | ||
| # | ||
| # Throughput runs pin synthetic acceptance to the committed golden AL; the | ||
| # EVAL_ONLY accuracy run keeps real target verification. See SYNTHETIC_ACCEPT_LEN. | ||
|
|
||
| source "$(dirname "$0")/../../benchmark_lib.sh" | ||
|
|
||
| check_env_vars MODEL TP CONC KV_OFFLOADING TOTAL_CPU_DRAM_GB RESULT_DIR DURATION EP_SIZE DP_ATTENTION | ||
|
|
||
| DRAFT_MODEL="Inferact/MiniMax-M3-EAGLE3-GQA" | ||
|
|
||
| if [[ -n "${SLURM_JOB_ID:-}" ]]; then | ||
| echo "JOB $SLURM_JOB_ID running on ${SLURMD_NODENAME:-unknown}" | ||
| fi | ||
|
|
||
| resolve_complete_model_snapshot() { | ||
| python3 - "$1" <<'PY' | ||
| import json | ||
| import sys | ||
| from pathlib import Path | ||
|
|
||
| model_cache_dir = Path(sys.argv[1]) | ||
| try: | ||
| revision = model_cache_dir.joinpath("refs/main").read_text().strip() | ||
| except OSError: | ||
| raise SystemExit | ||
|
|
||
| if not revision or Path(revision).name != revision: | ||
| raise SystemExit | ||
|
|
||
| snapshot = model_cache_dir / "snapshots" / revision | ||
| index_path = snapshot / "model.safetensors.index.json" | ||
| required_files = ( | ||
| snapshot / "config.json", | ||
| snapshot / "tokenizer_config.json", | ||
| index_path, | ||
| ) | ||
| if not all(path.is_file() for path in required_files): | ||
| raise SystemExit | ||
| try: | ||
| weight_map = json.loads(index_path.read_text())["weight_map"] | ||
| except (KeyError, json.JSONDecodeError, OSError): | ||
| raise SystemExit | ||
| shards = {snapshot / filename for filename in weight_map.values()} | ||
| if shards and all(path.is_file() for path in shards): | ||
|
Comment on lines
+31
to
+60
Contributor
There was a problem hiding this comment. 🟡 The Extended reasoning...What the duplication is: Why it's not caught by existing shared code: Both scripts already Concrete consequence — step-by-step:
Why the current approach doesn't prevent this: Nothing forces the two scripts to be updated together; they're independent shell scripts, and the shared Suggested fix: Extract |
||
| print(snapshot) | ||
| PY | ||
| } | ||
|
|
||
| if [[ -n "${MODEL_PATH:-}" ]]; then | ||
| if [[ ! -d "$MODEL_PATH" || -z "$(ls -A "$MODEL_PATH" 2>/dev/null)" ]]; then | ||
| hf download "$MODEL" --local-dir "$MODEL_PATH" | ||
| fi | ||
| else | ||
| MODEL_CACHE_ROOT="${HF_HUB_CACHE:-${HF_HOME:-$HOME/.cache/huggingface/hub}}" | ||
| MODEL_CACHE_DIR="$MODEL_CACHE_ROOT/models--${MODEL//\//--}" | ||
| mkdir -p "$MODEL_CACHE_ROOT" | ||
| MODEL_PATH=$(resolve_complete_model_snapshot "$MODEL_CACHE_DIR") | ||
| if [[ -z "$MODEL_PATH" ]]; then | ||
| exec 9>"$MODEL_CACHE_ROOT/.minimaxm3-download.lock" | ||
| flock -w 3600 9 | ||
| MODEL_PATH=$(resolve_complete_model_snapshot "$MODEL_CACHE_DIR") | ||
| if [[ -z "$MODEL_PATH" ]]; then | ||
| DOWNLOADED_MODEL_PATH=$(hf download "$MODEL") | ||
| MODEL_PATH=$(resolve_complete_model_snapshot "$MODEL_CACHE_DIR") | ||
| if [[ -z "$MODEL_PATH" ]]; then | ||
| echo "Downloaded model snapshot is incomplete: $DOWNLOADED_MODEL_PATH" >&2 | ||
| exit 1 | ||
| fi | ||
| fi | ||
| flock -u 9 | ||
| fi | ||
| echo "Using complete cached model snapshot: $MODEL_PATH" | ||
| export MODEL_PATH | ||
| fi | ||
|
|
||
| # The EAGLE3 draft is never pre-staged next to the target checkpoint; fetch it | ||
| # into the shared HF cache. That cache is a network FS where concurrent | ||
| # day-zero downloads hit huggingface_hub's WeakFileLock "[Errno 116] Stale file | ||
| # handle" race, so retry (the download resumes) as the fixed-seq-len MTP | ||
| # recipes do. | ||
| for attempt in 1 2 3 4 5; do | ||
| hf download "$DRAFT_MODEL" && break | ||
| if [ "$attempt" = 5 ]; then echo "hf download of $DRAFT_MODEL failed after $attempt attempts" >&2; exit 1; fi | ||
| echo "hf download attempt $attempt failed; retrying in 60s" >&2 | ||
| sleep 60 | ||
| done | ||
| nvidia-smi | ||
|
|
||
| export WEKA_LOADER_OVERRIDE=semianalysis_cc_traces_weka_062126 | ||
| resolve_trace_source | ||
| install_agentic_deps | ||
|
|
||
| export VLLM_ENGINE_READY_TIMEOUT_S=3600 | ||
| export PYTHONNOUSERSITE=1 | ||
|
|
||
| SERVER_LOG="$RESULT_DIR/server.log" | ||
| ROUTER_LOG="$RESULT_DIR/router.log" | ||
| MOONCAKE_MASTER_LOG="$RESULT_DIR/mooncake_master.log" | ||
| mkdir -p "$RESULT_DIR" | ||
|
|
||
| OFFLOAD_ARGS=() | ||
| MODEL_CHECKPOINT_PAGE_CACHE_GIB=414 | ||
| MOONCAKE_LOCAL_BUFFER_GIB=4 | ||
| case "${KV_OFFLOAD_BACKEND:-}" in | ||
| "") | ||
| require_agentic_kv_offload_none | ||
| ;; | ||
| vllm-simple) | ||
| require_agentic_kv_offload_backend vllm-simple | ||
| TOTAL_CPU_DRAM_GIB=$((TOTAL_CPU_DRAM_GB * 1000000000 / 1073741824)) | ||
| CPU_OFFLOAD_GIB_PER_RANK=$(((TOTAL_CPU_DRAM_GIB - MODEL_CHECKPOINT_PAGE_CACHE_GIB) / TP)) | ||
| if (( CPU_OFFLOAD_GIB_PER_RANK <= 0 )); then | ||
| echo "Error: CPU DRAM budget is too small for checkpoint cache and KV offload" >&2 | ||
| exit 1 | ||
| fi | ||
| CPU_BYTES_PER_RANK=$((CPU_OFFLOAD_GIB_PER_RANK * 1024 * 1024 * 1024)) | ||
| export PYTHONHASHSEED=42 | ||
| OFFLOAD_ARGS=( | ||
| --kv-transfer-config | ||
| "{\"kv_connector\":\"SimpleCPUOffloadConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{\"cpu_bytes_to_use_per_rank\":${CPU_BYTES_PER_RANK},\"lazy_offload\":false}}" | ||
| ) | ||
| ;; | ||
| mooncake) | ||
| require_agentic_kv_offload_backend mooncake | ||
| TOTAL_CPU_DRAM_GIB=$((TOTAL_CPU_DRAM_GB * 1000000000 / 1073741824)) | ||
| PER_RANK_GB=$(((TOTAL_CPU_DRAM_GIB - MODEL_CHECKPOINT_PAGE_CACHE_GIB) / TP - MOONCAKE_LOCAL_BUFFER_GIB)) | ||
| if (( PER_RANK_GB <= 0 )); then | ||
| echo "Error: CPU DRAM budget is too small for checkpoint cache and KV offload" >&2 | ||
| exit 1 | ||
| fi | ||
| MOONCAKE_VERSION=0.3.11.post1 | ||
| agentic_pip_install --quiet --no-cache-dir --no-deps \ | ||
| --force-reinstall "mooncake-transfer-engine-cuda13==$MOONCAKE_VERSION" | ||
| python3 -c "from mooncake.store import MooncakeDistributedStore" >/dev/null | ||
| MOONCAKE_MASTER_PORT=$((PORT + 12000)) | ||
| MOONCAKE_CONFIG_PATH="$RESULT_DIR/mooncake_config.json" | ||
| cat > "$MOONCAKE_CONFIG_PATH" <<EOF | ||
| { | ||
| "mode": "embedded", | ||
| "metadata_server": "P2PHANDSHAKE", | ||
| "master_server_address": "127.0.0.1:$MOONCAKE_MASTER_PORT", | ||
| "global_segment_size": "${PER_RANK_GB}GB", | ||
| "local_buffer_size": "4GB", | ||
|
Comment on lines
+119
to
+159
Contributor
There was a problem hiding this comment. 🟡 Line 139 hardcodes Extended reasoning...
The bug is a DRY violation: the same logical quantity (the reserved local buffer size) is expressed twice — once as the variable used in the capacity-guard arithmetic, and once as an independently-maintained literal in the generated config. Today both are Nothing in the surrounding code catches this: This is proven by direct comparison with the sibling recipe Step-by-step reproduction of the risk: (1) today Fix: change line 139 from |
||
| "protocol": "rdma", | ||
| "device_name": "", | ||
| "enable_offload": false | ||
| } | ||
| EOF | ||
| export MOONCAKE_CONFIG_PATH PYTHONHASHSEED=0 MC_SLICE_SIZE=1048576 MC_WORKERS_PER_CTX=4 | ||
| export MC_ENABLE_DEST_DEVICE_AFFINITY=1 | ||
| mooncake_master --port "$MOONCAKE_MASTER_PORT" \ | ||
| --eviction_high_watermark_ratio=0.80 \ | ||
| --eviction_ratio=0.10 > "$MOONCAKE_MASTER_LOG" 2>&1 & | ||
| MOONCAKE_MASTER_PID=$! | ||
| sleep 2 | ||
| kill -0 "$MOONCAKE_MASTER_PID" | ||
| OFFLOAD_ARGS=( | ||
| --kv-transfer-config | ||
| '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_both","kv_connector_extra_config":{"load_async":true}}' | ||
| ) | ||
| ;; | ||
| *) | ||
| echo "Error: unsupported KV_OFFLOAD_BACKEND='$KV_OFFLOAD_BACKEND'" >&2 | ||
| exit 1 | ||
| ;; | ||
| esac | ||
|
|
||
| PARALLEL_ARGS=(--tensor-parallel-size "$TP" --data-parallel-size 1) | ||
| if [[ "$DP_ATTENTION" == "true" ]]; then | ||
| PARALLEL_ARGS=(--tensor-parallel-size 1 --data-parallel-size "$TP") | ||
| fi | ||
|
|
||
| EP_ARGS=() | ||
| if (( EP_SIZE > 1 )); then | ||
| EP_ARGS=(--enable-expert-parallel) | ||
| fi | ||
|
|
||
| VLLM_BACKEND_PORT="$PORT" | ||
| if [[ "$DP_ATTENTION" == "true" ]]; then | ||
| VLLM_BACKEND_PORT=$((PORT + 1)) | ||
| export AIPERF_HTTP_X_SESSION_ID_FROM_CORRELATION_ID=1 | ||
| agentic_pip_install --quiet 'vllm-router==0.1.14' | ||
| fi | ||
|
|
||
| export AIPERF_SERVER_METRICS_URLS="http://localhost:${VLLM_BACKEND_PORT}/metrics" | ||
| export AIPERF_REQUIRED_SERVER_METRIC_PREFIX="vllm:" | ||
|
|
||
| # use 3 speculative tokens for all configs, matching the MiniMax-M3 MTP recipes | ||
| NUM_SPEC_TOKENS=3 | ||
| TOKENS_PER_SEQ=$((1 + NUM_SPEC_TOKENS)) | ||
|
|
||
| # AgentX pins acceptance to the committed golden AL so submissions are compared | ||
| # on system performance at a fixed acceptance target rather than on draft-head | ||
| # quality. 2.78 is minimaxm3_eagle3_gqa.yaml thinking_on[3]. | ||
| # | ||
| # EVAL_ONLY switches back to real verification: synthetic acceptance commits | ||
| # drafted tokens regardless of the target logits, so generated text is wrong and | ||
| # the eval would score ~0 (same split as dsv4_fp4_b*_vllm_mtp.sh). | ||
| SYNTHETIC_ACCEPT_LEN=2.78 | ||
| if [ "${EVAL_ONLY:-false}" = "true" ]; then | ||
| SPEC_CONFIG="{\"method\": \"eagle3\", \"model\": \"$DRAFT_MODEL\", \"num_speculative_tokens\": $NUM_SPEC_TOKENS, \"attention_backend\": \"FLASH_ATTN\"}" | ||
| else | ||
| SPEC_CONFIG="{\"method\": \"eagle3\", \"model\": \"$DRAFT_MODEL\", \"num_speculative_tokens\": $NUM_SPEC_TOKENS, \"attention_backend\": \"FLASH_ATTN\", \"rejection_sample_method\": \"synthetic\", \"synthetic_acceptance_length\": $SYNTHETIC_ACCEPT_LEN}" | ||
| fi | ||
|
|
||
| # DEP distributes the live AgentX session trees across its data-parallel ranks. | ||
| if [[ "$DP_ATTENTION" == "true" ]]; then | ||
| if (( 2 * CONC % TP != 0 )); then | ||
| echo "DEP requires 2*CONC divisible by TP (CONC=$CONC TP=$TP)" >&2 | ||
| exit 1 | ||
| fi | ||
| MAX_NUM_SEQS=$((2 * CONC / TP)) | ||
| else | ||
| MAX_NUM_SEQS=$((2 * CONC)) | ||
| fi | ||
| # Cudagraph capture sizes are in TOKENS: a decode batch of S sequences verifies | ||
| # S*(1+NUM_SPEC_TOKENS) tokens, so cap capture at MAX_NUM_SEQS*(1+N) or the | ||
| # FULL_DECODE_ONLY ladder tops out at MAX_NUM_SEQS/(1+N) sequences and the | ||
| # largest decode batches fall back to eager. | ||
| MAX_CUDAGRAPH_CAPTURE_SIZE=$((MAX_NUM_SEQS * TOKENS_PER_SEQ)) | ||
|
|
||
| vllm serve "$MODEL_PATH" --served-model-name "$MODEL" \ | ||
| --host 0.0.0.0 \ | ||
| --port "$VLLM_BACKEND_PORT" \ | ||
| "${PARALLEL_ARGS[@]}" \ | ||
| "${EP_ARGS[@]}" \ | ||
| --gpu-memory-utilization 0.90 \ | ||
| --kv-cache-dtype fp8 \ | ||
| --attention-backend TRITON_ATTN \ | ||
| --block-size 128 \ | ||
| --language-model-only \ | ||
| --enable-prefix-caching \ | ||
| --enable-prompt-tokens-details \ | ||
| --default-chat-template-kwargs '{"thinking_mode":"enabled"}' \ | ||
| --max-num-seqs "$MAX_NUM_SEQS" \ | ||
| --max-cudagraph-capture-size "$MAX_CUDAGRAPH_CAPTURE_SIZE" \ | ||
| --speculative-config "$SPEC_CONFIG" \ | ||
| --tool-call-parser minimax_m3 \ | ||
| --reasoning-parser minimax_m3 \ | ||
| --enable-auto-tool-choice \ | ||
| --trust-remote-code \ | ||
| "${OFFLOAD_ARGS[@]}" > "$SERVER_LOG" 2>&1 & | ||
| SERVER_PID=$! | ||
|
|
||
| wait_for_server_ready --port "$VLLM_BACKEND_PORT" --server-log "$SERVER_LOG" --server-pid "$SERVER_PID" | ||
|
|
||
| if [[ "$DP_ATTENTION" == "true" ]]; then | ||
| vllm-router \ | ||
| --worker-urls "http://localhost:$VLLM_BACKEND_PORT" \ | ||
| --policy consistent_hash \ | ||
| --intra-node-data-parallel-size "$TP" \ | ||
| --host 0.0.0.0 \ | ||
| --port "$PORT" \ | ||
| --prometheus-host 127.0.0.1 \ | ||
| --prometheus-port "$((PORT + 10000))" \ | ||
| --request-timeout-secs 14400 \ | ||
| --disable-retries > "$ROUTER_LOG" 2>&1 & | ||
| ROUTER_PID=$! | ||
| wait_for_server_ready --port "$PORT" --server-log "$ROUTER_LOG" --server-pid "$ROUTER_PID" | ||
| fi | ||
|
|
||
| if [ "${EVAL_ONLY}" = "true" ]; then | ||
| run_eval --port "$PORT" | ||
| else | ||
| build_replay_cmd "$RESULT_DIR" | ||
| run_agentic_replay_and_write_outputs "$RESULT_DIR" | ||
| fi | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🟡 The header comment (lines 7-10) states this recipe mirrors the non-MTP sibling at
gmu 0.92specifically so the spec-decode delta is readable at equal concurrency, but line 218 actually sets--gpu-memory-utilization 0.90(the sibling minimaxm3_fp8_h200.sh:145 does use 0.92). Either the comment is stale and should explain the 2% reduction (likely HBM headroom for the EAGLE3 draft head), or the value should be changed to 0.92 to match the stated invariant.Extended reasoning...
The bug: The header comment block at the top of
minimaxm3_fp8_h200_mtp.sh(lines 7-10) explicitly enumerates the settings this MTP recipe is supposed to mirror from its non-MTP sibling,agentic/minimaxm3_fp8_h200.sh: Mooncake host-DRAM KV offload,--block-size 128,--language-model-only,--kv-cache-dtype fp8,TRITON_ATTN,gmu 0.92, the minimax_m3 parsers, and vllm-router for DP-attention. The comment then states the purpose of this mirroring: so that 'the spec-decode delta is readable at equal concurrency' — i.e. the intent is to hold every non-speculative-decoding variable constant relative to the sibling, isolating the EAGLE3 contribution as the only difference between the two benchmark curves.\n\nThe code path that contradicts it: Line 218 of the same file sets--gpu-memory-utilization 0.90in the actualvllm serveinvocation. I verified the sibling script independently —benchmarks/single_node/agentic/minimaxm3_fp8_h200.sh:145sets--gpu-memory-utilization 0.92. So the comment's claim that gmu is held at 0.92 to match the sibling is factually false for the script as written; the actual value differs by 2 percentage points of GPU memory.\n\nWhy nothing catches this: There's no test or lint that cross-checks a recipe's header comment against its own flag values, and the comment reads as authoritative documentation of the experimental design (which knobs are controlled vs. varied). A maintainer or reviewer reading only the header would reasonably conclude gmu is not a confound between the two curves, when in fact it is.\n\nImpact: This isn't a runtime bug — the script runs fine at 0.90, and 0.90 is very plausibly the correct value (leaving extra HBM headroom for the EAGLE3 draft head's KV cache and weights, which the sibling doesn't need). But the documented invariant of the recipe — 'only the speculative-decoding block differs; everything else is held equal so the delta is attributable to spec decode alone' — is violated by a variable the comment claims is not different. Concretely: at equal concurrency, the MTP run has 2% less KV-cache capacity than the sibling for a reason unrelated to speculative decoding. This slightly reduces max batched sequences / KV-cache headroom, which could itself shift throughput or latency at high concurrency independent of the EAGLE3 draft's effect — exactly the kind of confound the comment claims doesn't exist. Future maintainers relying on this comment to reason about what varies between the two curves will be misled.\n\nProof walkthrough:\n1. Readminimaxm3_fp8_h200_mtp.shlines 7-10: comment lists 'gmu 0.92' as mirrored from the sibling, framed as necessary so the spec-decode delta is readable at equal concurrency.\n2. Readminimaxm3_fp8_h200_mtp.shline 218:--gpu-memory-utilization 0.90.\n3. Readminimaxm3_fp8_h200.shline 145 (the sibling referenced by the comment):--gpu-memory-utilization 0.92.\n4. 0.90 ≠ 0.92 — the comment's factual claim about what's held constant between the two recipes is wrong.\n\nFix: Either bump line 218 to--gpu-memory-utilization 0.92if there's no real need for the reduction, or (more likely correct, given the EAGLE3 draft's extra memory footprint) update the header comment to say 'gmu 0.90 (2% lower than the sibling's 0.92 to leave HBM headroom for the EAGLE3 draft head)' so the documented invariant matches reality and doesn't mislead readers about what's actually held constant across the two curves.