Skip to content

perf(agentx): refresh DeepSeek V4 MTP frontier on MI355X #2540

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
cquil11 wants to merge 23 commits into
base: main
Choose a base branch
from
+134 −68
Open

perf(agentx): refresh DeepSeek V4 MTP frontier on MI355X #2540

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
8c63451
perf(agentx): tune DSv4 MTP frontier on MI355X
cquil11 Aug 9, 2026
2d5d0c2
docs(perf): register MI355X DSv4 tuning submission
cquil11 Aug 9, 2026
323fc15
bench: report vLLM prompt cache usage on MI355X
cquil11 Aug 9, 2026
4d77e28
perf(mi355x): match DSV4 MTP search-space coverage
cquil11 Aug 9, 2026
3c2131e
perf(mi355x): add fit-validated TP4 MTP band
cquil11 Aug 9, 2026
d813d64
perf(mi355x): refine fast frontier probes
cquil11 Aug 10, 2026
8e6d1b2
perf(mi355x): probe upper DSV4 MTP frontier
cquil11 Aug 10, 2026
cbbc4bb
perf(mi355x): isolate native DRAM offload at c64
cquil11 Aug 10, 2026
96703c9
perf(mi355x): publish measured DSV4 MTP frontier
cquil11 Aug 10, 2026
747b36a
Merge remote-tracking branch 'origin/main' into agent/dsv4-mi355x-age…
cquil11 Aug 10, 2026
11edf7a
perf(mi355x): probe larger DSV4 prefill budget
cquil11 Aug 10, 2026
86773f8
perf(mi355x): probe shallower DSV4 MTP
cquil11 Aug 10, 2026
9e245f4
perf(mi355x): probe piecewise DSV4 graphs
cquil11 Aug 10, 2026
1de2b07
perf(mi355x): probe INT4 quick reduce
cquil11 Aug 10, 2026
4c52de6
perf(mi355x): test compiled DSV4 graph path
cquil11 Aug 10, 2026
70a42ce
Merge remote-tracking branch 'origin/main' into agent/dsv4-mi355x-age…
cquil11 Aug 10, 2026
b244cb0
Merge origin/main into agent/dsv4-mi355x-agentic-mtp-opt
cquil11 Aug 10, 2026
428e6b7
perf(mi355x): add offload and DEP collection points
cquil11 Aug 10, 2026
f894cc8
fix(mi355x): bound Mooncake transfer logging
cquil11 Aug 10, 2026
1b48a1a
fix(mi355x): fit native context on DEP4
cquil11 Aug 10, 2026
9d0cdf9
Merge remote-tracking branch 'origin/main' into agent/dsv4-mi355x-age…
cquil11 Aug 10, 2026
8a9cc12
Merge current main and refresh MI355X on vLLM v0.27.0
cquil11 Aug 11, 2026
42fc554
Simplify MI355X performance changelog
cquil11 Aug 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
163 changes: 106 additions & 57 deletions benchmarks/single_node/agentic/dsv4_fp4_mi355x_vllm_mtp.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,87 @@ mkdir -p "$RESULT_DIR"
SERVER_PID=""
ROUTER_PID=""
MOONCAKE_MASTER_PID=""
LMCACHE_PID=""

cleanup_agentic_services() {
local exit_code=$?
trap - EXIT INT TERM
set +e
stop_background_process_tree "$ROUTER_PID" "vLLM router"
stop_background_process_tree "$SERVER_PID" "vLLM server" 60
stop_background_process_tree "$MOONCAKE_MASTER_PID" "Mooncake master"
stop_background_process_tree "$LMCACHE_PID" "LMCache server"
exit "$exit_code"
}
trap cleanup_agentic_services EXIT
trap 'exit 130' INT
trap 'exit 143' TERM

# Mooncake does not publish a ROCm wheel. Build the pinned release once for
# each immutable OS/Python/ROCm tuple and reuse the shared cache on later jobs.
install_mooncake_rocm() {
local mooncake_tag="v0.3.11.post1"
local mooncake_src="/tmp/Mooncake-$mooncake_tag"
local mooncake_stage="/tmp/mooncake-stage-$mooncake_tag"
local build_jobs cache_root cache_key cache_archive cache_tmp
local engine_path os_version python_abi rocm_version

build_jobs=$(nproc)
if ((build_jobs > 32)); then
build_jobs=32
fi
os_version=$(. /etc/os-release && printf '%s-%s' "$ID" "$VERSION_ID")
python_abi=$(python3 -c 'import sys; print(f"cp{sys.version_info.major}{sys.version_info.minor}")')
rocm_version=$(sed -n '1p' /opt/rocm/.info/version 2>/dev/null || true)
if [[ -z "$rocm_version" ]]; then
rocm_version=$(hipconfig --version)
fi
rocm_version=${rocm_version//[^[:alnum:]._-]/_}
cache_root="${HF_HUB_CACHE:?HF_HUB_CACHE must be set}/inferencex/mooncake"
cache_key="${mooncake_tag}-${os_version}-${python_abi}-${rocm_version}-$(uname -m)-hip"
cache_archive="$cache_root/$cache_key.tar.gz"
mkdir -p "$cache_root"

apt-get update
DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
build-essential cmake git libasio-dev libboost-dev libcurl4-openssl-dev \
libgflags-dev libgoogle-glog-dev libibverbs-dev libjsoncpp-dev \
libnuma-dev libpython3-dev libssl-dev libunwind-dev liburing-dev \
libxxhash-dev libyaml-cpp-dev libzstd-dev ninja-build pybind11-dev

exec 9>"$cache_archive.lock"
flock -w 1800 9
if [[ -f "$cache_archive" ]] && ! tar -tzf "$cache_archive" >/dev/null 2>&1; then
rm -f "$cache_archive"
fi
if [[ ! -f "$cache_archive" ]]; then
echo "Building HIP Mooncake cache artifact: $cache_archive"
rm -rf "$mooncake_src" "$mooncake_stage"
git clone --depth 1 --branch "$mooncake_tag" --recurse-submodules \
--shallow-submodules https://github.com/kvcache-ai/Mooncake.git "$mooncake_src"
cmake -S "$mooncake_src/extern/yalantinglibs" \
-B "$mooncake_src/extern/yalantinglibs/build" \
-DBUILD_EXAMPLES=OFF -DBUILD_BENCHMARK=OFF -DBUILD_UNIT_TESTS=OFF
cmake --build "$mooncake_src/extern/yalantinglibs/build" -j "$build_jobs"
cmake --install "$mooncake_src/extern/yalantinglibs/build"
cmake -S "$mooncake_src" -B "$mooncake_src/build" -G Ninja \
-DCMAKE_BUILD_TYPE=Release -DUSE_CUDA=OFF -DUSE_HIP=ON \
-DWITH_EP=OFF -DWITH_STORE=ON -DWITH_STORE_RUST=OFF \
-DWITH_RUST_EXAMPLE=OFF -DBUILD_EXAMPLES=OFF -DBUILD_UNIT_TESTS=OFF
cmake --build "$mooncake_src/build" -j "$build_jobs"
mkdir -p "$mooncake_stage"
DESTDIR="$mooncake_stage" cmake --install "$mooncake_src/build"
cache_tmp=$(mktemp "$cache_root/$cache_key.tmp.XXXXXX")
tar -C "$mooncake_stage" -czf "$cache_tmp" .
mv -f "$cache_tmp" "$cache_archive"
else
echo "Using HIP Mooncake cache artifact: $cache_archive"
fi
tar -C / -xzf "$cache_archive"
engine_path=$(python3 -c 'import mooncake.engine; print(mooncake.engine.__file__)')
ldd "$engine_path" | grep -q 'libamdhip64.so'
exec 9>&-
}

OFFLOAD_ARGS=()

Expand Down Expand Up @@ -139,30 +220,13 @@ case "${KV_OFFLOAD_BACKEND:-}" in
;;
mooncake)
require_agentic_kv_offload_backend mooncake
# ---- Mooncake config ----------------------------------------------------------
# Embedded mode contributes one segment per GPU rank to a shared
# distributed store, so pre-divide the aggregate host-memory budget.
# distributed store, so pre-divide the generated aggregate budget.
PER_RANK_GB=$((TOTAL_CPU_DRAM_GB / TP))

#MOONCAKE_VERSION=0.3.11.post1
#apt-get update && apt-get install -y libcurl4 libibverbs1 rdma-core librdmacm1 libnuma1 liburing2
#agentic_pip_install --quiet --no-cache-dir --no-deps \
# --force-reinstall "mooncake-transfer-engine-non-cuda==$MOONCAKE_VERSION"

git clone https://github.com/kvcache-ai/Mooncake.git
cd Mooncake
bash dependencies.sh
mkdir build
cd build
cmake ..
make -j
sudo make install # optional, make it ready to be used by vLLM/SGLang
cd ..
cd ..

if ! python3 -c "from mooncake.store import MooncakeDistributedStore" >/dev/null 2>&1; then
install_mooncake_rocm
fi
python3 -c "from mooncake.store import MooncakeDistributedStore" >/dev/null
export INFERENCEX_MOONCAKE_MAX_TRANSFER_BATCH_KEYS=32
python3 "$(dirname "$0")/patch_vllm_mooncake_transfer_batches.py"

MOONCAKE_MASTER_PORT=$((PORT + 12000))
MOONCAKE_CONFIG_PATH="$RESULT_DIR/mooncake_config.json"
Expand All @@ -178,22 +242,18 @@ case "${KV_OFFLOAD_BACKEND:-}" in
"enable_offload": false
}
EOF
# (srok)
#"protocol": "rdma",
#"device_name": "mlx5_0",
#"local_buffer_size": "4GB",
# Mooncake v0.3.11.post1 emits its transfer polling loop at VLOG(1).
# Keep normal INFO diagnostics while suppressing that unbounded hot-loop output.
export GLOG_v=0
export MOONCAKE_CONFIG_PATH
export MC_ENABLE_DEST_DEVICE_AFFINITY=1
export PYTHONHASHSEED=0
export MC_SLICE_SIZE=1048576
# (srok)
#export MC_WORKERS_PER_CTX=4
export MC_WORKERS_PER_CTX=8

MOONCAKE_EVICTION_HIGH_WATERMARK_RATIO=0.80
MOONCAKE_EVICTION_RATIO=0.10
MOONCAKE_KV_LEASE_TTL=60s
#MOONCAKE_KV_LEASE_TTL=3600s

echo "Starting Mooncake master on port $MOONCAKE_MASTER_PORT..."
mooncake_master --port "$MOONCAKE_MASTER_PORT" \
Expand All @@ -219,30 +279,6 @@ EOF
lmcache)
require_agentic_kv_offload_backend lmcache
# ---- Lmcache config ----------------------------------------------------------
LMCACHE_PID=""

cleanup_lmcache_server() {
if [[ -n "$LMCACHE_PID" ]] && kill -0 "$LMCACHE_PID" 2>/dev/null; then
kill "$LMCACHE_PID" 2>/dev/null || true
wait "$LMCACHE_PID" 2>/dev/null || true
fi
}

trap cleanup_lmcache_server EXIT

cleanup_agentic_services() {
local exit_code=$?
trap - EXIT INT TERM
set +e
stop_background_process_tree "$ROUTER_PID" "vLLM router"
stop_background_process_tree "$SERVER_PID" "vLLM server" 60
stop_background_process_tree "$MOONCAKE_MASTER_PID" "Mooncake master"
exit "$exit_code"
}
trap cleanup_agentic_services EXIT
trap 'exit 130' INT
trap 'exit 143' TERM

wait_for_lmcache_ready() {
{ set +x; } 2>/dev/null
local attempts="${LMCACHE_READY_ATTEMPTS:-120}"
Expand Down Expand Up @@ -379,6 +415,9 @@ MAX_NUM_SEQS=$((2 * CONC))
# DeepSeek-V4-Pro ships a native MTP head. AgentX throughput pins its
# three-token draft to the committed thinking-on golden acceptance length;
# eval-only runs use real target verification so accuracy remains meaningful.
# The isolated scheduler, MTP-depth, graph-mode, collective, and breakable-
# graph probes did not produce a clean improvement. Keep the verified
# K=3/8K/FULL_DECODE_ONLY baseline while expanding topology coverage.
NUM_SPEC_TOKENS=3
SYNTHETIC_ACCEPT_LEN=2.49
if [ "${EVAL_ONLY:-false}" = "true" ]; then
Expand All @@ -387,14 +426,21 @@ else
SPEC_CONFIG="{\"method\": \"mtp\", \"num_speculative_tokens\": $NUM_SPEC_TOKENS, \"rejection_sample_method\": \"synthetic\", \"synthetic_acceptance_length\": $SYNTHETIC_ACCEPT_LEN}"
fi

# DEP4 replicates the attention weights across four DP ranks. At the common
# 0.90 setting vLLM leaves only 12.06--12.50 GiB for KV on MI355X, below the
# 14.88 GiB required to admit one native 1,048,576-token request. Give only
# this topology the extra headroom it needs; established TP/TEP/DEP8 results
# retain their original memory fraction.
GPU_MEM_UTIL=0.9
if [ "$DP_ATTENTION" = "true" ] && [ "$TP" -eq 4 ]; then
GPU_MEM_UTIL=0.92
fi

echo "Starting vllm server..."
set -x
export VLLM_ROCM_USE_AITER=1
#export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4
export VLLM_ROCM_USE_AITER_MOE=1

sleep 180

{ set +x; } 2>/dev/null
VLLM_CMD=(
vllm serve "$MODEL_PATH" --served-model-name "$MODEL"
Expand All @@ -406,15 +452,18 @@ VLLM_CMD=(
--kv-cache-dtype fp8
"${PARALLEL_ARGS[@]}"
"${EP_ARGS[@]}"
--gpu-memory-utilization 0.8
--gpu-memory-utilization "$GPU_MEM_UTIL"
--block-size 256
--max-num-batched-tokens 8192
--moe-backend aiter
--compilation-config '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE"}'
--compilation-config '{"mode":3,"cudagraph_mode":"FULL_DECODE_ONLY"}'
--speculative-config "$SPEC_CONFIG"
--tokenizer-mode deepseek_v4
--tool-call-parser deepseek_v4
--reasoning-parser deepseek_v4
--enable-auto-tool-choice
--enable-prefix-caching
--enable-prompt-tokens-details
--no-disable-hybrid-kv-cache-manager
--max-num-seqs "$MAX_NUM_SEQS"
"${OFFLOAD_ARGS[@]}"
Comment on lines 452 to 469

@claude claude Bot Aug 9, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Switching --compilation-config to FULL_DECODE_ONLY (line 411-412) without also passing a capture-size fix-up means vLLM's adjust_cudagraph_sizes_for_spec_decode() rounds captured decode-batch sizes up to multiples of (1+NUM_SPEC_TOKENS)=4 and dedups, collapsing capture coverage to MAX_NUM_SEQS/4 seqs — the largest decode batches at the new high-concurrency cells (c40/c56/c224, MAX_NUM_SEQS up to 448) silently fall back to eager execution. The sibling B200/B300 MTP recipes in this same directory (dsv4_fp4_b200_vllm_mtp.sh:239-285, dsv4_fp4_b300_vllm_mtp.sh:261-286) hit this exact MTP+FULL_DECODE_ONLY interaction and fix it with --max-cudagraph-capture-size / an explicit cudagraph_capture_sizes list sized to MAX_NUM_SEQS*(1+NUM_SPEC_TOKENS); this script needs the same treatment.

Extended reasoning...

The bug: this PR switches --compilation-config from {"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE"} to {"mode":3,"cudagraph_mode":"FULL_DECODE_ONLY"} (dsv4_fp4_mi355x_vllm_mtp.sh:411-412) but does not carry over the capture-size mitigation that this exact mode+MTP combination requires. Every row in this recipe's search space runs spec-decoding: mtp with NUM_SPEC_TOKENS=3, so TOKENS_PER_SEQ = 1 + NUM_SPEC_TOKENS = 4.

Why it triggers: vLLM auto-derives its CUDA-graph capture-size ladder from --max-num-seqs assuming one token per sequence. When speculative decoding is active and cudagraph_mode is FULL_DECODE_ONLY, adjust_cudagraph_sizes_for_spec_decode() rounds each auto-derived size up to the nearest multiple of TOKENS_PER_SEQ and dedups the resulting list. With no explicit override, this collapses the usable capture range down to roughly MAX_NUM_SEQS / TOKENS_PER_SEQ seqs — i.e. MAX_NUM_SEQS / 4 here. Any decode batch larger than that silently falls back to eager execution instead of using a captured graph.

Why the existing code doesn't prevent it: the previous mode, FULL_AND_PIECEWISE, uses piecewise CUDA graphs for the portion outside the captured range, so this collapse was masked. Switching to FULL_DECODE_ONLY removes that fallback path, and this PR's --compilation-config JSON only sets mode and cudagraph_mode — no --max-cudagraph-capture-size flag, and no explicit cudagraph_capture_sizes list. Both sibling recipes in the same directory, dsv4_fp4_b200_vllm_mtp.sh and dsv4_fp4_b300_vllm_mtp.sh, hit this identical interaction (same NUM_SPEC_TOKENS=3) and each carry an in-repo comment describing it verbatim ("vLLM's FULL_DECODE_ONLY ladder tops out at MAX_NUM_SEQS/(1+N) seqs and the largest decode batches fall back to eager"). B200 fixes it with MAX_CUDAGRAPH_CAPTURE_SIZE=0 passed via --max-cudagraph-capture-size; B300 builds an explicit cudagraph_capture_sizes list of every multiple of TOKENS_PER_SEQ up to MAX_NUM_SEQS*TOKENS_PER_SEQ. Neither mitigation is present in the MI355X script under review.

Step-by-step proof (TP8+EP8, DP-attn cell at c224): MAX_NUM_SEQS = 2*CONC = 448. With NUM_SPEC_TOKENS=3, TOKENS_PER_SEQ=4. vLLM's default ladder for max_num_seqs=448 includes sizes up through 448 (in seqs, i.e. tokens under the 1-token/seq assumption). adjust_cudagraph_sizes_for_spec_decode() rounds every size up to the next multiple of 4 and dedups — so a would-be capture point at, say, 400 seqs rounds to 400 tokens = 100 seqs worth of MTP-verified batch, and no capture point exists above roughly 448/4 = 112 seqs. Any decode batch above ~112 seqs at this cell (which is exactly the new high-concurrency point this PR adds) runs eager instead of a captured graph, degrading measured decode throughput at precisely the topology (DP8+EP8, up to c224) that the PR states is 'the intended throughput topology.'

Impact: this is a pure benchmarking recipe whose only deliverable is throughput numbers; an unintended eager fallback at the largest, newest concurrency cells silently produces degraded/misleading numbers rather than a crash, which is worse for a benchmark refresh whose stated purpose is establishing an accurate frontier.

Fix: mirror either sibling's mitigation — add NUM_SPEC_TOKENS-derived TOKENS_PER_SEQ=$((1 + NUM_SPEC_TOKENS)), compute MAX_CUDAGRAPH_CAPTURE_SIZE=$((MAX_NUM_SEQS * TOKENS_PER_SEQ)), and pass it via --max-cudagraph-capture-size (matching the B200 recipe's simpler approach) alongside the existing --compilation-config flag.

Expand Down
28 changes: 17 additions & 11 deletions configs/amd-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1227,13 +1227,13 @@ qwen3.5-fp8-mi355x-sglang-agentic-hicache:
- { tp: 8, ep: 1, kv-offloading: none, conc-list: [1, 2, 4, 8, 16, 32] }
- { tp: 8, ep: 1, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [16, 32, 48, 64] }

# MTP twin of dsv4-fp4-mi355x-vllm-agentic. The topologies, KV-offload backends,
# and concurrency points are identical; speculative decoding is enabled on the
# current immutable ROCm nightly because the base config's retired nightly tag
# is no longer available from Docker Hub. Throughput uses the three-token golden
# synthetic acceptance length, while eval-only runs retain real verification.
# DeepSeek-V4-Pro FP4 MTP collection sweep for MI355X. Keep the verified
# GPU-resident frontier, add denser c52/c60/c68 TEP guards, collect the same
# points with pinned Mooncake DRAM offload, and retain representative DEP8 and
# DEP4 cells even when dominated. The fast pass is a functionality gate;
# Pareto optimality is deliberately not required for inclusion.
dsv4-fp4-mi355x-vllm-agentic-mtp:
image: vllm/vllm-openai-rocm:nightly-821717118fc26667dd474b9b0ab81d29259dfc5c
image: vllm/vllm-openai-rocm:v0.27.0
model: deepseek-ai/DeepSeek-V4-Pro
model-prefix: dsv4
runner: cluster:mi355x-amds
Comment on lines 1227 to 1239

@claude claude Bot Aug 9, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 AGENTS.md requires bilingual PR titles/descriptions (' / <中文标题>' plus a mirroring '## 中文说明' section), but this PR's title and description are English-only. Please add the Chinese title suffix and a '## 中文说明' section mirroring the Summary/Search space/Validation content.

Extended reasoning...

AGENTS.md line 7 states an explicit, named repository convention: "PR and GitHub-issue titles & descriptions must be bilingual — include a Simplified Chinese version in addition to English. Title format: <English title> / <中文标题>. In the PR/issue body, follow the English content with its Chinese translation (e.g. a ## 中文说明 section mirroring the summary...)."

This PR's title, perf(agentx): refresh DeepSeek V4 MTP frontier on MI355X, has no / <中文标题> suffix, and the description contains only English ## Summary, ## Search space, and ## Validation sections — there is no ## 中文说明 (or equivalent) section mirroring that content in Chinese.

Since this is a named, verbatim rule in the repo's own contributor guidance rather than a subjective style preference, it's a legitimate finding even though it concerns PR metadata rather than a specific line in configs/amd-master.yaml (the anchor line is arbitrary — the metadata field itself is what's non-compliant). No code in the diff prevents or enforces this; it's purely a PR-authoring step that was skipped.

Proof: compare the PR title/body against the AGENTS.md template. The template requires <English title> / <中文标题>; the actual title is perf(agentx): refresh DeepSeek V4 MTP frontier on MI355X with nothing after it. The template requires the body's English content to be followed by "its Chinese translation (e.g. a ## 中文说明 section mirroring the summary)"; the actual body ends after ## Validation with no Chinese section at all.

Fix: append / <中文标题> to the PR title with a Simplified Chinese translation of "refresh DeepSeek V4 MTP frontier on MI355X", and add a ## 中文说明 section at the end of the description that mirrors the Summary, Search space, and Validation content in Chinese.

This is metadata-only and has no bearing on the correctness of the config or benchmark script changes, so it should not block merging — it's a nit that the author (or a bot) can fix by editing the PR title/description directly.

Expand All @@ -1244,11 +1244,17 @@ dsv4-fp4-mi355x-vllm-agentic-mtp:
agentic-coding:
- dram-utilization: 0.60
search-space:
- { tp: 8, ep: 1, dp-attn: false, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 4, 8, 16, 32, 40, 48] }
- { tp: 8, ep: 1, dp-attn: true, kv-offloading: none, spec-decoding: mtp, conc-list: [64], router: { name: vllm-router, version: "0.1.14" } }
# LMCache invalid-block recovery currently assumes one KV-cache group,
# while MTP creates two. Restore these points after the upstream hybrid
# KV recovery fix lands: https://github.com/vllm-project/vllm/pull/45497
- { tp: 8, ep: 1, dp-attn: false, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 2, 4, 6, 8, 12] }
- { tp: 8, ep: 8, dp-attn: false, kv-offloading: none, spec-decoding: mtp, conc-list: [16, 24, 32, 40, 48, 52, 56, 60, 64, 68] }
- { tp: 8, ep: 8, dp-attn: false, kv-offloading: dram, kv-offload-backend: { name: mooncake, version: "0.3.11.post1" }, spec-decoding: mtp, conc-list: [52, 60, 68] }
- { tp: 8, ep: 8, dp-attn: true, kv-offloading: none, spec-decoding: mtp, conc-list: [64, 128, 192, 224], router: { name: vllm-router, version: "0.1.14" } }
- { tp: 4, ep: 4, dp-attn: true, kv-offloading: none, spec-decoding: mtp, conc-list: [4, 8], router: { name: vllm-router, version: "0.1.14" } }
# DEP4 previously had only 12.09 GiB/GPU available for the native
# 1,048,576-token context versus a 14.88 GiB minimum. Retain two cells in
# the fast gate to reconfirm whether the current base can now initialize;
# do not weaken the AgentX context limit to make them pass.
# The stable-v0.27 Mooncake c52 gate completed its full 575-request
# warmup and measured profile without hybrid-KV or request errors.

dsr1-fp4-mi355x-sglang-disagg-mtp:
image: lmsysorg/sglang-rocm:v0.5.12-rocm720-mi35x-20260519
Expand Down
11 changes: 11 additions & 0 deletions perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5726,6 +5726,17 @@
- "Use supported header-based Dynamo session routing with the in-repo AIPerf build."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2520

- config-keys:
- dsv4-fp4-mi355x-vllm-agentic-mtp
scenario-type:
- agentic-coding
description:
- "Refresh the MI355X DeepSeek-V4-Pro MTP submission on stable vLLM ROCm v0.27.0; raise GPU-memory utilization from 0.80 to 0.90 and use 256-token blocks, an 8192-token scheduler budget, and FULL_DECODE_ONLY graphs."
- "Keep the measured MTP operating point at three draft tokens with golden synthetic acceptance length 2.49."
- "Expand collection to TP8 c1-c12, TP8/EP8 c16-c68, Mooncake DRAM offload c52/c60/c68, DEP8 c64-c224, and DEP4 c4/c8, including dominated points requested for comparison."
- "Use 0.92 GPU-memory utilization only for DEP4 so the native 1,048,576-token context fits; all other points retain 0.90."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2540

- config-keys:
- dsv4-fp4-gb300-dynamo-sglang-agentic-agg
- dsv4-fp4-gb300-dynamo-sglang-agentic-disagg
Expand Down