From 878c3a0f2e0ce0ba0b8b891fa49d33d1afe16973 Mon Sep 17 00:00:00 2001 From: adibarra <93070681+adibarra@users.noreply.github.com> Date: Mon, 10 Aug 2026 11:07:39 -0500 Subject: [PATCH 1/3] docs: improve agent documentation and prose MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add a bilingual low-context documentation hub, canonical agent skills, and the performance changelog policy. Humanize Markdown prose without changing technical behavior. 中文：新增双语低上下文文档入口、规范化 Agent 技能，并明确性能变更日志要求。在不改变技术行为的前提下优化 Markdown 文案。 [skip-sweep] --- .agents/skills/debug-agentx-runs/SKILL.md | 48 +- .../skills/debug-runs/SKILL.md | 98 ++-- .agents/skills/kimi-k3-b200-agentx/SKILL.md | 166 +++++++ .../skills/tune-agentx-submission/SKILL.md | 70 +++ .claude/commands/add-model-hardware.md | 72 +-- .../clean-amd-mi355-runner-root-files.md | 18 +- .claude/commands/debug-mi300-enroot-pyxis.md | 27 +- .claude/commands/find-mergeable-claude-prs.md | 2 +- .claude/commands/fix-klaud-cron-prs.md | 14 +- .claude/commands/klaud-pr-status-html.md | 20 +- .claude/commands/list-claude-pr-status.md | 12 +- .claude/commands/merge-prs.md | 2 +- .claude/commands/nuke.md | 14 +- .claude/commands/recover-failed-ingest.md | 42 +- .claude/skills/debug-agentx-runs | 1 + .claude/skills/debug-runs | 1 + .claude/skills/kimi-k3-b200-agentx | 1 + .claude/skills/tune-agentx-submission | 1 + .github/AGENT_OPERATIONS.md | 16 +- .../pull_request_template.md | 5 +- .github/codeowner-signoff-verify-prompt.md | 188 +++---- .github/workflows/README.md | 24 +- .github/workflows/claude.yml | 4 +- AGENTS.md | 180 +------ CLAUDE.md | 4 +- CONTRIBUTING.md | 24 +- CONTRIBUTING_zh.md | 22 +- KLAUD_DEBUG.md | 46 +- MODELS.md | 58 +-- MODELS_zh.md | 60 +-- README.md | 3 +- README_zh.md | 5 +- benchmarks/multi_node/llm-d/README.md | 6 +- benchmarks/single_node/agentic/README.md | 8 +- configs/CONFIGS.md | 12 +- docs/DOCUMENTATION_PLAN.md | 126 +++++ docs/DOCUMENTATION_PLAN_zh.md | 126 +++++ docs/PR_REVIEW_CHECKLIST.md | 4 +- docs/PR_REVIEW_CHECKLIST_zh.md | 6 +- docs/agent-guide.md | 33 ++ docs/agent-guide_zh.md | 33 ++ docs/architecture.md | 347 +++++++++++++ docs/architecture_zh.md | 347 +++++++++++++ docs/ci-procedures.md | 469 ++++++++++++++++++ docs/ci-procedures_zh.md | 469 ++++++++++++++++++ docs/configuration-procedures.md | 275 ++++++++++ docs/configuration-procedures_zh.md | 275 ++++++++++ docs/documentation-procedures.md | 322 ++++++++++++ docs/documentation-procedures_zh.md | 322 ++++++++++++ docs/eval-agentx-procedures.md | 335 +++++++++++++ docs/eval-agentx-procedures_zh.md | 335 +++++++++++++ docs/index.md | 51 ++ docs/index_zh.md | 51 ++ docs/procedures.md | 60 +++ docs/procedures_zh.md | 60 +++ docs/recovery-results-procedures.md | 427 ++++++++++++++++ docs/recovery-results-procedures_zh.md | 427 ++++++++++++++++ docs/results-and-ingestion.md | 387 +++++++++++++++ docs/results-and-ingestion_zh.md | 387 +++++++++++++++ docs/testing.md | 163 ++++++ docs/testing_zh.md | 163 ++++++ docs/troubleshooting.md | 171 +++++++ docs/troubleshooting_zh.md | 171 +++++++ experimental/CollectiveX/README.md | 110 ++-- experimental/CollectiveX/docs/methodology.md | 311 ++++++------ experimental/operatorx/CLUSTERS.md | 34 +- experimental/operatorx/README.md | 6 +- .../scripts/inferencex_testlist/README.md | 14 +- golden_al_distribution/README.md | 18 +- golden_al_distribution/README_zh.md | 16 +- utils/evals/EVALS.md | 59 +-- utils/runner_setup/RUNNER_SETUP.md | 92 ++-- 72 files changed, 7355 insertions(+), 921 deletions(-) rename {.claude => .agents}/skills/debug-runs/SKILL.md (55%) create mode 100644 .agents/skills/kimi-k3-b200-agentx/SKILL.md create mode 100644 .agents/skills/tune-agentx-submission/SKILL.md create mode 120000 .claude/skills/debug-agentx-runs create mode 120000 .claude/skills/debug-runs create mode 120000 .claude/skills/kimi-k3-b200-agentx create mode 120000 .claude/skills/tune-agentx-submission create mode 100644 docs/DOCUMENTATION_PLAN.md create mode 100644 docs/DOCUMENTATION_PLAN_zh.md create mode 100644 docs/agent-guide.md create mode 100644 docs/agent-guide_zh.md create mode 100644 docs/architecture.md create mode 100644 docs/architecture_zh.md create mode 100644 docs/ci-procedures.md create mode 100644 docs/ci-procedures_zh.md create mode 100644 docs/configuration-procedures.md create mode 100644 docs/configuration-procedures_zh.md create mode 100644 docs/documentation-procedures.md create mode 100644 docs/documentation-procedures_zh.md create mode 100644 docs/eval-agentx-procedures.md create mode 100644 docs/eval-agentx-procedures_zh.md create mode 100644 docs/index.md create mode 100644 docs/index_zh.md create mode 100644 docs/procedures.md create mode 100644 docs/procedures_zh.md create mode 100644 docs/recovery-results-procedures.md create mode 100644 docs/recovery-results-procedures_zh.md create mode 100644 docs/results-and-ingestion.md create mode 100644 docs/results-and-ingestion_zh.md create mode 100644 docs/testing.md create mode 100644 docs/testing_zh.md create mode 100644 docs/troubleshooting.md create mode 100644 docs/troubleshooting_zh.md diff --git a/.agents/skills/debug-agentx-runs/SKILL.md b/.agents/skills/debug-agentx-runs/SKILL.md index 4aaa7a1c93..a30cad52fe 100644 --- a/.agents/skills/debug-agentx-runs/SKILL.md +++ b/.agents/skills/debug-agentx-runs/SKILL.md @@ -6,13 +6,13 @@ description: Debug long-running AgentX benchmark jobs from live cluster logs and # Debug AgentX runs from the cluster AgentX runs have long model-load, warmup, drain, and profiling phases. Treat GitHub Actions -as the orchestration and final-status view; use the cluster as the primary source of live +as the orchestration and final-status view. Use the cluster as the primary source of live diagnostic signal. Compose with `$debug-runs` when it is available for the general full-sweep and reproduction workflow. Cluster login addresses, users, jumpboxes, runner directories, and storage paths belong in the access-controlled InferenceX Clusters Slack canvas, not this repository. Ask the user -for the canvas link or the intended SSH alias when access is not already configured; never +for the canvas link or the intended SSH alias when access is not already configured. Never guess or publish private infrastructure details. Read-only SSH inspection is allowed. Before editing cluster files, restarting or killing processes, draining nodes, cancelling Slurm jobs, or otherwise mutating shared infrastructure, stop and ask for approval unless @@ -55,7 +55,7 @@ scontrol show job -o | tr " " "\n" | grep -E "^(JobId|JobState|RunTime|TimeLimit|NodeList|WorkDir)=" ``` -Derive paths from `WorkDir`; do not guess or hardcode a cluster path. For an srt-slurm +Derive paths from `WorkDir`. Do not guess or hardcode a cluster path. For an srt-slurm job, logs normally live under: ```text @@ -129,11 +129,11 @@ curl -fsS "" | Prefer the metric names actually exposed by the running image instead of assuming a specific vLLM, SGLang, or Dynamo version. Track at least: -- active/running and waiting requests; -- KV-cache usage and prefix-cache hit rate; -- input and output token rates; -- completed, cancelled, and errored requests; -- frontend active requests and per-worker routing balance; +- active/running and waiting requests +- KV-cache usage and prefix-cache hit rate +- input and output token rates +- completed, cancelled, and errored requests +- frontend active requests and per-worker routing balance - KV-transfer activity and failures for disaggregated runs. Interpret trends, not one scrape: @@ -164,10 +164,10 @@ calculate the nominal end from the `Phase profiling started` timestamp plus the duration. Then allow a few minutes for cutoff drain, aggregation, staging, and artifact upload. State separately: -1. phase elapsed and remaining time; -2. whether logs are still updating; -3. errors observed; -4. expected benchmark completion; +1. phase elapsed and remaining time +2. whether logs are still updating +3. errors observed +4. expected benchmark completion 5. expected GitHub job completion. For warmup drains, report returned, sent, in-flight, errors, elapsed time, and the @@ -179,10 +179,10 @@ when validating a timeout change. Do not burn hours waiting for final JSON when direct signals already disqualify a config. Use server logs and metrics to make the decision early. Examples include: -- deterministic OOM, NCCL/RCCL failure, parser crash, or missing worker; -- no forward progress across repeated samples; -- persistent KV saturation and queue growth with unusable latency; -- throughput that has plateaued while added concurrency only increases latency; +- deterministic OOM, NCCL/RCCL failure, parser crash, or missing worker +- no forward progress across repeated samples +- persistent KV saturation and queue growth with unusable latency +- throughput that has plateaued while added concurrency only increases latency - a disaggregated pool or metrics source that never registered. Before cancellation, capture the relevant log lines, timestamps, topology, and metric @@ -196,7 +196,7 @@ gh run cancel --repo SemiAnalysisAI/InferenceX ``` Use `scancel` or direct process termination only with explicit approval and a concrete -reason; doing so can bypass cleanup or strand shared-cluster state. Never kill only the +reason. Doing so can bypass cleanup or strand shared-cluster state. Never kill only the backend and leave the workflow silently occupying a runner. After a recipe/config fix, use a targeted e2e dispatch for fast feedback. Reserve another @@ -206,13 +206,13 @@ official full sweep for the candidate that has passed direct cluster inspection. For each active point, report: -- GitHub job and Slurm job links/IDs; -- aggregate versus disaggregated topology; -- phase, elapsed time, remaining time, and last log update; -- log files and metrics sources inspected; -- request, queue, KV-cache, cache-hit, and token-rate trends; -- errors and the likely root cause; -- whether to continue, short-circuit, or rerun; +- GitHub job and Slurm job links/IDs +- aggregate versus disaggregated topology +- phase, elapsed time, remaining time, and last log update +- log files and metrics sources inspected +- request, queue, KV-cache, cache-hit, and token-rate trends +- errors and the likely root cause +- whether to continue, short-circuit, or rerun - which points are fully green versus merely healthy in progress. Do not call a run successful until GitHub has accepted its result artifacts and the diff --git a/.claude/skills/debug-runs/SKILL.md b/.agents/skills/debug-runs/SKILL.md similarity index 55% rename from .claude/skills/debug-runs/SKILL.md rename to .agents/skills/debug-runs/SKILL.md index 898c117381..cae671aba1 100644 --- a/.claude/skills/debug-runs/SKILL.md +++ b/.agents/skills/debug-runs/SKILL.md @@ -1,16 +1,16 @@ --- name: debug-runs -description: Drive a full-sweep benchmark config to green with a tight feedback loop — trigger/monitor the sweep, root-cause failures, and (for fast iteration) SSH onto the runner's cluster to reproduce a single config directly on the node instead of waiting for full CI. Use when bringing up a new model/precision/SKU recipe, debugging a failing or flaky sweep, debugging node-level issues, or gathering context on a cluster before a run. Cluster access details are NOT in this repo — read them from the shared InferenceX Clusters canvas. +description: Drive a full-sweep benchmark config to green with a tight feedback loop by triggering and monitoring the sweep, finding the root causes of failures, and, for fast iteration, using SSH to reproduce a single config directly on the runner's cluster instead of waiting for full CI. Use when bringing up a new model, precision, or SKU recipe, debugging a failing or flaky sweep, debugging node-level issues, or gathering context on a cluster before a run. Cluster access details are not in this repo and must be read from the shared InferenceX Clusters canvas. --- # Debug runs (tight feedback loop) -Use this when the goal is to get a **full-sweep config passing** — and you want to verify -on the actual nodes first (tighter loop than the full CI dispatch cycle), or to debug -node/infra issues, or just to gather context on a cluster. +Use this when the goal is to get a **full-sweep config passing** and you want to verify it +on the actual nodes first for a tighter loop than the full CI dispatch cycle, debug +node or infrastructure issues, or gather context on a cluster. -This composes with the other skills: use **`/nuke`** (or `/add-model-hardware`) to create -the PR(s) with the image bump + perf-changelog entry + `full-sweep-fail-fast` label; this +This composes with the other skills. Use **`/nuke`** (or `/add-model-hardware`) to create +the PR(s) with the image bump, perf-changelog entry, and `full-sweep-fail-fast` label. This skill is the **monitor → root-cause → fix → re-verify → merge-gate** loop that follows. ## Cluster access — read it from the canvas, never hardcode it @@ -33,7 +33,7 @@ and the benchmark command. If this is a **fork** (i.e. not the SemiAnalysis upstream, where the canvas won't apply), ask the user to replace this skill with their own fork's runner/cluster access information. -- A SemiAnalysis operator may also have these as `~/.ssh/config` aliases — prefer those if +- A SemiAnalysis operator may also have these as `~/.ssh/config` aliases. Prefer those if present. ## Inputs @@ -46,10 +46,10 @@ and the benchmark command. ### 1. Trigger (or reuse) the sweep -A PR's sweep is kicked by labels (the `/sweep` comment trigger was removed — use the label): +A PR's sweep is kicked by labels. The `/sweep` comment trigger was removed, so use the label: -- **`full-sweep-fail-fast`** — full sweep that bails on first failure per matrix (faster signal while debugging; **strongly recommended default**, and what `/nuke` attaches). -- **`full-sweep-enabled`** — full GPU sweep that runs every job to completion despite failures; use only when a flaky job killing its matrix's in-flight results is unacceptable. +- **`full-sweep-fail-fast`** runs the full sweep and bails on the first failure per matrix for faster feedback while debugging. It is the **strongly recommended default** and the label that `/nuke` attaches. +- **`full-sweep-enabled`** runs the full GPU sweep and lets every job complete despite failures. Use it only when a flaky job killing its matrix's in-flight results is unacceptable. - To re-trigger a sweep without a new commit, remove and re-add the sweep label. For a **single config** (tightest CI loop, skips the rest of the matrix), dispatch e2e directly: @@ -58,11 +58,11 @@ For a **single config** (tightest CI loop, skips the rest of the matrix), dispat gh workflow run e2e-tests.yml -f generate-cli-command="test-config --config-key --config-file " -f test-name="debug " ``` -(`generate-cli-command` is the required input; `--target` is NOT a real arg.) +(`generate-cli-command` is the required input. `--target` is NOT a real arg.) ### 2. Monitor continuously -Find the run, then watch it — don't poll by hand. Prefer the **Monitor** tool with a +Find the run, then watch it instead of polling by hand. Prefer the **Monitor** tool with a filter that catches both progress and failure signatures so silence never reads as success. ```bash @@ -73,7 +73,7 @@ gh run watch "$RUN_ID" --repo SemiAnalysisAI/InferenceX --interval 30 ``` When monitoring several runs at once (e.g. 4 SKUs), track them by `databaseId` and report -each as it lands — never declare success from absence of output. +each as it lands. Never declare success from absence of output. ### 3. Root-cause a failure @@ -90,25 +90,25 @@ in one or two sentences before changing anything. ### 4. Tight loop: reproduce on the node directly -This is the point of the skill — instead of re-dispatching CI for every hypothesis, get on +This is the point of the skill. Instead of re-dispatching CI for every hypothesis, get on the box and reproduce the **single** failing config. -Why this is tighter: under e2e / the matrix, **each concurrency / config runs against its -own freshly-spun engine** (a new server per matrix job). On the node you can spin up a -**single** server once and fire many requests / sweep multiple concurrencies against it — -far faster iteration when you're probing behavior or tuning, since you skip a fresh model -load per data point. +Why this is tighter: in e2e or full-matrix runs, **each matrix point runs against its own +freshly spun engine** (a new server per matrix job). On the node you can spin up a +**single** server once and fire many requests or sweep multiple concurrencies against it. +This makes iteration much faster when you're probing behavior or tuning because you skip a +fresh model load per data point. Steps: -1. Look up the cluster's access + which node ran the failing job (from the job name / runner - name) in the canvas; SSH in (use `ssh -A` when a jumpbox/agent-forwarding is involved). -2. Reproduce the exact benchmark the launcher runs — read `runners/launch_.sh` for - the image, container mounts, and the `benchmarks/single_node/<...>.sh` command + env - (`IMAGE`, `TP`, `PRECISION`, `EXP_NAME`, `SPEC_DECODING`, …). On Slurm clusters that's a - `salloc`/`srun` with the squash image; on the **bare-metal `-tw` pools it's `docker run`** - on the node directly (no `srun`). -3. **Always diff against a working node / working SKU** for reference — most node failures +1. Use the job or runner name to identify the node. Look up that cluster's access details in + the canvas, then SSH in with `ssh -A` when a jumpbox or agent forwarding is involved. +2. Reproduce the exact benchmark the launcher runs. Read `runners/launch_.sh` for + the image, container mounts, and the `benchmarks/single_node/<...>.sh` command and env + (`IMAGE`, `TP`, `PRECISION`, `EXP_NAME`, `SPEC_DECODING`, …). On Slurm clusters, use + `salloc` or `srun` with the squash image. On the **bare-metal `-tw` pools, use `docker run`** + on the node directly without `srun`. +3. **Always diff against a working node or working SKU** for reference. Most node failures are environment drift (driver, ROCm/CUDA, missing mount, stale squash image), not code. 4. Iterate on the node until the single config passes, then push the fix and re-run CI. @@ -123,30 +123,30 @@ enroot list -f # find the running container's PID enroot exec bash # drop into the container ``` -(On the bare-metal `-tw` pools there's no Slurm/enroot — use `docker ps` + `docker exec -it bash`.) +(On the bare-metal `-tw` pools, there is no Slurm/enroot. Use `docker ps` and `docker exec -it bash`.) -**Node-level fixes are in scope** when you have operator access (e.g. AMD nodes where you -have sudo) — but **ask the user before executing any of them** (see guardrail below). The -kinds of fixes that are on the table: bringing a node's environment in line with the -working reference, and — if one or two nodes are unrecoverable — **draining them** or +**Node-level fixes are in scope** when you have operator access, such as on AMD nodes where +you have sudo, but **ask the user before executing any of them** (see guardrail below). The +kinds of fixes that are on the table include bringing a node's environment in line with the +working reference and, if one or two nodes are unrecoverable, **draining them** or explicitly **ignoring them in the run script** rather than blocking the whole sweep. Note any such change in the report. -> Guardrail — ask before changing infra. SSHing in to **read/investigate** (logs, +> Guardrail. Ask before changing infra. SSHing in to **read/investigate** (logs, > `rocm-smi`/`nvidia-smi`, `sinfo`/`squeue`, `df`, env, config inspection) is fine. But -> before making **any actual change on the cluster** — installing/updating anything, -> editing configs or files, restarting/killing processes, draining or ignoring nodes, -> changing the run script, or anything else that mutates node/shared state — **stop and ask -> the user for permission first**, describing exactly what you intend to run. Don't assume -> standing authorization just because you have sudo or operator access. +> before making **any actual change on the cluster**, including installing or updating +> anything, editing configs or files, restarting or killing processes, draining or ignoring +> nodes, changing the run script, or anything else that mutates node or shared state, +> **stop and ask the user for permission first**, describing exactly what you intend to run. +> Do not assume standing authorization just because you have sudo or operator access. > -> Also do **not** apply hacky engine-side (e.g. vLLM) workarounds to force a pass — prefer -> recipe fixes and (once approved) node-environment fixes that match a working reference. +> Also do **not** apply hacky engine-side (e.g. vLLM) workarounds to force a pass. Prefer +> recipe fixes and, once approved, node-environment fixes that match a working reference. ### 5. Flakes: rerun, don't relaunch If a job flaked (infra, transient network, runner pickup) rather than a real failure, rerun -just the failed jobs on the existing run — don't dispatch a fresh sweep: +just the failed jobs on the existing run. Don't dispatch a fresh sweep: ```bash gh run rerun "$RUN_ID" --repo SemiAnalysisAI/InferenceX --failed @@ -154,22 +154,22 @@ gh run rerun "$RUN_ID" --repo SemiAnalysisAI/InferenceX --failed ### 6. Report results — do NOT merge -**Never merge.** Merging is the user's call — only merge if the user **explicitly** tells -you to in this session. Even when everything looks perfect, stop and report; do not +**Never merge.** Merging is the user's call. Only merge if the user **explicitly** tells +you to in this session. Even when everything looks perfect, stop and report. Do not admin-merge on your own judgment. Report the two things the user will decide on: -1. **Sweep status** — is it 100% of full-sweep jobs passing (green), or fail-fast-truncated / partial? +1. **Sweep status.** Is it 100% of full-sweep jobs passing (green), or fail-fast-truncated or partial? 2. **Perf delta** vs the most recent official `main` run for that SKU. Compare against the - latest main results — e.g. on inferencex.semianalysis.com + latest main results, e.g. on inferencex.semianalysis.com (`https://inferencex.semianalysis.com/inference?...&i_active=_`) or the stored results for that SKU's last main `run-id`. -Present green-ness + the perf comparison, then **wait for the user** to decide whether to merge. +Present green-ness and the perf comparison, then **wait for the user** to decide whether to merge. ## Final report -Per config-key: final state (green / failing / flaky), root cause(s) found, node-level -changes made (and any nodes drained/ignored), and the perf delta vs main. Link the run(s) -and PR(s). End by asking the user whether to merge — do not merge yourself. +Per config-key, give the final state (green, failing, or flaky), the root cause(s) found, +node-level changes made (and any nodes drained or ignored), and the perf delta vs main. Link +the run(s) and PR(s). End by asking the user whether to merge. Do not merge yourself. diff --git a/.agents/skills/kimi-k3-b200-agentx/SKILL.md b/.agents/skills/kimi-k3-b200-agentx/SKILL.md new file mode 100644 index 0000000000..ec24ba738c --- /dev/null +++ b/.agents/skills/kimi-k3-b200-agentx/SKILL.md @@ -0,0 +1,166 @@ +--- +name: kimi-k3-b200-agentx +description: Maintain the evolving target-hardware evidence, topology decisions, concurrency ladders, and run record for Kimi K3 AgentX tuning on 16 B200 GPUs in InferenceX PR #2475. Use with tune-agentx-submission and debug-agentx-runs. +--- + +# Kimi K3 B200 AgentX working record + +Local-only working skill for PR #2475. Do not commit, push, publish, or include private cluster addresses. Replace stale estimates when target-hardware measurements become available. + +Last updated 2026-08-03. + +## Scope and constraints + +- PR: https://github.com/SemiAnalysisAI/InferenceX/pull/2475 +- Hardware: 2 B200 nodes, 8 GPUs per node, 16 GPUs total. +- All aggregate multinode serving must use checked-in srt-slurm recipes. +- The recipe is the serving topology source of truth. Matrix metadata must match it. +- Preserve the native 1,048,576-token model context and the existing AgentX dataset. +- Synthetic acceptance is for comparable performance measurement. Real block verification remains the correctness path. + +## Blocking finding + +The branch's current TP8 by PP2 DSpark recipe is not upstream-supported. The official Kimi K3 recipe excludes DSpark from pipeline parallelism, and vLLM issue #50098 records the `SupportsPP` startup failure. + +Keep TP8 by PP2 only as a non-DSpark baseline until upstream support changes. Do not spend B200 time tuning a known unsupported DSpark composition. + +## Official Blackwell baseline + +Every candidate must verify the official settings rather than inherit the branch's omissions: + +- FP8 KV cache +- prefill query quantization with `TRTLLM_RAGGED` +- prefix caching enabled +- K3 latent-MoE tail fusion +- Model Runner v2 and Rust frontend where supported +- fastsafetensors loading +- native context length +- text-only mode for AgentX +- no FlashInfer autotune. + +For DSpark, official block verification uses a 7-token draft with probabilistic sampling. The checked-in performance profiles preserve K7 and the committed synthetic AL 3.84. Consider K2 only after the official K7 canaries establish a target-B200 baseline and only if one small comparison can change the selected Pareto point. + +## AgentX dataset evidence + +Dataset: https://huggingface.co/datasets/semianalysisai/cc-traces-weka-062126 + +It contains 393 trajectories and 98,827 model requests. + +| Statistic | Input tokens | +|---|---:| +| p50 | 142,016 | +| p75 | 310,464 | +| p90 | 549,504 | +| p95 | 682,880 | +| p99 | 863,471 | +| mean | 218,922 | + +Median cached fraction is about 99.1 percent. Median uncached input is 1,664 tokens. Prefix caching is required for representative performance. Disabling it converts incremental agent turns into repeated long prefills. + +## Prior B200 evidence + +Run: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/30327547020 + +This was a non-DSpark TP8 by PP2 baseline, not proof for the candidate topologies. Startup reported: + +- auto or BF16 KV +- 58.89 GiB KV cache per rank +- 4,147,581 logical KV tokens +- 3.96 maximum concurrency at 1M tokens +- prefix caching resolved false. + +Observed curve: + +| Conc | Total tok/s | Output tok/s | p50 TTFT | p50 TPOT | KV usage | +|---:|---:|---:|---:|---:|---:| +| 8 | 13,817 | 98.5 | 6.2 s | 45 ms | 45% | +| 16 | 11,366 | 53.2 | 12.2 s | 245 ms | 97% | +| 32 | 18,298 | 98.5 | 90.9 s | 198 ms | 100% | + +Concurrency 32 increases input processing but does not improve output throughput over concurrency 8. It is not a useful operating point for that topology. + +## Implemented profile matrix + +| Orientation | Topology | Initial concurrency | +|---|---|---| +| Latency | TP16, EP1 | 1, 2, 4, 8 | +| Balanced | TEP16, EP16 | 8, 16, 24, 32 | +| GPU throughput | DEP16, TP1 by DP16 by EP16 | 32, 64, 96, 128, 192, 256 | +| CPU KV | DEP16 plus SimpleCPUOffloadConnector | 128, 192, 256, 384 | + +This is an 18-point starting matrix. Concurrency 384 exercises nearly all 393 trajectories. + +Checked-in serving recipes: + +- `agg-b200-tp16-latency-dspark-agentic.yaml`, direct multi-node vLLM +- `agg-b200-tep16-balanced-dspark-agentic.yaml`, direct multi-node vLLM +- `agg-b200-dep16-throughput-dspark-agentic.yaml`, Dynamo aggregate with per-node DP ranks +- `agg-b200-dep16-throughput-vllm-simple-offload-dspark-agentic.yaml`, the same DEP topology with 220 GiB of CPU KV per rank. + +The launcher pins srt-slurm renderer commit `df5baa93f4caf5169dea2a4236ad2cc742fe40e7`. + +Run no more than two `agentx-fast` probes concurrently: + +1. First wave, TP16 at concurrency 1 and DEP16 without offload at concurrency 32. +2. Second wave, TEP16 at concurrency 8 and DEP16 with CPU offload at concurrency 128. + +Do not start the second wave until the first wave's live evidence is understood. Each probe must confirm all workers, all metrics sources, resolved FP8 KV, prefix caching, reported cache blocks, KDA headroom, routing balance, and zero deterministic request errors before expansion. + +## Capacity model + +K3 has 24 paged MLA layers with a 576-element latent per layer. Full-model FP8 MLA is approximately 13,824 bytes per token per rank before hybrid-cache overhead. KDA contributes fixed per-request recurrent and draft state. + +With the prior 58.89 GiB HBM reservation as a contingent assumption: + +| Layout | Estimated p50 request capacity | Meaning | +|---|---:|---| +| TP16 or TEP16 | about 32 | One logical cache namespace. Never multiply by 16. | +| DEP16 | about 26 per pool, 413 aggregate | Sixteen independent pools, contingent on balanced routing. | +| DEP16 plus 220 GiB CPU KV per rank | about 97 per pool, 1,544 aggregate | Cache-only upper bound, not a useful-throughput claim. | + +Startup HBM, KDA retention, allocator padding, scheduler limits, prefix-cache policy, and transfer bandwidth override these estimates. + +## Renderer decision + +Direct multi-node vLLM correctly renders leader and headless commands for TP16 and TEP16. It does not derive node-local DP ranges. + +DEP16 therefore uses srt-slurm's Dynamo aggregate path with `dp_launch_mode: per_node`. The renderer derives local size 8, start ranks 0 and 8, a shared DP address and RPC port, and hybrid load balancing. Do not move DEP to the direct frontend without a source-level renderer change and command-construction test. + +## Live decision gates + +Continue expanding a topology only while output throughput rises and latency remains on the intended Pareto frontier. + +Stop expanding when any condition persists: + +- deterministic startup or request failure +- missing or idle DP engine while peers queue +- KV usage pinned near 100 percent with growing queues +- flat or falling output throughput with sharply worse TTFT or TPOT +- CPU transfer traffic rising without useful throughput gain +- routing imbalance that invalidates aggregate capacity arithmetic +- no forward progress across repeated compute-visible metric samples. + +Record the evidence before recommending cancellation. While the user is unavailable, do not cancel or mutate shared-cluster state without prior authorization. + +## Access state + +The private route and file-based SSH key work when the local credential agent is bypassed with `IdentityAgent=none`. Compute-visible Slurm inspection is available. Keep host addresses outside this file. + +## Living run record + +Keep only current evidence. After each targeted or official run, update this table and the decisions above. + +| Date | Run and job | Profile and conc | Result | Measured cache and metrics | Decision | +|---|---|---|---|---|---| +| 2026-08-03 | 30327547020, prior baseline | TP8 by PP2, no DSpark, conc 1 to 32 | Completed, unofficial baseline | 58.89 GiB, 4.15M tokens, prefix false, KV cliff at 16 to 32 | Replace unsupported DSpark PP with TP16, TEP16, and DEP16 candidates | +| 2026-08-03 | 30837472733, job 91766231186, Slurm 29091 | TP16 direct vLLM, conc 1, `agentx-fast` | Canceled after deterministic startup stall | All 16 ranks reached NCCL 2.30.7. FlashInfer selected its cross-node MNNVL path, the pool reported no multicast support, GPU use stayed near 2 GiB/rank, and no weight load began for 16 minutes | Disable `VLLM_ALLREDUCE_USE_FLASHINFER` for cross-node TP16/TEP16 and rerun the same canary. The symmetric-memory warning was only a failed capability probe | +| 2026-08-03 | 30841915075, job 91780887584, Slurm 29113 | TP16 direct vLLM, conc 1, `agentx-fast` | Pending | Same profile with FlashInfer cross-node all-reduce disabled. Slurm allocation verified on the assigned runner | Monitor startup through accepted artifacts before dispatching another profile | + +## Sources + +- Official interactive recipe: https://recipes.vllm.ai/moonshotai/Kimi-K3 +- Official source recipe: https://github.com/vllm-project/recipes/blob/main/models/moonshotai/Kimi-K3.yaml +- Pipeline limitation: https://github.com/vllm-project/vllm/issues/50098 +- Kimi K3 model: https://huggingface.co/moonshotai/Kimi-K3 +- DSpark draft model: https://huggingface.co/Inferact/Kimi-K3-DSpark +- AgentX dataset: https://huggingface.co/datasets/semianalysisai/cc-traces-weka-062126 diff --git a/.agents/skills/tune-agentx-submission/SKILL.md b/.agents/skills/tune-agentx-submission/SKILL.md new file mode 100644 index 0000000000..b4a0daeecc --- /dev/null +++ b/.agents/skills/tune-agentx-submission/SKILL.md @@ -0,0 +1,70 @@ +--- +name: tune-agentx-submission +description: Supervise an AgentX performance submission from a bare recipe through a defensible Pareto sweep and accepted artifacts. Use for topology selection, staged tuning, and coordinating focused research and live-debug subskills. +--- + +# Tune an AgentX submission + +Local-only orchestration skill. Do not commit, push, publish, or store private cluster addresses. Keep this workflow short. Put model and hardware evidence in a focused subskill and update it after every material result. + +## Supporting skills + +Load only the skills that match the current phase: + +- `kimi-k3-b200-agentx` for PR #2475 decisions, measurements, concurrency ladders, and the living run record. +- `debug-agentx-runs` for compute-visible AgentX phase and health monitoring. +- `.agents/skills/debug-runs/SKILL.md` for the general CI, cluster reproduction, and merge-gate loop. + +Create another model and hardware subskill when the evidence cannot be kept cleanly separate. Do not turn this coordinator into a model-specific notebook. + +## Guardrails + +1. Work only on the assigned PR and hardware. +2. For aggregate multinode jobs, srt-slurm owns allocation, rank-aware startup, readiness, and logs. Never hand-roll node orchestration. +3. Treat the checked-in recipe as serving-topology truth. Matrix metadata must match it. +4. Start from the official upstream recipe. Deviations need target-hardware evidence. +5. Preserve native context and the selected AgentX dataset. +6. Use synthetic acceptance only for comparable performance. Require real verification for correctness. +7. Read-only cluster inspection is allowed. Without explicit prior authorization, leave cancellation and infrastructure mutation untouched. +8. Run at most one or two small cluster experiments concurrently. Expand only after their live evidence is understood. +9. Success requires accepted artifacts, not healthy-looking logs. + +## Supervision pattern + +Keep the top-level decision with the lead agent. Delegate bounded slices such as upstream compatibility, model and KV geometry, prior artifacts, or renderer behavior. + +Every slice must return: + +- observed facts with paths, run IDs, and source links +- estimates with formulas and assumptions +- contradictions or blockers +- the smallest experiment that can resolve uncertainty. + +Steer immediately when an agent follows an unsupported topology, treats an estimate as a measurement, or proposes custom orchestration. Cross-check consequential claims before they enter the case subskill. + +## Workflow + +1. **Ground the case.** Read the PR diff, current recipe, matrix entry, launcher, benchmark path, official recipe, model config, dataset distribution, and closest accepted artifacts. +2. **Design the Pareto profiles.** Cover low latency, balanced throughput, GPU-resident high concurrency, and CPU-offloaded maximum concurrency. Keep logical TP or TEP cache capacity separate from independent DEP pools. +3. **Verify rendering.** Generate the matrix and dry-run srt-slurm. Inspect leader and every non-leader command, especially rank-varying DP arguments. +4. **Run sparse canaries.** Use `agentx-fast` only for bring-up. Run one or two small points at a time, each exercising a distinct topology or mechanism. +5. **Monitor live and adapt.** Use compute-visible logs and metrics. Expand only while output throughput improves without leaving the intended latency frontier. +6. **Run the official sweep.** Remove fast mode, use the appropriate full-sweep label, verify correctness coverage and artifacts, then record the actual Pareto points. + +## Expansion and stop rules + +Continue only when all workers are active, routing is balanced, request errors are acceptable, and useful output throughput rises. + +Stop expanding a branch when KV stays saturated with a growing queue, output throughput flattens while TTFT or TPOT worsens, offload traffic rises without benefit, a worker remains idle, or progress stops across repeated metric samples. + +Capture evidence before recommending cancellation. If the user is unavailable and cancellation was not explicitly authorized, continue read-only monitoring and do not cancel. + +## Update protocol + +After every decision or run: + +1. Update the focused case subskill with measured values and links. +2. Replace estimates that the run resolved. +3. Remove superseded topology or concurrency guidance. +4. State the next hypothesis and smallest disconfirming run. +5. Keep private access details outside the repository. diff --git a/.claude/commands/add-model-hardware.md b/.claude/commands/add-model-hardware.md index 32f33ec95b..f28049b4b6 100644 --- a/.claude/commands/add-model-hardware.md +++ b/.claude/commands/add-model-hardware.md @@ -6,37 +6,37 @@ argument-hint: [recipes-link] [draft-model-link] [mtp] Add a new single-node benchmark recipe for a `model × hardware` combination, basing it on the closest existing sibling recipe, then ship it as one `[Klaud Cold]` PR with a full GPU sweep. -Inputs from `$ARGUMENTS` (links first, then SKU; the rest optional and order-tolerant): +Inputs from `$ARGUMENTS` (links first, then SKU. The rest are optional and order-tolerant): -- **model-link** (required) — HuggingFace URL of the exact checkpoint to benchmark, e.g. +- **model-link** (required). HuggingFace URL of the exact checkpoint to benchmark, e.g. `https://huggingface.co/MiniMaxAI/MiniMax-M3-MXFP8`. Derive from it: the `model:` id - (`org/repo`), the **precision** (from the repo name — `MXFP8`/`FP8`→`fp8`, `NVFP4`/`FP4`→`fp4`, + (`org/repo`), the **precision** (from the repo name, where `MXFP8`/`FP8`→`fp8`, `NVFP4`/`FP4`→`fp4`, `INT4`→`int4`, else `bf16`), and the **model-prefix** (e.g. `minimaxm3`). -- **gpu-sku** (required) — `b200 | b300 | h100 | h200 | gb200 | mi300x | mi325x | mi355x`. +- **gpu-sku** (required). Choose from `b200 | b300 | h100 | h200 | gb200 | mi300x | mi325x | mi355x`. Determines the master config (`mi*`→`amd-master.yaml`, else `nvidia-master.yaml`), the image repo (`vllm/vllm-openai` vs `vllm/vllm-openai-rocm`), and the launcher. -- **recipes-link** (optional) — the model's `recipes.vllm.ai` page (or a vendor recipe commit). +- **recipes-link** (optional). The model's `recipes.vllm.ai` page (or a vendor recipe commit). Consult it for the authoritative serve flags (block size, parsers, attention backend, parallelism guidance). If omitted, copy the sibling recipe's flags. -- **draft-model-link** (optional) — HF URL of a speculative-decoding draft (e.g. +- **draft-model-link** (optional). HF URL of a speculative-decoding draft (e.g. `https://huggingface.co/Inferact/MiniMax-M3-EAGLE3`). Its presence means **build the MTP - variant** (EAGLE3 with this draft); see the MTP appendix. -- **mtp** (optional) — force the `spec-decoding: mtp` variant even without a draft link (use + variant** (EAGLE3 with this draft). See the MTP appendix. +- **mtp** (optional). Force the `spec-decoding: mtp` variant even without a draft link (use native MTP if the checkpoint ships `num_mtp_modules > 0`). -**engine** defaults to `vllm`; infer otherwise from the sibling / recipes page. +**engine** defaults to `vllm`. Infer otherwise from the sibling / recipes page. -Standing prefs: PR title prefixed `[Klaud Cold]`; add `full-sweep-fail-fast` (strongly recommended over `full-sweep-enabled`) via the REST API -(`gh pr edit` hits the projects-classic GraphQL bug); fill the perf-changelog `pr-link` after -the PR exists; then monitor the sweep to a fail/success conclusion and report the job -breakdown. Do **not** invent image tags — verify on the registry first. +Standing prefs: Prefix the PR title with `[Klaud Cold]`. Add `full-sweep-fail-fast` (strongly recommended over `full-sweep-enabled`) through the REST API +because `gh pr edit` hits the projects-classic GraphQL bug. Fill the perf-changelog `pr-link` after +the PR exists. Then monitor the sweep to a fail/success conclusion and report the job +breakdown. Do **not** invent image tags. Verify them on the registry first. ## Step 0 — deep-research the recipe (do this thoroughly before writing anything) -Don't guess flags or concurrencies — **deep-research the InferenceX codebase first**, then +Don't guess flags or concurrencies. **Deep-research the InferenceX codebase first**, then the external sources. Read *several* similar files, not just one, and copy what actually runs. -**A. In-codebase research (primary — the repo is the source of truth):** +**A. In-codebase research (primary because this repo is the source of truth):** ```bash # similar benchmark scripts: same model on other SKUs, AND same SKU on other models ls benchmarks/single_node/fixed_seq_len/_*.sh benchmarks/single_node/fixed_seq_len/*_*.sh @@ -52,17 +52,17 @@ grep -nE "run_benchmark_serving|setup_eval_context|wait_for_server_ready|start_g KV-cache dtype, attention/MoE backend, parsers). These are the truth for each runner. - **Compare several master-config search spaces** (e.g. `dsv4`, `glm5`, the same model on a sibling SKU) to choose `{tp, ep, dp-attn} × concurrency` combos that fit *this* hardware's - memory (small-mem SKUs like h100/mi300x go TP8-only; bigger SKUs add tp4/tp2/DEP). + memory. Small-memory SKUs like h100/mi300x go TP8-only, while bigger SKUs add tp4/tp2/DEP. - **Internalize the fixed-seq-len nuances from the existing configs**: `8k1k`/`1k8k` do **not** need the full `MAX_MODEL_LEN` (the matrix supplies `isl + osl + slack`), and graph-capture - batch sizes are scaled to concurrency/scenario (and spec-token count for MTP), not maxed — - copy how sibling scripts/configs already do it. + batch sizes are scaled to concurrency/scenario (and spec-token count for MTP), not maxed. + Copy how sibling scripts/configs already do it. **B. External research (confirm against upstream guidance):** - **`WebFetch` the model-link card + its `config.json`** → confirm `model:` id, precision, max context, architecture, spec-decode fields (`num_mtp_modules`, etc.). -- **`WebFetch` the recipes-link** (if given) → canonical `vllm serve` flags + troubleshooting; - reconcile with what the sibling scripts do (the repo wins if they conflict — note why). +- **`WebFetch` the recipes-link** (if given) → canonical `vllm serve` flags + troubleshooting. + Reconcile with what the sibling scripts do. If they conflict, follow the repo and note why. - If a **draft-model-link** is given, note its id for `--speculative-config` and check the card for method (`eagle3` vs native `mtp`) and recommended token count. - Pick the **image tag** from the sibling's master-config entry (or recipes page) and **verify @@ -73,7 +73,7 @@ This research directly feeds Step 2 (script flags/env) and Step 3 (search space) ## What you're producing (4–5 files) 1. `benchmarks/single_node/fixed_seq_len/__[_][_mtp].sh` -2. an entry in the master config — **`configs/nvidia-master.yaml`** (b*/h*/gb* SKUs) or +2. an entry in either master config, **`configs/nvidia-master.yaml`** (b*/h*/gb* SKUs) or **`configs/amd-master.yaml`** (mi* SKUs) 3. a `perf-changelog.yaml` entry (this diff vs main is what selects the sweep) 4. (if missing) `SPEC_SUFFIX`/framework-suffix routing in `runners/launch_*.sh` @@ -88,7 +88,7 @@ ls benchmarks/single_node/fixed_seq_len/_* # same model, other ls benchmarks/single_node/fixed_seq_len/*_*.sh # same hardware, other model grep -n "--" configs/{nvidia,amd}-master.yaml ``` -Read the closest sibling script **and** its master-config entry — copy their flag shapes and +Read the closest sibling script **and** its master-config entry. Copy their flag shapes and search-space structure rather than inventing. The right model is "same model on a sibling SKU, adjusted for this hardware's quirks." @@ -99,16 +99,16 @@ the model's `recipes.vllm.ai` page: - **Mandatory model flags** (carry from the sibling): block size, parser flags (`--tool-call-parser` / `--reasoning-parser`), `--language-model-only` for text-only sweeps, `--trust-remote-code` where the model needs it. -- **Per-hardware deltas** — KV cache dtype (e.g. mi300x/gfx942 keeps **BF16**: no calibrated - ROCm FP8 attn scales; most others use `fp8`), attention backend (CUDA: FlashInfer default; - ROCm: `--attention-backend TRITON_ATTN`), and graph capture vs `--enforce-eager` (several +- **Per-hardware deltas.** KV cache dtype (e.g. mi300x/gfx942 keeps **BF16** because it has no calibrated + ROCm FP8 attn scales, while most others use `fp8`), attention backend (CUDA uses FlashInfer by default, + while ROCm uses `--attention-backend TRITON_ATTN`), and graph capture vs `--enforce-eager` (several AMD recipes use eager). -- **Capture sizing** — fixed-seq-len runs don't need graphs past the request concurrency: - capture up to the next power of two ≥ `CONC` (≥ `CONC * (1 + NUM_SPEC_TOKENS)` with spec +- **Capture sizing.** Fixed-seq-len runs don't need graphs past the request concurrency. + Capture up to the next power of two ≥ `CONC` (≥ `CONC * (1 + NUM_SPEC_TOKENS)` with spec decoding), capped at vLLM's 2048. -- **`MAX_MODEL_LEN`** is the matrix-supplied scenario value (`isl + osl + slack`) — never +- **`MAX_MODEL_LEN`** is the matrix-supplied scenario value (`isl + osl + slack`). Never hardcode the full context for 8k1k / 1k8k. -- **Memory headroom** — bigger checkpoints constrain TP/EP; if the sibling on a smaller-memory +- **Memory headroom.** Bigger checkpoints constrain TP/EP. If the sibling on a smaller-memory SKU is TP8-only (e.g. h100), match that. Validate as you go: `bash -n