diff --git a/.agents/skills/debug-agentx-runs/SKILL.md b/.agents/skills/debug-agentx-runs/SKILL.md index 4aaa7a1c93..a30cad52fe 100644 --- a/.agents/skills/debug-agentx-runs/SKILL.md +++ b/.agents/skills/debug-agentx-runs/SKILL.md @@ -6,13 +6,13 @@ description: Debug long-running AgentX benchmark jobs from live cluster logs and # Debug AgentX runs from the cluster AgentX runs have long model-load, warmup, drain, and profiling phases. Treat GitHub Actions -as the orchestration and final-status view; use the cluster as the primary source of live +as the orchestration and final-status view. Use the cluster as the primary source of live diagnostic signal. Compose with `$debug-runs` when it is available for the general full-sweep and reproduction workflow. Cluster login addresses, users, jumpboxes, runner directories, and storage paths belong in the access-controlled InferenceX Clusters Slack canvas, not this repository. Ask the user -for the canvas link or the intended SSH alias when access is not already configured; never +for the canvas link or the intended SSH alias when access is not already configured. Never guess or publish private infrastructure details. Read-only SSH inspection is allowed. Before editing cluster files, restarting or killing processes, draining nodes, cancelling Slurm jobs, or otherwise mutating shared infrastructure, stop and ask for approval unless @@ -55,7 +55,7 @@ scontrol show job -o | tr " " "\n" | grep -E "^(JobId|JobState|RunTime|TimeLimit|NodeList|WorkDir)=" ``` -Derive paths from `WorkDir`; do not guess or hardcode a cluster path. For an srt-slurm +Derive paths from `WorkDir`. Do not guess or hardcode a cluster path. For an srt-slurm job, logs normally live under: ```text @@ -129,11 +129,11 @@ curl -fsS "" | Prefer the metric names actually exposed by the running image instead of assuming a specific vLLM, SGLang, or Dynamo version. Track at least: -- active/running and waiting requests; -- KV-cache usage and prefix-cache hit rate; -- input and output token rates; -- completed, cancelled, and errored requests; -- frontend active requests and per-worker routing balance; +- active/running and waiting requests +- KV-cache usage and prefix-cache hit rate +- input and output token rates +- completed, cancelled, and errored requests +- frontend active requests and per-worker routing balance - KV-transfer activity and failures for disaggregated runs. Interpret trends, not one scrape: @@ -164,10 +164,10 @@ calculate the nominal end from the `Phase profiling started` timestamp plus the duration. Then allow a few minutes for cutoff drain, aggregation, staging, and artifact upload. State separately: -1. phase elapsed and remaining time; -2. whether logs are still updating; -3. errors observed; -4. expected benchmark completion; +1. phase elapsed and remaining time +2. whether logs are still updating +3. errors observed +4. expected benchmark completion 5. expected GitHub job completion. For warmup drains, report returned, sent, in-flight, errors, elapsed time, and the @@ -179,10 +179,10 @@ when validating a timeout change. Do not burn hours waiting for final JSON when direct signals already disqualify a config. Use server logs and metrics to make the decision early. Examples include: -- deterministic OOM, NCCL/RCCL failure, parser crash, or missing worker; -- no forward progress across repeated samples; -- persistent KV saturation and queue growth with unusable latency; -- throughput that has plateaued while added concurrency only increases latency; +- deterministic OOM, NCCL/RCCL failure, parser crash, or missing worker +- no forward progress across repeated samples +- persistent KV saturation and queue growth with unusable latency +- throughput that has plateaued while added concurrency only increases latency - a disaggregated pool or metrics source that never registered. Before cancellation, capture the relevant log lines, timestamps, topology, and metric @@ -196,7 +196,7 @@ gh run cancel --repo SemiAnalysisAI/InferenceX ``` Use `scancel` or direct process termination only with explicit approval and a concrete -reason; doing so can bypass cleanup or strand shared-cluster state. Never kill only the +reason. Doing so can bypass cleanup or strand shared-cluster state. Never kill only the backend and leave the workflow silently occupying a runner. After a recipe/config fix, use a targeted e2e dispatch for fast feedback. Reserve another @@ -206,13 +206,13 @@ official full sweep for the candidate that has passed direct cluster inspection. For each active point, report: -- GitHub job and Slurm job links/IDs; -- aggregate versus disaggregated topology; -- phase, elapsed time, remaining time, and last log update; -- log files and metrics sources inspected; -- request, queue, KV-cache, cache-hit, and token-rate trends; -- errors and the likely root cause; -- whether to continue, short-circuit, or rerun; +- GitHub job and Slurm job links/IDs +- aggregate versus disaggregated topology +- phase, elapsed time, remaining time, and last log update +- log files and metrics sources inspected +- request, queue, KV-cache, cache-hit, and token-rate trends +- errors and the likely root cause +- whether to continue, short-circuit, or rerun - which points are fully green versus merely healthy in progress. Do not call a run successful until GitHub has accepted its result artifacts and the diff --git a/.claude/skills/debug-runs/SKILL.md b/.agents/skills/debug-runs/SKILL.md similarity index 55% rename from .claude/skills/debug-runs/SKILL.md rename to .agents/skills/debug-runs/SKILL.md index 898c117381..cae671aba1 100644 --- a/.claude/skills/debug-runs/SKILL.md +++ b/.agents/skills/debug-runs/SKILL.md @@ -1,16 +1,16 @@ --- name: debug-runs -description: Drive a full-sweep benchmark config to green with a tight feedback loop — trigger/monitor the sweep, root-cause failures, and (for fast iteration) SSH onto the runner's cluster to reproduce a single config directly on the node instead of waiting for full CI. Use when bringing up a new model/precision/SKU recipe, debugging a failing or flaky sweep, debugging node-level issues, or gathering context on a cluster before a run. Cluster access details are NOT in this repo — read them from the shared InferenceX Clusters canvas. +description: Drive a full-sweep benchmark config to green with a tight feedback loop by triggering and monitoring the sweep, finding the root causes of failures, and, for fast iteration, using SSH to reproduce a single config directly on the runner's cluster instead of waiting for full CI. Use when bringing up a new model, precision, or SKU recipe, debugging a failing or flaky sweep, debugging node-level issues, or gathering context on a cluster before a run. Cluster access details are not in this repo and must be read from the shared InferenceX Clusters canvas. --- # Debug runs (tight feedback loop) -Use this when the goal is to get a **full-sweep config passing** — and you want to verify -on the actual nodes first (tighter loop than the full CI dispatch cycle), or to debug -node/infra issues, or just to gather context on a cluster. +Use this when the goal is to get a **full-sweep config passing** and you want to verify it +on the actual nodes first for a tighter loop than the full CI dispatch cycle, debug +node or infrastructure issues, or gather context on a cluster. -This composes with the other skills: use **`/nuke`** (or `/add-model-hardware`) to create -the PR(s) with the image bump + perf-changelog entry + `full-sweep-fail-fast` label; this +This composes with the other skills. Use **`/nuke`** (or `/add-model-hardware`) to create +the PR(s) with the image bump, perf-changelog entry, and `full-sweep-fail-fast` label. This skill is the **monitor → root-cause → fix → re-verify → merge-gate** loop that follows. ## Cluster access — read it from the canvas, never hardcode it @@ -33,7 +33,7 @@ and the benchmark command. If this is a **fork** (i.e. not the SemiAnalysis upstream, where the canvas won't apply), ask the user to replace this skill with their own fork's runner/cluster access information. -- A SemiAnalysis operator may also have these as `~/.ssh/config` aliases — prefer those if +- A SemiAnalysis operator may also have these as `~/.ssh/config` aliases. Prefer those if present. ## Inputs @@ -46,10 +46,10 @@ and the benchmark command. ### 1. Trigger (or reuse) the sweep -A PR's sweep is kicked by labels (the `/sweep` comment trigger was removed — use the label): +A PR's sweep is kicked by labels. The `/sweep` comment trigger was removed, so use the label: -- **`full-sweep-fail-fast`** — full sweep that bails on first failure per matrix (faster signal while debugging; **strongly recommended default**, and what `/nuke` attaches). -- **`full-sweep-enabled`** — full GPU sweep that runs every job to completion despite failures; use only when a flaky job killing its matrix's in-flight results is unacceptable. +- **`full-sweep-fail-fast`** runs the full sweep and bails on the first failure per matrix for faster feedback while debugging. It is the **strongly recommended default** and the label that `/nuke` attaches. +- **`full-sweep-enabled`** runs the full GPU sweep and lets every job complete despite failures. Use it only when a flaky job killing its matrix's in-flight results is unacceptable. - To re-trigger a sweep without a new commit, remove and re-add the sweep label. For a **single config** (tightest CI loop, skips the rest of the matrix), dispatch e2e directly: @@ -58,11 +58,11 @@ For a **single config** (tightest CI loop, skips the rest of the matrix), dispat gh workflow run e2e-tests.yml -f generate-cli-command="test-config --config-key --config-file " -f test-name="debug " ``` -(`generate-cli-command` is the required input; `--target` is NOT a real arg.) +(`generate-cli-command` is the required input. `--target` is NOT a real arg.) ### 2. Monitor continuously -Find the run, then watch it — don't poll by hand. Prefer the **Monitor** tool with a +Find the run, then watch it instead of polling by hand. Prefer the **Monitor** tool with a filter that catches both progress and failure signatures so silence never reads as success. ```bash @@ -73,7 +73,7 @@ gh run watch "$RUN_ID" --repo SemiAnalysisAI/InferenceX --interval 30 ``` When monitoring several runs at once (e.g. 4 SKUs), track them by `databaseId` and report -each as it lands — never declare success from absence of output. +each as it lands. Never declare success from absence of output. ### 3. Root-cause a failure @@ -90,25 +90,25 @@ in one or two sentences before changing anything. ### 4. Tight loop: reproduce on the node directly -This is the point of the skill — instead of re-dispatching CI for every hypothesis, get on +This is the point of the skill. Instead of re-dispatching CI for every hypothesis, get on the box and reproduce the **single** failing config. -Why this is tighter: under e2e / the matrix, **each concurrency / config runs against its -own freshly-spun engine** (a new server per matrix job). On the node you can spin up a -**single** server once and fire many requests / sweep multiple concurrencies against it — -far faster iteration when you're probing behavior or tuning, since you skip a fresh model -load per data point. +Why this is tighter: in e2e or full-matrix runs, **each matrix point runs against its own +freshly spun engine** (a new server per matrix job). On the node you can spin up a +**single** server once and fire many requests or sweep multiple concurrencies against it. +This makes iteration much faster when you're probing behavior or tuning because you skip a +fresh model load per data point. Steps: -1. Look up the cluster's access + which node ran the failing job (from the job name / runner - name) in the canvas; SSH in (use `ssh -A` when a jumpbox/agent-forwarding is involved). -2. Reproduce the exact benchmark the launcher runs — read `runners/launch_.sh` for - the image, container mounts, and the `benchmarks/single_node/<...>.sh` command + env - (`IMAGE`, `TP`, `PRECISION`, `EXP_NAME`, `SPEC_DECODING`, …). On Slurm clusters that's a - `salloc`/`srun` with the squash image; on the **bare-metal `-tw` pools it's `docker run`** - on the node directly (no `srun`). -3. **Always diff against a working node / working SKU** for reference — most node failures +1. Use the job or runner name to identify the node. Look up that cluster's access details in + the canvas, then SSH in with `ssh -A` when a jumpbox or agent forwarding is involved. +2. Reproduce the exact benchmark the launcher runs. Read `runners/launch_.sh` for + the image, container mounts, and the `benchmarks/single_node/<...>.sh` command and env + (`IMAGE`, `TP`, `PRECISION`, `EXP_NAME`, `SPEC_DECODING`, …). On Slurm clusters, use + `salloc` or `srun` with the squash image. On the **bare-metal `-tw` pools, use `docker run`** + on the node directly without `srun`. +3. **Always diff against a working node or working SKU** for reference. Most node failures are environment drift (driver, ROCm/CUDA, missing mount, stale squash image), not code. 4. Iterate on the node until the single config passes, then push the fix and re-run CI. @@ -123,30 +123,30 @@ enroot list -f # find the running container's PID enroot exec bash # drop into the container ``` -(On the bare-metal `-tw` pools there's no Slurm/enroot — use `docker ps` + `docker exec -it bash`.) +(On the bare-metal `-tw` pools, there is no Slurm/enroot. Use `docker ps` and `docker exec -it bash`.) -**Node-level fixes are in scope** when you have operator access (e.g. AMD nodes where you -have sudo) — but **ask the user before executing any of them** (see guardrail below). The -kinds of fixes that are on the table: bringing a node's environment in line with the -working reference, and — if one or two nodes are unrecoverable — **draining them** or +**Node-level fixes are in scope** when you have operator access, such as on AMD nodes where +you have sudo, but **ask the user before executing any of them** (see guardrail below). The +kinds of fixes that are on the table include bringing a node's environment in line with the +working reference and, if one or two nodes are unrecoverable, **draining them** or explicitly **ignoring them in the run script** rather than blocking the whole sweep. Note any such change in the report. -> Guardrail — ask before changing infra. SSHing in to **read/investigate** (logs, +> Guardrail. Ask before changing infra. SSHing in to **read/investigate** (logs, > `rocm-smi`/`nvidia-smi`, `sinfo`/`squeue`, `df`, env, config inspection) is fine. But -> before making **any actual change on the cluster** — installing/updating anything, -> editing configs or files, restarting/killing processes, draining or ignoring nodes, -> changing the run script, or anything else that mutates node/shared state — **stop and ask -> the user for permission first**, describing exactly what you intend to run. Don't assume -> standing authorization just because you have sudo or operator access. +> before making **any actual change on the cluster**, including installing or updating +> anything, editing configs or files, restarting or killing processes, draining or ignoring +> nodes, changing the run script, or anything else that mutates node or shared state, +> **stop and ask the user for permission first**, describing exactly what you intend to run. +> Do not assume standing authorization just because you have sudo or operator access. > -> Also do **not** apply hacky engine-side (e.g. vLLM) workarounds to force a pass — prefer -> recipe fixes and (once approved) node-environment fixes that match a working reference. +> Also do **not** apply hacky engine-side (e.g. vLLM) workarounds to force a pass. Prefer +> recipe fixes and, once approved, node-environment fixes that match a working reference. ### 5. Flakes: rerun, don't relaunch If a job flaked (infra, transient network, runner pickup) rather than a real failure, rerun -just the failed jobs on the existing run — don't dispatch a fresh sweep: +just the failed jobs on the existing run. Don't dispatch a fresh sweep: ```bash gh run rerun "$RUN_ID" --repo SemiAnalysisAI/InferenceX --failed @@ -154,22 +154,22 @@ gh run rerun "$RUN_ID" --repo SemiAnalysisAI/InferenceX --failed ### 6. Report results — do NOT merge -**Never merge.** Merging is the user's call — only merge if the user **explicitly** tells -you to in this session. Even when everything looks perfect, stop and report; do not +**Never merge.** Merging is the user's call. Only merge if the user **explicitly** tells +you to in this session. Even when everything looks perfect, stop and report. Do not admin-merge on your own judgment. Report the two things the user will decide on: -1. **Sweep status** — is it 100% of full-sweep jobs passing (green), or fail-fast-truncated / partial? +1. **Sweep status.** Is it 100% of full-sweep jobs passing (green), or fail-fast-truncated or partial? 2. **Perf delta** vs the most recent official `main` run for that SKU. Compare against the - latest main results — e.g. on inferencex.semianalysis.com + latest main results, e.g. on inferencex.semianalysis.com (`https://inferencex.semianalysis.com/inference?...&i_active=_`) or the stored results for that SKU's last main `run-id`. -Present green-ness + the perf comparison, then **wait for the user** to decide whether to merge. +Present green-ness and the perf comparison, then **wait for the user** to decide whether to merge. ## Final report -Per config-key: final state (green / failing / flaky), root cause(s) found, node-level -changes made (and any nodes drained/ignored), and the perf delta vs main. Link the run(s) -and PR(s). End by asking the user whether to merge — do not merge yourself. +Per config-key, give the final state (green, failing, or flaky), the root cause(s) found, +node-level changes made (and any nodes drained or ignored), and the perf delta vs main. Link +the run(s) and PR(s). End by asking the user whether to merge. Do not merge yourself. diff --git a/.claude/commands/add-model-hardware.md b/.claude/commands/add-model-hardware.md index 32f33ec95b..f28049b4b6 100644 --- a/.claude/commands/add-model-hardware.md +++ b/.claude/commands/add-model-hardware.md @@ -6,37 +6,37 @@ argument-hint: [recipes-link] [draft-model-link] [mtp] Add a new single-node benchmark recipe for a `model × hardware` combination, basing it on the closest existing sibling recipe, then ship it as one `[Klaud Cold]` PR with a full GPU sweep. -Inputs from `$ARGUMENTS` (links first, then SKU; the rest optional and order-tolerant): +Inputs from `$ARGUMENTS` (links first, then SKU. The rest are optional and order-tolerant): -- **model-link** (required) — HuggingFace URL of the exact checkpoint to benchmark, e.g. +- **model-link** (required). HuggingFace URL of the exact checkpoint to benchmark, e.g. `https://huggingface.co/MiniMaxAI/MiniMax-M3-MXFP8`. Derive from it: the `model:` id - (`org/repo`), the **precision** (from the repo name — `MXFP8`/`FP8`→`fp8`, `NVFP4`/`FP4`→`fp4`, + (`org/repo`), the **precision** (from the repo name, where `MXFP8`/`FP8`→`fp8`, `NVFP4`/`FP4`→`fp4`, `INT4`→`int4`, else `bf16`), and the **model-prefix** (e.g. `minimaxm3`). -- **gpu-sku** (required) — `b200 | b300 | h100 | h200 | gb200 | mi300x | mi325x | mi355x`. +- **gpu-sku** (required). Choose from `b200 | b300 | h100 | h200 | gb200 | mi300x | mi325x | mi355x`. Determines the master config (`mi*`→`amd-master.yaml`, else `nvidia-master.yaml`), the image repo (`vllm/vllm-openai` vs `vllm/vllm-openai-rocm`), and the launcher. -- **recipes-link** (optional) — the model's `recipes.vllm.ai` page (or a vendor recipe commit). +- **recipes-link** (optional). The model's `recipes.vllm.ai` page (or a vendor recipe commit). Consult it for the authoritative serve flags (block size, parsers, attention backend, parallelism guidance). If omitted, copy the sibling recipe's flags. -- **draft-model-link** (optional) — HF URL of a speculative-decoding draft (e.g. +- **draft-model-link** (optional). HF URL of a speculative-decoding draft (e.g. `https://huggingface.co/Inferact/MiniMax-M3-EAGLE3`). Its presence means **build the MTP - variant** (EAGLE3 with this draft); see the MTP appendix. -- **mtp** (optional) — force the `spec-decoding: mtp` variant even without a draft link (use + variant** (EAGLE3 with this draft). See the MTP appendix. +- **mtp** (optional). Force the `spec-decoding: mtp` variant even without a draft link (use native MTP if the checkpoint ships `num_mtp_modules > 0`). -**engine** defaults to `vllm`; infer otherwise from the sibling / recipes page. +**engine** defaults to `vllm`. Infer otherwise from the sibling / recipes page. -Standing prefs: PR title prefixed `[Klaud Cold]`; add `full-sweep-fail-fast` (strongly recommended over `full-sweep-enabled`) via the REST API -(`gh pr edit` hits the projects-classic GraphQL bug); fill the perf-changelog `pr-link` after -the PR exists; then monitor the sweep to a fail/success conclusion and report the job -breakdown. Do **not** invent image tags — verify on the registry first. +Standing prefs: Prefix the PR title with `[Klaud Cold]`. Add `full-sweep-fail-fast` (strongly recommended over `full-sweep-enabled`) through the REST API +because `gh pr edit` hits the projects-classic GraphQL bug. Fill the perf-changelog `pr-link` after +the PR exists. Then monitor the sweep to a fail/success conclusion and report the job +breakdown. Do **not** invent image tags. Verify them on the registry first. ## Step 0 — deep-research the recipe (do this thoroughly before writing anything) -Don't guess flags or concurrencies — **deep-research the InferenceX codebase first**, then +Don't guess flags or concurrencies. **Deep-research the InferenceX codebase first**, then the external sources. Read *several* similar files, not just one, and copy what actually runs. -**A. In-codebase research (primary — the repo is the source of truth):** +**A. In-codebase research (primary because this repo is the source of truth):** ```bash # similar benchmark scripts: same model on other SKUs, AND same SKU on other models ls benchmarks/single_node/fixed_seq_len/_*.sh benchmarks/single_node/fixed_seq_len/*_*.sh @@ -52,17 +52,17 @@ grep -nE "run_benchmark_serving|setup_eval_context|wait_for_server_ready|start_g KV-cache dtype, attention/MoE backend, parsers). These are the truth for each runner. - **Compare several master-config search spaces** (e.g. `dsv4`, `glm5`, the same model on a sibling SKU) to choose `{tp, ep, dp-attn} × concurrency` combos that fit *this* hardware's - memory (small-mem SKUs like h100/mi300x go TP8-only; bigger SKUs add tp4/tp2/DEP). + memory. Small-memory SKUs like h100/mi300x go TP8-only, while bigger SKUs add tp4/tp2/DEP. - **Internalize the fixed-seq-len nuances from the existing configs**: `8k1k`/`1k8k` do **not** need the full `MAX_MODEL_LEN` (the matrix supplies `isl + osl + slack`), and graph-capture - batch sizes are scaled to concurrency/scenario (and spec-token count for MTP), not maxed — - copy how sibling scripts/configs already do it. + batch sizes are scaled to concurrency/scenario (and spec-token count for MTP), not maxed. + Copy how sibling scripts/configs already do it. **B. External research (confirm against upstream guidance):** - **`WebFetch` the model-link card + its `config.json`** → confirm `model:` id, precision, max context, architecture, spec-decode fields (`num_mtp_modules`, etc.). -- **`WebFetch` the recipes-link** (if given) → canonical `vllm serve` flags + troubleshooting; - reconcile with what the sibling scripts do (the repo wins if they conflict — note why). +- **`WebFetch` the recipes-link** (if given) → canonical `vllm serve` flags + troubleshooting. + Reconcile with what the sibling scripts do. If they conflict, follow the repo and note why. - If a **draft-model-link** is given, note its id for `--speculative-config` and check the card for method (`eagle3` vs native `mtp`) and recommended token count. - Pick the **image tag** from the sibling's master-config entry (or recipes page) and **verify @@ -73,7 +73,7 @@ This research directly feeds Step 2 (script flags/env) and Step 3 (search space) ## What you're producing (4–5 files) 1. `benchmarks/single_node/fixed_seq_len/__[_][_mtp].sh` -2. an entry in the master config — **`configs/nvidia-master.yaml`** (b*/h*/gb* SKUs) or +2. an entry in either master config, **`configs/nvidia-master.yaml`** (b*/h*/gb* SKUs) or **`configs/amd-master.yaml`** (mi* SKUs) 3. a `perf-changelog.yaml` entry (this diff vs main is what selects the sweep) 4. (if missing) `SPEC_SUFFIX`/framework-suffix routing in `runners/launch_*.sh` @@ -88,7 +88,7 @@ ls benchmarks/single_node/fixed_seq_len/_* # same model, other ls benchmarks/single_node/fixed_seq_len/*_*.sh # same hardware, other model grep -n "--" configs/{nvidia,amd}-master.yaml ``` -Read the closest sibling script **and** its master-config entry — copy their flag shapes and +Read the closest sibling script **and** its master-config entry. Copy their flag shapes and search-space structure rather than inventing. The right model is "same model on a sibling SKU, adjusted for this hardware's quirks." @@ -99,16 +99,16 @@ the model's `recipes.vllm.ai` page: - **Mandatory model flags** (carry from the sibling): block size, parser flags (`--tool-call-parser` / `--reasoning-parser`), `--language-model-only` for text-only sweeps, `--trust-remote-code` where the model needs it. -- **Per-hardware deltas** — KV cache dtype (e.g. mi300x/gfx942 keeps **BF16**: no calibrated - ROCm FP8 attn scales; most others use `fp8`), attention backend (CUDA: FlashInfer default; - ROCm: `--attention-backend TRITON_ATTN`), and graph capture vs `--enforce-eager` (several +- **Per-hardware deltas.** KV cache dtype (e.g. mi300x/gfx942 keeps **BF16** because it has no calibrated + ROCm FP8 attn scales, while most others use `fp8`), attention backend (CUDA uses FlashInfer by default, + while ROCm uses `--attention-backend TRITON_ATTN`), and graph capture vs `--enforce-eager` (several AMD recipes use eager). -- **Capture sizing** — fixed-seq-len runs don't need graphs past the request concurrency: - capture up to the next power of two ≥ `CONC` (≥ `CONC * (1 + NUM_SPEC_TOKENS)` with spec +- **Capture sizing.** Fixed-seq-len runs don't need graphs past the request concurrency. + Capture up to the next power of two ≥ `CONC` (≥ `CONC * (1 + NUM_SPEC_TOKENS)` with spec decoding), capped at vLLM's 2048. -- **`MAX_MODEL_LEN`** is the matrix-supplied scenario value (`isl + osl + slack`) — never +- **`MAX_MODEL_LEN`** is the matrix-supplied scenario value (`isl + osl + slack`). Never hardcode the full context for 8k1k / 1k8k. -- **Memory headroom** — bigger checkpoints constrain TP/EP; if the sibling on a smaller-memory +- **Memory headroom.** Bigger checkpoints constrain TP/EP. If the sibling on a smaller-memory SKU is TP8-only (e.g. h100), match that. Validate as you go: `bash -n