[Examples]: MiniMax-M3 DSpark#1965
Conversation
📝 WalkthroughWalkthroughThe changes add Gemma final-norm support, validate generation tags for answer-only loss masking, add a MiniMax-M3 chat template, and provide a multi-node DSpark streaming training configuration with configurable vLLM block size. ChangesSpeculative streaming support
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant DSparkPipeline
participant MakeDataset
participant TrainEagleStreaming
participant VLLMServe
DSparkPipeline->>MakeDataset: create train.jsonl
DSparkPipeline->>TrainEagleStreaming: launch multi-node training
TrainEagleStreaming->>VLLMServe: start vLLM with block size 128
VLLMServe-->>TrainEagleStreaming: serve speculative decoding
Caution Pre-merge checks failedPlease resolve all errors before merging. Addressing warnings is optional.
❌ Failed checks (1 error, 1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Warning
CodeRabbit couldn't request changes on this pull request because it doesn't have sufficient GitHub permissions.
Please grant CodeRabbit Pull requests: Read and write permission and re-run the review.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@modelopt/torch/speculative/plugins/hf_streaming_dataset.py`:
- Around line 116-124: Update the error message in the assistant-mask validation
guard to reference the actual template location,
tools/launcher/examples/MiniMaxAI/MiniMax-M3/m3_chat_template_generation.jinja,
while leaving the surrounding guidance unchanged.
- Around line 106-124: The guard around answer_only_loss must also raise when
the tokenizer is non-fast and no loss-mask recovery is available, regardless of
whether the chat template contains {% generation %} tags. Update the condition
using recovery, tokenizer.is_fast, and the existing template check, and adjust
the RuntimeError message to identify the unsupported slow-tokenizer path while
preserving the current guidance.
In
`@tools/launcher/examples/MiniMaxAI/MiniMax-M3/hf_streaming_dspark_multi_node.yaml`:
- Around line 99-100: Add the required MLM_MODEL_CFG environment variable
alongside HF_MODEL_CKPT in the MiniMax-M3 configuration, setting it to the
model’s Hugging Face repository ID and preserving the existing HF_MODEL_CKPT
reference.
In
`@tools/launcher/examples/MiniMaxAI/MiniMax-M3/m3_chat_template_generation.jinja`:
- Line 1: Add the required provenance and licensing notices at the top of the
template before the special token variables: include the MiniMax source
reference, the upstream copyright and license notice, and the NVIDIA Apache 2.0
header. Also update the repository LICENSE to include the MiniMax Community
License.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 18176f48-ae3d-49dc-b512-26580fa2bc17
📒 Files selected for processing (8)
examples/speculative_decoding/requirements.txtmodelopt/torch/speculative/plugins/hf_streaming_dataset.pymodelopt/torch/speculative/plugins/modeling_fakebase.pymodelopt/torch/speculative/plugins/modeling_final_norm.pytests/unit/torch/speculative/plugins/test_hf_streaming_dataset.pytools/launcher/common/eagle3/train_eagle_streaming.shtools/launcher/examples/MiniMaxAI/MiniMax-M3/hf_streaming_dspark_multi_node.yamltools/launcher/examples/MiniMaxAI/MiniMax-M3/m3_chat_template_generation.jinja
| recovery = None | ||
| if answer_only_loss and not getattr(tokenizer, "is_fast", False): | ||
| recovery = get_loss_mask_recovery(tokenizer) | ||
| if answer_only_loss and recovery is None: | ||
| # Fail loudly on a template without {% generation %} tags: transformers only | ||
| # warns and returns an ALL-ZERO assistant mask, which trains every sample at | ||
| # zero loss with no other symptom. (The regex matches the tag itself, not the | ||
| # unrelated `add_generation_prompt` identifier most templates contain.) | ||
| template = getattr(tokenizer, "chat_template", None) or "" | ||
| if not re.search(r"\{%-?\s*generation\b", template): | ||
| raise RuntimeError( | ||
| "answer_only_loss=True needs assistant masks, but the tokenizer's chat " | ||
| "template has no {% generation %} tags, so apply_chat_template would " | ||
| "return an all-zero mask and training would silently run at zero loss. " | ||
| "Pass a tagged template copy via data.chat_template (see " | ||
| "examples/MiniMaxAI/MiniMax-M3/m3_chat_template_generation.jinja for a " | ||
| "worked example), register a loss-mask recovery " | ||
| "(modelopt.torch.utils.loss_mask), or set answer_only_loss=false." | ||
| ) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
python3 - <<'PY'
from pathlib import Path
path = Path("modelopt/torch/speculative/plugins/hf_streaming_dataset.py")
text = path.read_text().splitlines()
for start, end in [(1, 220)]:
for i, line in enumerate(text[start-1:end], start):
print(f"{i:4d}: {line}")
PYRepository: NVIDIA/Model-Optimizer
Length of output: 11809
🌐 Web query:
Transformers apply_chat_template return_assistant_tokens_mask slow tokenizer char_to_token fast tokenizer
💡 Result:
The return_assistant_tokens_mask parameter in transformers.apply_chat_template is used to generate a binary mask identifying tokens produced by the assistant (1 for assistant tokens, 0 for others), which is essential for excluding non-assistant text during supervised fine-tuning [1][2][3]. Key Technical Details: 1. Mechanism: This feature requires chat templates to use the {% generation %} keyword to explicitly define the assistant's response segments [1][4][3]. During tokenization, the method calculates the character-level offsets of these generation blocks and aligns them with the tokenized sequence [2]. 2. Dependency on Fast Tokenizers: The implementation relies heavily on the fast tokenizer's ability to map between character space and token space (e.g., using char_to_token mapping methods) [2][5]. Historically, issues with inaccurate masks (e.g., all zeros) were often linked to underlying defects in the fast tokenizer's alignment logic [5]. 3. Constraints: The parameter functions only when tokenize=True and return_dict=True [6]. Some classes inheriting from ProcessorMixin have encountered issues where these arguments were inadvertently constrained, blocking the use of the assistant mask [6]. While fast tokenizers are preferred due to their robust alignment capabilities, slow (Python-based) tokenizers lack the optimized alignment infrastructure required for this feature and generally do not support or correctly implement the advanced character-to-token mappings needed for this specific mask generation [3][7][8]. If you encounter problems with the mask being all zeros, ensure you are using a recent version of the library, as several fixes have been merged into the rust-based tokenizers backend to resolve alignment inaccuracies [5].
Citations:
- 1: Return assistant generated tokens mask in apply_chat_template huggingface/transformers#30650
- 2: Return assistant generated tokens mask in apply_chat_template huggingface/transformers#30650
- 3: https://huggingface.co/docs/transformers/main/en/main%5Fclasses/tokenizer
- 4: huggingface/transformers@74d0eb3
- 5: apply_chat_template method not working correctly for llama 3 tokenizer huggingface/transformers#33091
- 6:
return_assistant_tokens_maskargument is blocked inProcessorMixin.apply_chat_templatehuggingface/transformers#36713 - 7: https://huggingface.co/docs/transformers/main/en/internal/tokenization%5Futils
- 8: https://huggingface.co/docs/transformers/v5.10.2/en/main_classes/tokenizer
Handle non-fast tokenizers in this guard. If tokenizer.is_fast is false and no recovery is registered, a template with {% generation %} tags still passes this check and falls through to apply_chat_template(..., return_assistant_tokens_mask=True), which depends on fast-tokenizer alignment. Fold the slow-tokenizer case into the same RuntimeError path so the failure is explicit.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@modelopt/torch/speculative/plugins/hf_streaming_dataset.py` around lines 106
- 124, The guard around answer_only_loss must also raise when the tokenizer is
non-fast and no loss-mask recovery is available, regardless of whether the chat
template contains {% generation %} tags. Update the condition using recovery,
tokenizer.is_fast, and the existing template check, and adjust the RuntimeError
message to identify the unsupported slow-tokenizer path while preserving the
current guidance.
| raise RuntimeError( | ||
| "answer_only_loss=True needs assistant masks, but the tokenizer's chat " | ||
| "template has no {% generation %} tags, so apply_chat_template would " | ||
| "return an all-zero mask and training would silently run at zero loss. " | ||
| "Pass a tagged template copy via data.chat_template (see " | ||
| "examples/MiniMaxAI/MiniMax-M3/m3_chat_template_generation.jinja for a " | ||
| "worked example), register a loss-mask recovery " | ||
| "(modelopt.torch.utils.loss_mask), or set answer_only_loss=false." | ||
| ) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Stale path in the error message.
The pointer examples/MiniMaxAI/MiniMax-M3/m3_chat_template_generation.jinja doesn't match the actual recipe location; per the PR's file list the template lives at tools/launcher/examples/MiniMaxAI/MiniMax-M3/m3_chat_template_generation.jinja. The repo also has a distinct top-level examples/ directory (e.g. examples/speculative_decoding/requirements.txt), so this path silently points to a directory that doesn't contain the file, defeating the "actionable error" goal of this guard.
📝 Proposed fix
- "Pass a tagged template copy via data.chat_template (see "
- "examples/MiniMaxAI/MiniMax-M3/m3_chat_template_generation.jinja for a "
- "worked example), register a loss-mask recovery "
+ "Pass a tagged template copy via data.chat_template (see "
+ "tools/launcher/examples/MiniMaxAI/MiniMax-M3/m3_chat_template_generation.jinja "
+ "for a worked example), register a loss-mask recovery "📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| raise RuntimeError( | |
| "answer_only_loss=True needs assistant masks, but the tokenizer's chat " | |
| "template has no {% generation %} tags, so apply_chat_template would " | |
| "return an all-zero mask and training would silently run at zero loss. " | |
| "Pass a tagged template copy via data.chat_template (see " | |
| "examples/MiniMaxAI/MiniMax-M3/m3_chat_template_generation.jinja for a " | |
| "worked example), register a loss-mask recovery " | |
| "(modelopt.torch.utils.loss_mask), or set answer_only_loss=false." | |
| ) | |
| raise RuntimeError( | |
| "answer_only_loss=True needs assistant masks, but the tokenizer's chat " | |
| "template has no {% generation %} tags, so apply_chat_template would " | |
| "return an all-zero mask and training would silently run at zero loss. " | |
| "Pass a tagged template copy via data.chat_template (see " | |
| "tools/launcher/examples/MiniMaxAI/MiniMax-M3/m3_chat_template_generation.jinja " | |
| "for a worked example), register a loss-mask recovery " | |
| "(modelopt.torch.utils.loss_mask), or set answer_only_loss=false." | |
| ) |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@modelopt/torch/speculative/plugins/hf_streaming_dataset.py` around lines 116
- 124, Update the error message in the assistant-mask validation guard to
reference the actual template location,
tools/launcher/examples/MiniMaxAI/MiniMax-M3/m3_chat_template_generation.jinja,
while leaving the surrounding guidance unchanged.
| environment: | ||
| - HF_MODEL_CKPT: <<global_vars.hf_model>> |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Add the required MLM_MODEL_CFG repository ID.
The new model configuration omits the launcher’s model-staging variable.
Proposed fix
environment:
- HF_MODEL_CKPT: <<global_vars.hf_model>>
+ - MLM_MODEL_CFG: "MiniMaxAI/MiniMax-M3"As per path instructions, new model configs must set MLM_MODEL_CFG to the Hugging Face repository ID.
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| environment: | |
| - HF_MODEL_CKPT: <<global_vars.hf_model>> | |
| environment: | |
| - HF_MODEL_CKPT: <<global_vars.hf_model>> | |
| - MLM_MODEL_CFG: "MiniMaxAI/MiniMax-M3" |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@tools/launcher/examples/MiniMaxAI/MiniMax-M3/hf_streaming_dspark_multi_node.yaml`
around lines 99 - 100, Add the required MLM_MODEL_CFG environment variable
alongside HF_MODEL_CKPT in the MiniMax-M3 configuration, setting it to the
model’s Hugging Face repository ID and preserving the existing HF_MODEL_CKPT
reference.
Source: Path instructions
| @@ -0,0 +1,250 @@ | |||
| {# ---------- special token variables ---------- #} | |||
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect the target file and related example templates for provenance/licensing patterns.
git ls-files 'tools/launcher/examples/MiniMaxAI/MiniMax-M3/*' | sed 's#^`#FILE`: #'
echo
wc -l tools/launcher/examples/MiniMaxAI/MiniMax-M3/m3_chat_template_generation.jinja
echo
cat -n tools/launcher/examples/MiniMaxAI/MiniMax-M3/m3_chat_template_generation.jinja | sed -n '1,80p'
echo
rg -n --hidden --no-messages 'SPDX-License-Identifier|copyright|license|provenance|source' tools/launcher/examples/MiniMaxAI/MiniMax-M3 tools/launcher/examples/MiniMaxAI -g '!**/*.pyc'Repository: NVIDIA/Model-Optimizer
Length of output: 5262
🌐 Web query:
MiniMaxAI MiniMax-M3 chat_template.jinja license source commit minimax-community
💡 Result:
The MiniMax-M3 model is released under the MiniMax Community License [1][2][3], which is not an unrestricted permissive open-source license [1]. The chat_template.jinja file for MiniMax-M3 is hosted in the official MiniMaxAI/MiniMax-M3 repository on Hugging Face [4][5]. As of July 12, 2026, the model repository indicates that the license governing its use is the "minimax-community" license [2][3]. This license typically includes specific terms such as attribution requirements and restrictions on certain commercial uses [1]. You can view the specific content of the chat template and track its history directly via the Hugging Face repository [4][6]. The repository for MiniMax-M3 is maintained under the MiniMaxAI organization on Hugging Face [4][3], and related code can also be found in the MiniMax-AI GitHub organization [7]. Always refer to the official LICENSE file [8] located in the repository for the most accurate and legally binding terms regarding the model's usage.
Citations:
- 1: https://minimax-ai.chat/models/minimax-m3/
- 2: https://huggingface.co/MiniMaxAI/MiniMax-M3/raw/ca22f9aa9ab883c93f0154f6f94e0f585ee60e21/README.md
- 3: https://huggingface.co/MiniMaxAI/MiniMax-M3
- 4: https://huggingface.co/MiniMaxAI/MiniMax-M3/blob/main/chat_template.jinja
- 5: https://huggingface.co/MiniMaxAI/MiniMax-M3/tree/main
- 6: https://huggingface.co/MiniMaxAI/MiniMax-M3/blob/2e2cee36b34856cb2a446d2f31677e51c97dbcbe/chat_template.jinja
- 7: https://github.com/MiniMax-AI/MiniMax-M3
- 8: https://huggingface.co/MiniMaxAI/MiniMax-M3/blob/main/LICENSE
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Check whether the repo already has license notices or hook exclusions for this file/path.
rg -n --no-messages 'MiniMax|minimax-community|third-party|Third-Party Software Notices|insert-license|m3_chat_template_generation\.jinja' LICENSE .pre-commit-config.yaml tools/launcher/examples/MiniMaxAI -g '!**/*.pyc'
echo
echo "LICENSE lines:"
wc -l LICENSE
echo
sed -n '1,260p' LICENSE | cat -n | sed -n '1,260p'
echo
echo ".pre-commit-config.yaml lines:"
wc -l .pre-commit-config.yaml
echo
sed -n '1,260p' .pre-commit-config.yaml | cat -n | sed -n '1,260p'Repository: NVIDIA/Model-Optimizer
Length of output: 26892
Add the MiniMax provenance and license notices here.
This template needs a source reference, the upstream copyright/license notice, and the NVIDIA Apache 2.0 header, plus an update to LICENSE for the MiniMax Community License.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@tools/launcher/examples/MiniMaxAI/MiniMax-M3/m3_chat_template_generation.jinja`
at line 1, Add the required provenance and licensing notices at the top of the
template before the special token variables: include the MiniMax source
reference, the upstream copyright and license notice, and the NVIDIA Apache 2.0
header. Also update the repository LICENSE to include the MiniMax Community
License.
Source: Coding guidelines
h-guo18
changed the title
Signed-off-by: h-guo18 <67671475+h-guo18@users.noreply.github.com>
h-guo18
force-pushed
the
haoguo/m3-dspark-recipe
branch
from
468013d to
b31352a
Compare
|
/claude review |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| "deepseek_v3": "rmsnorm", | ||
| "kimi_k2": "rmsnorm", # Kimi-K2 / K2-Thinking (DeepSeek-V3 arch) report model_type "kimi_k2" | ||
| "kimi_k25": "rmsnorm", # Kimi-K2.5 / K2.6 / K2.7 all report model_type "kimi_k25" | ||
| "minimax_m3_vl_text": "rmsnorm", # overridden to gemma_rmsnorm via use_gemma_norm below |
There was a problem hiding this comment.
[IMPORTANT ModeState] This new minimax_m3_vl_text → "rmsnorm" table entry is a silent-corruption fallback that undermines the fail-safe this file is built around.
The selection order in _select_final_norm_type is: use_gemma_norm first, then this table. So the entry is only ever consulted when use_gemma_norm is absent/falsy but model_type == "minimax_m3_vl_text". In that case it returns plain "rmsnorm", which drops the (1 + weight) scale — exactly the silent bias into the reconstructed logits / distillation target that the rest of this module (and the gpt_oss-disabled comment above) deliberately avoids.
The file's stated contract is "unlisted → no norm → hard error" precisely because a wrong norm is worse than no norm. Mapping M3 to rmsnorm as a fallback trades a loud RuntimeError (from _maybe_apply_base_final_norm) for a silent +1 drop.
Since every M3 config that reaches here is gemma-normed, either:
- Remove this entry so a
use_gemma_norm-less M3 falls through toNone→ hard error, or - Map it to
"gemma_rmsnorm"so the correct flavor is applied even if the flag is missing.
Both are safer than the current "rmsnorm".
There was a problem hiding this comment.
Claude Review Summary
Findings — CRITICAL: 0, IMPORTANT: 1, SUGGESTION: 0
Reviewed all 8 changed files (5 source/example, 1 test, 1 shell, 1 requirements). The four fixes are well-motivated and the per-fix rationale in the PR body / inline comments is unusually thorough. The Gemma-norm math (fp32 normalize -> *(1+weight) -> type_as) is correct and matches MiniMaxM3VLRMSNorm; the constructor signature matches _FinalRMSNorm so the registry dispatch is consistent; the transformers pin (<5.13) is consistent with pyproject.toml; and the SERVE_BLOCK_SIZE knob is a clean, opt-in addition.
Most impactful finding
[IMPORTANT ModeState] modeling_final_norm.py:93 — the new minimax_m3_vl_text -> "rmsnorm" table entry is a silent-corruption fallback. Because use_gemma_norm is checked first, this entry only fires when the gemma flag is absent — and then it applies plain RMSNorm, dropping the (1 + weight) scale and silently biasing the reconstructed distillation target. That is the exact failure mode this file's "unlisted -> hard error" design (and the disabled gpt_oss entry) exists to prevent. Map it to gemma_rmsnorm or remove the entry entirely so a flag-less M3 hard-errors rather than mis-norms.
Assessment
Low-to-moderate risk. The change is backward compatible (norm selection only affects use_gemma_norm=True models, the mask guard converts a silent zero-loss run into a loud error, and the block-size/pin changes are additive). The one IMPORTANT item is a latent fail-safe regression rather than a live bug, since M3 configs set use_gemma_norm in practice — but it is worth closing since defeating the safety net is precisely what the surrounding code warns against.
Note: CodeRabbit's pre-merge gate flags the hardcoded trust_remote_code=true in the YAML (SECURITY.md exception needed) and a slow-tokenizer edge in the mask guard — deferring those to CodeRabbit rather than duplicating.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1965 +/- ##
==========================================
- Coverage 77.92% 77.29% -0.63%
==========================================
Files 519 519
Lines 58217 58236 +19
==========================================
- Hits 45366 45015 -351
- Misses 12851 13221 +370
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
1 participant
What does this PR do?
Type of change: new example + bug fixes
Adds a MiniMax-M3 DSpark streaming training recipe under
tools/launcher/examples/MiniMaxAI/MiniMax-M3/, plus the four fixes it needs to actually run. Each fix addresses a failure mode that is silent or misleading without it:modeling_final_norm.py,modeling_fakebase.py): M3 uses a gemma-style final RMSNorm ((1 + weight)scale, fp32 multiply-then-cast). Selecting the norm bymodel_typealone picks plainrmsnorm— MiniMax's VL remote code coerces itsmodel_type-lesstext_configto mixtral — which silently drops the+1and corrupts the distillation target. New_FinalGemmaRMSNorm+ selection by the explicituse_gemma_normconfig flag (only MiniMax sets it).answer_only_loss(hf_streaming_dataset.py): with a fast tokenizer whose chat template has no{% generation %}tags,apply_chat_template(return_assistant_tokens_mask=True)only warns and returns an all-zero mask — training runs at zero loss on every sample with no other symptom. Now raises at tokenization with an actionable message.SERVE_BLOCK_SIZEknob (train_eagle_streaming.sh): nemo_run exports env values unquoted, so a multi-tokenSERVE_EXTRA_ARGS="--trust-remote-code --block-size 128"loses everything after the first token. M3's MSA sparse attention requires KV block 128 (ValueError: No common block size for 16at engine init otherwise), so--block-sizegets a dedicated single-token knob.transformerspin to<5.13(matchpyproject.toml): the old<5.4pin downgrades recent vLLM containers (e.g. transformers 5.12.1, which also provides in-treeminimax_m3_vl) and breaksvllm serve(ALLOWED_LAYER_TYPESneeds >=5.5.3).The example itself encodes the validated M3 specifics: generation-tagged chat template copy (required for
answer_only_loss— see fix 2), draft dims + baserope_theta=5e6set explicitly (not inherited), mask token 200063 (reserved slot; added tokens end at 200060),EAGLE_CAPTURE_IDS= draft defaulttarget_layer_ids+1+ final layer,trust_remote_codeat serve/export, and AWS-EFA NIXL notes (UCX segfaults at agent init on EFA nodes; LIBFABRIC required there).Usage
Testing
tests/unit/torch/speculative/plugins/: 129 passed inside a current vLLM x86_64 nightly container (transformers 5.12.1), including 3 new tests for the assistant-mask guard.input_idsidentical to the original template, mask covers exactly the assistant turns (think prefix + content + eos), contiguous, no user-prompt leak.ruff check/ruff format(0.15.20) clean.Before your PR is "Ready for review"
use_gemma_norm=True, previously mis-normed; the guard turns a silent zero-loss run into an error;SERVE_BLOCK_SIZEis opt-in; the pin relax widens the allowed range)🤖 Generated with Claude Code
Summary by CodeRabbit
SERVE_BLOCK_SIZEsupport for vLLM serve launches.