Add llava onevision 1.5 - #47795
Open
chengzheng345 wants to merge 4 commits into
Open
Conversation
Adds LlavaOnevision1_5Config/Model/ForConditionalGeneration: a Qwen3 text backbone with a custom RICE vision encoder (patch embed + per-image CLS token + 2D rotary + block-diagonal attention + patch merger), wired together Llava-style (masked_scatter, plain 1D position ids, no M-RoPE). - configuration_llava_onevision1_5.py: text/vision/top-level configs - modular_llava_onevision1_5.py -> modeling_llava_onevision1_5.py: vision tower built from qwen2_vl primitives, text model reuses Qwen3Model, multimodal wiring reuses LlavaModel/LlavaForConditionalGeneration - convert_llava_onevision1_5_weights_to_hf.py: converts the original trust_remote_code checkpoint (remaps visual.* / model.* keys; the checkpoint's lm_head.weight is untied from embed_tokens.weight) - Registers the model in the auto classes (AutoModel/AutoConfig family) and reuses Qwen2VLImageProcessor + Qwen2_5_VLProcessor for preprocessing - Adds docs page and model tests Verified end-to-end parity against the original implementation (lmms-lab/LLaVA-OneVision-1.5-4B-Instruct): max abs logit diff ~1e-4 (float32 numerical noise), identical top-5 token ranking, and AutoProcessor + AutoModelForImageTextToText.from_pretrained + generate all work correctly on the converted checkpoint.
Matches the convention used by LlavaModel (and the most recently added VLM, Inkling) for torch.compile/export-safe assertions, instead of a raw if/raise ValueError.
Contributor
|
Thank you for your contribution 🤗! CI Security Gate — automatic approval blockedThis PR was not automatically approved for CI because the security gate failed. Possible reasons:
See the workflow run for the exact violations. A maintainer can review and manually approve CI if a finding is a false positive. |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds native Hugging Face Transformers support for the LLaVA-OneVision-1.5 family (llava_onevision1_5), including configs, model implementation (modular + generated modeling), auto-mappings, docs, a conversion script, and an initial test suite.
Changes:
- Introduces
LlavaOnevision1_5*Configand the full PyTorch model stack (vision, text, composite, conditional generation) undersrc/transformers/models/llava_onevision1_5/. - Registers the new model/configs into Transformers auto-mappings (modeling + processing + image/video processing) and repo consistency checks.
- Adds model documentation and a basic modeling test.
Reviewed changes
Copilot reviewed 16 out of 18 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| utils/check_repo.py | Exempts new text/vision submodules from standalone test expectations. |
| utils/check_config_attributes.py | Allows temporal_patch_size as an intentional extra vision-config attribute. |
| tests/models/llava_onevision1_5/test_modeling_llava_onevision1_5.py | Adds initial modeling tests for the new model (incl. mismatch image-token error). |
| tests/models/llava_onevision1_5/init.py | Adds the new test package marker. |
| src/transformers/models/llava_onevision1_5/modular_llava_onevision1_5.py | Adds modular source for the new model (authoritative implementation for generation). |
| src/transformers/models/llava_onevision1_5/modeling_llava_onevision1_5.py | Adds generated modeling file produced from the modular definition. |
| src/transformers/models/llava_onevision1_5/convert_llava_onevision1_5_weights_to_hf.py | Adds checkpoint conversion utility from original safetensors layout to HF format. |
| src/transformers/models/llava_onevision1_5/configuration_llava_onevision1_5.py | Adds vision/text/composite config definitions for OV1.5. |
| src/transformers/models/llava_onevision1_5/init.py | Adds lazy import structure for the new model package. |
| src/transformers/models/auto/video_processing_auto.py | Registers llava_onevision1_5 video processor mapping. |
| src/transformers/models/auto/processing_auto.py | Registers llava_onevision1_5 processor mapping. |
| src/transformers/models/auto/modeling_auto.py | Registers the new model classes for AutoModel dispatch (base + conditional generation). |
| src/transformers/models/auto/image_processing_auto.py | Registers llava_onevision1_5 image processor mapping. |
| src/transformers/models/auto/auto_mappings.py | Registers config mappings and links text/vision sub-config model types to the base model family. |
| src/transformers/models/init.py | Exposes the new model package at the transformers.models level. |
| docs/source/en/model_doc/llava_onevision1_5.md | Adds model documentation page and usage snippet. |
| docs/source/en/_toctree.yml | Adds the new doc page to the sidebar TOC. |
| .gitignore | Ignores a local scratch directory used during porting. |
Suppressed comments (4)
src/transformers/models/llava_onevision1_5/modular_llava_onevision1_5.py:225
max_grid_size = grid_thw[:, 1:].max()is a scalar tensor, butLlavaOnevision1_5RiceRotaryEmbedding.forwardexpects anintseqlen (it passes the value totorch.arange). Convert the scalar to a Python int to avoid type errors.
max_grid_size = grid_thw[:, 1:].max()
rotary_pos_emb_full = self.rotary_pos_emb(max_grid_size)
rotary_pos_emb = rotary_pos_emb_full[pos_ids].flatten(1)
src/transformers/models/llava_onevision1_5/modular_llava_onevision1_5.py:290
- After inserting one
[CLS]token per segment, the code that removes the[CLS]tokens useshidden_states[seg_start + 1 : seg_end + 1], which does not account for the per-segment offset introduced by earlier insertions. This will include a segment’s[CLS]token in the output for all but the first segment and drop the last patch of each segment.
for i in range(1, num_segments + 1):
seg_start = cu[i - 1].item()
seg_end = cu[i].item()
new_hidden[seg_start:seg_end] = hidden_states[seg_start + 1 : seg_end + 1]
hidden_states = new_hidden
src/transformers/models/llava_onevision1_5/modeling_llava_onevision1_5.py:367
- (Generated file)
max_grid_sizeis a scalar tensor but is passed intoself.rotary_pos_emb(...), whose forward usestorch.arange(seqlen, ...)and expects anint. Convert with.item()to avoid type errors. Please apply the fix in the modular file and regenerate.
max_grid_size = grid_thw[:, 1:].max()
rotary_pos_emb_full = self.rotary_pos_emb(max_grid_size)
rotary_pos_emb = rotary_pos_emb_full[pos_ids].flatten(1)
src/transformers/models/llava_onevision1_5/modeling_llava_onevision1_5.py:432
- (Generated file) After inserting one
[CLS]token per segment, the removal step does not account for the per-segment offset, so later segments will copy the wrong slice (includes[CLS], drops last patch). Please apply the fix in the modular file and regenerate.
for i in range(1, num_segments + 1):
seg_start = cu[i - 1].item()
seg_end = cu[i].item()
new_hidden[seg_start:seg_end] = hidden_states[seg_start + 1 : seg_end + 1]
hidden_states = new_hidden
Comment on lines
+200
to
+222
| pos_ids = [] | ||
| for t, h, w in grid_thw: | ||
| hpos_ids = torch.arange(h).unsqueeze(1).expand(-1, w) | ||
| hpos_ids = hpos_ids.reshape( | ||
| h // self.spatial_merge_size, | ||
| self.spatial_merge_size, | ||
| w // self.spatial_merge_size, | ||
| self.spatial_merge_size, | ||
| ) | ||
| hpos_ids = hpos_ids.permute(0, 2, 1, 3) | ||
| hpos_ids = hpos_ids.flatten() | ||
|
|
||
| wpos_ids = torch.arange(w).unsqueeze(0).expand(h, -1) | ||
| wpos_ids = wpos_ids.reshape( | ||
| h // self.spatial_merge_size, | ||
| self.spatial_merge_size, | ||
| w // self.spatial_merge_size, | ||
| self.spatial_merge_size, | ||
| ) | ||
| wpos_ids = wpos_ids.permute(0, 2, 1, 3) | ||
| wpos_ids = wpos_ids.flatten() | ||
| pos_ids.append(torch.stack([hpos_ids, wpos_ids], dim=-1).repeat(t, 1)) | ||
| pos_ids = torch.cat(pos_ids, dim=0) |
Comment on lines
+342
to
+364
| pos_ids = [] | ||
| for t, h, w in grid_thw: | ||
| hpos_ids = torch.arange(h).unsqueeze(1).expand(-1, w) | ||
| hpos_ids = hpos_ids.reshape( | ||
| h // self.spatial_merge_size, | ||
| self.spatial_merge_size, | ||
| w // self.spatial_merge_size, | ||
| self.spatial_merge_size, | ||
| ) | ||
| hpos_ids = hpos_ids.permute(0, 2, 1, 3) | ||
| hpos_ids = hpos_ids.flatten() | ||
|
|
||
| wpos_ids = torch.arange(w).unsqueeze(0).expand(h, -1) | ||
| wpos_ids = wpos_ids.reshape( | ||
| h // self.spatial_merge_size, | ||
| self.spatial_merge_size, | ||
| w // self.spatial_merge_size, | ||
| self.spatial_merge_size, | ||
| ) | ||
| wpos_ids = wpos_ids.permute(0, 2, 1, 3) | ||
| wpos_ids = wpos_ids.flatten() | ||
| pos_ids.append(torch.stack([hpos_ids, wpos_ids], dim=-1).repeat(t, 1)) | ||
| pos_ids = torch.cat(pos_ids, dim=0) |
Contributor
|
[For maintainers] Suggested jobs to run (before merge) run-slow: auto, llava_onevision1_5 |
Contributor
CI recapDashboard: View test results in Grafana |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Add
llava_onevision1_5to TransformersWhat this PR adds
This PR introduces native Transformers support for LLaVA-OneVision-1.5 (
llava_onevision1_5), including:configuration,modular, generatedmodeling)convert_llava_onevision1_5_weights_to_hf.py)tests/models/llava_onevision1_5/)Target checkpoint used during integration:
lmms-lab/LLaVA-OneVision-1.5-4B-InstructKey implementation notes
During porting, two correctness issues were fixed in the modeling path:
Post-load re-initialization overwrite in
from_pretrainedpathclass_embedding/class_pos_emb.RiceRotaryEmbedding.inv_freqinitialization under meta-device fast loadingAlso aligned checkpoint behavior with:
tie_word_embeddings=False(matches OV1.5 checkpoint behavior with independentlm_head.weight).Validation summary
Main acceptance criterion: single-image numerical alignment
OV1.5 training is single-image-focused, so the primary parity gate is single-image alignment.
Completed checks:
Outcome:
max abs logit diff ~1e-4)Video and multi-image
Video and multi-image inputs were validated as smoke tests (functional path verification):
forward + generatepassforward + generatepassNote: These are treated as functional smoke checks rather than strict numerical parity gates in this PR.
8B compatibility smoke
To reduce reviewer risk regarding "same architecture, larger checkpoint", an 8B functional smoke check was also performed:
LLaVA-OneVision-1.5-8B-Instructforward + generatepassLocal checks run for this branch