[Bug] QwenImageEdit produces incorrect outputs with transformers >=5 due to missing mm_token_type_ids

[wutachiang]

Describe the bug

[Bug] QwenImage pipeline does not forward mm_token_type_ids for transformers 5.x

Describe the bug

When using QwenImageEdit with newer versions of transformers, the generated image output can differ from previous versions even with identical:

• model weights
• input image
• prompt
• random seed
• initial latent

The issue is caused by missing forwarding of mm_token_type_ids from the processor output to the Qwen2.5-VL text encoder.

Environment

Affected combinations:

• diffusers: 0.38.0+
• transformers: 5.x

Tested pipeline:

• QwenImageEdit pipeline

Root cause

transformers 5.x introduced mm_token_type_ids for explicit multimodal token identification in Qwen2.5-VL models.

This field is required by the model to construct correct multimodal 3D RoPE position ids.

The expected flow is:

processor output
input_ids
image_grid_thw
mm_token_type_ids
        |
        v
Qwen2.5-VL text encoder
        |
        v
3D RoPE position_ids

However, the current diffusers QwenImage pipeline only forwards:

input_ids
attention_mask
pixel_values
image_grid_thw

and does not forward:

mm_token_type_ids

As a result, with transformers 5.x, the model cannot determine the multimodal token ranges and falls back to the default position encoding path instead of constructing multimodal 3D RoPE.
The fallback does not raise an error because position_ids=None is a valid model input. However, for image-text generation, this changes the attention position information and leads to different outputs.

Evidence

With transformers 5.x:

• processor correctly generates mm_token_type_ids
• input_ids, attention_mask, pixel_values, and image_grid_thw are identical
• the first divergence appears during language model attention computation
• multimodal 3D RoPE is not constructed because mm_token_type_ids is missing

After manually forwarding mm_token_type_ids:

• 3D RoPE position ids become identical to the previous working path
• language hidden states become identical
• final generated image MD5 matches the baseline

Proposed fix

Forward mm_token_type_ids when available in the QwenImage pipeline:

outputs = self.text_encoder(
    input_ids=model_inputs["input_ids"],
    attention_mask=model_inputs["attention_mask"],
    pixel_values=model_inputs.get("pixel_values"),
    image_grid_thw=model_inputs.get("image_grid_thw"),
    mm_token_type_ids=model_inputs.get("mm_token_type_ids"),
    output_hidden_states=True,
)

This keeps compatibility with both:

• older transformers versions where this argument is not required
• newer transformers versions where multimodal RoPE depends on this field

Additional notes

This is a compatibility issue between the updated multimodal input interface in transformers 5.x and the current diffusers QwenImage pipeline.
The fix only forwards an existing processor output field and does not change model behavior for existing supported configurations.

Reproduction

Install the affected versions:

pip install diffusers==0.38.0
pip install transformers>=5.0.0

Run the following example:

import torch
from diffusers import QwenImageEditPipeline
from PIL import Image

model_id = "<your-qwen-image-model-id>"

pipe = QwenImageEditPipeline.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
)
pipe.to("cuda")

image = Image.open("input.png")

prompt = "Edit the image according to the instruction."

generator = torch.Generator(device="cuda").manual_seed(42)

output = pipe(
    image=image,
    prompt=prompt,
    generator=generator,
)

output.images[0].save("output.png")

The issue can be reproduced by checking the processor output:

model_inputs = pipe.processor(
    text=[prompt],
    images=[image],
    padding=True,
    return_tensors="pt",
)

print(model_inputs.keys())

With transformers >= 5.x, the processor returns:

input_ids
attention_mask
pixel_values
image_grid_thw
mm_token_type_ids

to the Qwen2.5-VL text encoder and drops

mm_token_type_ids

As a result, Qwen2.5-VL cannot construct multimodal 3D RoPE position ids and falls back to the default position encoding path.
The missing argument can be verified by adding:

mm_token_type_ids=model_inputs.get("mm_token_type_ids")

to the text_encoder call inside the QwenImage pipeline.
After forwarding mm_token_type_ids:

• multimodal 3D RoPE position ids are correctly generated
• language hidden states match the previous working configuration
• generated image output becomes consistent with the expected result

Logs

System Info

Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

• 🤗 Diffusers version: 0.38.0
• Platform: Linux-6.6.0-100.jd_b003.x86_64-x86_64-with-glibc2.35
• Running on Google Colab?: No
• Python version: 3.10.20
• PyTorch version (GPU?): 2.12.0+cu130 (True)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Huggingface_hub version: 1.17.0
• Transformers version: 5.10.1
• Accelerate version: 1.13.0
• PEFT version: 0.19.1
• Bitsandbytes version: not installed
• Safetensors version: 0.8.0
• xFormers version: not installed

Who can help?

No response