[Multimodal][Model] Make Qwen3.5-VL work with packed sequences

[TobyYang7]

Qwen3.5-VL cannot currently run through the bridge path, and the example still installs a forked Megatron Bridge to work around it. This makes it run against the official package.

What is broken today

The Qwen3.5-VL providers build their Gated DeltaNet layers from megatron-core's experimental_attention_variant="gated_delta_net", and megatron-core's GatedDeltaNet.forward refuses packed sequences outright:

if packed_seq_params is not None:
    raise NotImplementedError("GDN does not support packed sequence for now.")

Since #2100 removed BSHD, every microbatch is packed THD, so the first GDN layer raises. Reproduced on main with Qwen/Qwen3.5-2B and the current image:

[unpatched] NotImplementedError: GDN does not support packed sequence for now.

Megatron Bridge has shipped Qwen35VLBridge / Qwen35VLMoEBridge since v0.4.0 and registers them on import, so nothing needs to be registered on our side — the missing piece is only the interaction with our data layout.

Approach

With --micro-batch-size 1 and no dynamic batching, which GDN already requires, a THD microbatch is a single sequence followed by right padding, and the hidden states are [T, 1, H] — exactly the layout the unpacked path expects. GDN is causal, so the trailing padding cannot affect the outputs of the real tokens. So the module is subclassed and the packed metadata dropped, but only after checking that we really are in that regime.

The subclass adds no parameters and renames nothing, so every official GDN weight mapping (in_proj / conv1d / A_log / dt_bias / out_norm / out_proj) keeps working for checkpoint load, --save-hf and weight sync to SGLang. AutoMapping dispatches on the exact class name, so the subclass is registered the same way megatron-bridge registers GatedDeltaNet itself.

Rejected rather than silently miscomputed:

• context parallel — each rank holds a zigzag slice while cu_seqlens stays global, and the recurrence cannot be split that way;
• micro batch size > 1 and dynamic batching — several real sequences in one microbatch would be fused into a single recurrent stream. This one cannot be detected from the packed metadata, since one sequence plus padding and two sequences both expose two cu_seqlens segments, so it is checked when the model is built.

Lifting the last restriction means forwarding cu_seqlens into the varlen kernels instead of dropping it; that seemed worth keeping separate from this change.

Verification

Qwen/Qwen3.5-2B, Megatron Bridge 0.5.0, megatron-core 0.16.0rc0, single node, slimerl/slime:latest.

check	result
unpatched	raises in the first GDN layer
packed forward, TP=1 and TP=2	logits agree with HuggingFace on 47/48 argmax positions (bf16), every Megatron top-1 inside the HuggingFace top-5
image forward	succeeds with real pixel_values and image_grid_thw
export to HuggingFace	621/632 tensors, 0 unexpected, including all 162 GDN tensors; max weight difference 0.0039, one bf16 ulp. The 11 absent tensors are MTP layers this configuration does not build
TP=2 sharding	layers.0.self_attention.in_proj.weight is half size per rank, i.e. the AutoMapping registration takes effect
gradients, TP=2, sequence length 512	finite throughout, including all 90 GDN parameters
optimizer step, TP=2	completes, grad_norm 135.6

Not covered: MoE, weight sync against a live SGLang engine, and PP above 1.

One note that cost me some time and is now in the README: the gradient checks need the example's --attention-backend flash. Letting Megatron choose the backend selects Transformer Engine's cuDNN fused attention, whose backward goes non-finite for this model under bf16 with packed sequences and TP above 1. Qwen3-VL is unaffected under the same settings.

Also in this PR

--moe-aux-loss-coeff and new --freeze-language-model / --freeze-vision-model / --freeze-vision-projection flags are forwarded to bridge providers when the provider defines them, so examples/geo3k_vlm/README.md no longer has to ask users to hand-edit model_provider.py. The forwarding is a name allowlist with hasattr checks on both sides, so it is a no-op for providers without those fields.

examples/geo3k_vlm/run_geo3k_qwen35.sh drops the coding-famer/Megatron-Bridge-slime@qwen35 install.

Tests are CPU-only (NUM_GPUS = 0) and registered in pr-test.yml.j2 with the workflow regenerated.

This overlaps with #2075, which takes the registration-shim approach; that path stops at the GDN refusal above, which is why this one goes further.