feat: Qwen3.6 target tensor parallelism - #532
Conversation
There was a problem hiding this comment.
All reported issues were addressed across 22 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
0cf3b47 to
3eb9d70
Compare
82db48c to
d538e6c
Compare
|
Rebased onto current main and updated this branch at d538e6c. Additional fixes preserve centralized placement validation (2-16 unique, local, homogeneous CUDA devices; dense Qwen3.6; no weights, remote shard, PFlash, or draft), document the vendored lucebox-ggml#39 source, and extend test_cuda_comm_api with an opt-in real selected-device all-reduce check via DFLASH_TP_TEST_DEVICES. Validation on lucebox3: CUDA build plus 336/336 relevant tests; HIP build plus 324/324 relevant tests. lucebox3 has one NVIDIA GPU and this CUDA build has no NCCL, so the real two-NVIDIA reduction path compiled but could not be executed there. |
Add target-only tensor parallelism for dense Qwen3.5/Qwen3.6 models through
the GGML meta backend.
The public interface is:
layerremains the default. Tensor mode validates homogeneous local CUDAdevices and rejects incompatible weighted placement, IPC, PFlash, and draft
configurations. The implementation splits attention, fused projections,
DeltaNet, FFN, KV cache, and recurrent state tensors. The LM head remains
mirrored because Lucebox performs argmax inside the target graph.
Verification
and placement validation.
--draftin tensor mode; DFlash support is inthe stacked follow-up.
Sustained stock-power results
Each configuration processed the same 26,758-token OpenClaw payload and
generated all 1,024 requested tokens with Q8 KV, no cache, and 98,304 context.
The RTX 5090 used its 600 W limit and each RTX 3090 used its stock 350 W limit.
These are single sustained runs per configuration. Peak sampled power was
552.9 W on the RTX 5090 and 349.8 W on the single RTX 3090. Q6 on the 5090
peaked at 31,441 MiB VRAM.
This requires this PR: Luce-Org/lucebox-ggml#39