feat: optimize minimax support and add demo - #79
Merged
Conversation
- fix Qwen3-VL Chinese tokenization and disable the incorrect long-sequence masked flash-attention path - restore Qwen3-VL vision patch and attention biases, including cuDNN Conv3D same-dtype bias application - add selectable Euler/RES samplers and discrete/simple schedules, and correct RES audio carry scaling - expose reference-image sizing controls and document the four-image Ref2VA validation and performance results
Implement a MiniMax-H3-specific phase lifecycle that stages Qwen and conditioning VAEs, keeps the DiT resident across denoise steps, and reloads decode VAEs only after releasing the DiT. Harden ultra-long Ref2VA execution with synchronous cuDNN SDPA planning, mask elision, 64-bit ggml Flash Attention addressing, and flattened RMSNorm grids. Preserve already-low source quantization during auto-fit instead of increasing Q4 inputs to Q8. Extend the official Diffusers runner with component metrics, configurable FPS, 362-frame support, and JSON profiles. Clarify that max-vram is a planning budget rather than a hard process cap. Publish two aligned 15-second Edge-DiT.cpp/ComfyUI/Diffusers stacked demos with six reference assets, ComfyUI API graphs, Diffusers profiles, lifecycle and 24 GiB metrics, plus Ref2VA/FL2VA/Wan regression results.
Record a complete 20-step Ref2VA run with max-vram 24 and auto-allocate: 39,505 MiB sampled peak, 854.183s DiT, 911.731s generation, and 935.137s end to end. Add resident-versus-budget quality measurements and a labeled side-by-side video. The graph-cut path preserves visual quality but remains above a physical 24 GiB limit.
Document that the published Edge and ComfyUI demos used match-sized references while official Diffusers upscaled every image to a 2048-pixel short edge, producing 40-79x larger reference image areas and non-comparable DiT workloads. Clarify that Edge max mode matches Diffusers geometry, while ComfyUI max does not upscale small inputs. Remove seven intermediate workflow/profile JSON attachments and retain only the report, six inputs, and three comparison videos.
Implement four focused changes across 27 tracked paths: 1. Guard single-device CUDA allocations for --auto-fit --max-vram, including ggml buffers and MiniMax cuDNN convolution/SDPA workspaces, with 1 GiB reserved for external workspaces and safe rejection of oversized graphs. 2. Add automatic MiniMax phase lifecycle staging so Qwen and conditioning VAEs are released before denoising, the DiT is released before decode, and decode VAEs load only for their phase. 3. Reduce long-sequence Ref2VA graph memory with split QKV projections, BF16 flash-attention inputs, MLP token chunking, and an attention/MLP graph cut. 4. Document Diffusers/ComfyUI reference-image resize differences, component precision mixing and acceleration limits, VAE storage precision, and exact max-vram semantics; remove two temporary reports and nine demo assets that should not live in the repository.
- stage Qwen, video VAE, and audio VAE sequentially during conditioning, then release each component before the next phase\n- avoid duplicate parameter staging when a component is already resident and preserve async offload behavior for segmented models\n- remove the incorrect initial RES audio-noise x4 scaling while retaining per-step audio carry conversion\n- keep auto-fit from increasing already lower-precision DiT tensors and document TE/DiT versus VAE policy precedence\n- expose 'preserve' as the clear default --type spelling while retaining 'auto' as a compatibility alias\n\nValidated with the current CUDA build, Ref2VA regular/24 GiB runs, and all four FL2VA Q8 workflows with generated speech/audio.
- describe reference-image sizing only in terms of edge-dit.cpp modes\n- remove ComfyUI template and VAE implementation commentary from the user guide\n- add the recommended res_multistep/simple pair to every runnable example\n- replace internal benchmark wording with user-facing Q8_0 conversion guidance\n- retain Comfy-Org only where it names the actual weight repository\n\nValidated documented model filenames against the Hugging Face mirror, checked every project flag against ed-cli help, and syntax-checked all bash blocks.
Add the two 15-second Ref2VA three-framework quality reels and machine-readable benchmark metrics as stable documentation assets.\n\nDocument the real 1280x736 and 736x1280 workflow geometry, model representations, per-stage timings, reference-resize differences, and allocator/device-memory measurement caveats.\n\nProfile the missing four-image ComfyUI run at 0.2-second intervals, recording a 49,413 MiB peak and 1,055.354-second wall time, and verify that its decoded video frames and PCM audio are bit-identical to the original result.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.