feat: update Minimax-H3 support - #77
Merged
Merged
Conversation
Stage 11 code files only: 573 insertions and 158 deletions. No benchmark scripts, generated assets, docs, or submodule changes are included. Align Qwen3-VL text attention with the sd.cpp-compatible non-flash path to restore Ref2VA reference-image fidelity. Add Ref2VA conditioning support for mixed reference images, videos, paired video audio, and standalone audio, including CLI validation that audio-only references are rejected. Add MiniMax H3 profiling and tensor-name mappings used to audit conditioning, diffusion, VAE decode, and model loading stages. Enable the validated CUDA/cuDNN and VAE fast paths by default while keeping environment variables available for opt-out debugging. Validation: full CUDA build completed in build-cuda-refvideo-align-20260810; reran Ref2VA image/video/video-audio/mixed benchmark and strong-prompt image rerun on CUDA.
Allow ed-cli --ref-video to accept either an existing frame directory or a media file such as mp4, mov, mkv, webm, or avi. Decode media references through ffmpeg into temporary PNG frames and cap extracted frames to the requested generation frame count to avoid expanding long videos unnecessarily. When a media reference carries embedded audio, extract it to 32 kHz stereo WAV and bind it to the same ed_ref_video_t unless an explicit --ref-video-audio is supplied for that video index. Document the updated Ref2VA CLI reference flags in --help while preserving the existing repeatable image, video, paired-audio, and standalone-audio behavior. Validation: rebuilt ed-cli in build-cuda-refvideo-align-20260810; smoke-tested MP4 reference input with embedded audio and without audio on CUDA; compared prompt-only temporal reversal against diffusers with matching seed, size, frames, and steps.
Advance the ggml submodule to b8a5ef0c. The update enables the validated Hopper Q4_K cuBLAS and F32 QKV/FC1 paths by default with explicit opt-outs, and adds a configurable long-sequence threshold that retains 16000 as the default.
Infer MiniMax-H3 for standalone and sharded safetensors, normalize Diffusers SwiGLU projection layout before quantization, and synthesize omitted RoPE frequencies at load time. Teach offline conversion to apply the transformer diffusion prefix, add the validated Diffusers benchmark runners, and preserve the Q4 Edge-vs-Diffusers and H200 8-bit Edge-vs-sd.cpp benchmark reports. The reports retain exact prompts, inputs, stage timings, VRAM peaks, and artifact links.
- Support complete Ref2VA weight layouts and improve MiniMax-H3 tensor mapping and conditioning.\n- Scope cuDNN SDPA and transposed-convolution acceleration to validated MiniMax-H3 operations.\n- Improve residency and auto-fit accounting for DiT, Qwen3-VL, video VAE, and audio VAE.\n- Refine SM90 Q8/Q4 projection routing through the updated ggml submodule.\n- Update the CLI and core MiniMax-H3 usage, model, and memory documentation.
- Align Ref2VA image/video preprocessing, validate aspect ratios, and build multimodal token sequences directly. - Add conditioning, Qwen Vision, reference VAE, diffusion, and decode phase profiling. - Enable validated fused video-VAE reflect padding, temporal group norm, Conv3D, and temporal assembly paths. - Update ggml for Qwen Vision correctness, temporal padding acceleration, and default SM90 Q8 FC2 cuBLAS routing. - Document the validated FC2, Qwen Vision, and visual VAE optimization gates.
Merge Flux2 Klein, standalone transformer conversion, website, and upstream ggml updates. Resolve model detection so bare MiniMax safetensors and official shard indexes remain loadable alongside the new Flux2 paths. Retain the current MiniMax pipeline and remove incoming ad-hoc Diffusers benchmark scripts.
Add a MiniMax-only --video-duration CLI option while retaining exact --video-frames control. Convert seconds at the fixed 24 FPS rate, align to the nearest legal 17k+5 frame count, reject ambiguous combinations, and use the resolved count for auto-fit and reference decoding. Refresh the formal MiniMax model, CLI, and usage documentation with verified BF16/Q8/Q4 sources, offline Q8 conversion commands, corrected full-versus-pruned checkpoint names, concise inference examples, and clearly separated pre-alignment versus Diffusers-aligned H200 generation/VRAM data. Remove legacy no-op flags from examples and untrack non-formal MiniMax benchmark/handoff documents without deleting their local copies.
Document full and pruned BF16 DiT loading, Q8_0 conversion, and downloadable pruned checkpoint sources. Replace stale Ref2VA resize-era measurements with current BF16 results showing Edge ahead of Diffusers. Refresh README model coverage, latest news, and H200 performance summary.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.