Add Ideogram support and improve BF16 dequantization handling

README.md

@@ -1,29 +1,27 @@
 # ComfyUI-GGUF
-GGUF Quantization support for native ComfyUI models
+GGUF Quantization support for native ComfyUI models including the custom Q8_CR
 
-This is currently very much WIP. These custom nodes provide support for model files stored in the GGUF format popularized by [llama.cpp](https://github.com/ggerganov/llama.cpp).
+> [!NOTE]  
+> This is a fork of the original nodes, updated to support loading Ideogram 4 GGUFs and Krea 2 GGUFs. 
+> To use it, clone `https://github.com/city96/ComfyUI-GGUF`and not the original repo.
 
 While quantization wasn't feasible for regular UNET models (conv2d), transformer/DiT models such as flux seem less affected by quantization. This allows running it in much lower bits per weight variable bitrate quants on low-end GPUs. For further VRAM savings, a node to load a quantized version of the T5 text encoder is also included.
 
-![Comfy_Flux1_dev_Q4_0_GGUF_1024](https://github.com/user-attachments/assets/70d16d97-c522-4ef4-9435-633f128644c8)
-
-Note: The "Force/Set CLIP Device" is **NOT** part of this node pack. Do not install it if you only have one GPU. Do not set it to cuda:0 then complain about OOM errors if you do not undestand what it is for. There is not need to copy the workflow above, just use your own workflow and replace the stock "Load Diffusion Model" with the "Unet Loader (GGUF)" node.
-
 ## Installation
 
 > [!IMPORTANT]  
-> Make sure your ComfyUI is on a recent-enough version to support custom ops when loading the UNET-only.
+> Make sure your ComfyUI is on v0.27.0 or later.
 
 To install the custom node normally, git clone this repository into your custom nodes folder (`ComfyUI/custom_nodes`) and install the only dependency for inference (`pip install --upgrade gguf`)
 
 ```
-git clone https://github.com/city96/ComfyUI-GGUF
+git clone https://github.com/molbal/ComfyUI-GGUF
 ```
 
 To install the custom node on a standalone ComfyUI release, open a CMD inside the "ComfyUI_windows_portable" folder (where your `run_nvidia_gpu.bat` file is) and use the following commands:
 
 ```
-git clone https://github.com/city96/ComfyUI-GGUF ComfyUI/custom_nodes/ComfyUI-GGUF
+git clone https://github.com/molbal/ComfyUI-GGUF ComfyUI/custom_nodes/ComfyUI-GGUF
 .\python_embeded\python.exe -s -m pip install -r .\ComfyUI\custom_nodes\ComfyUI-GGUF\requirements.txt
 ```
 
@@ -35,15 +33,144 @@ Simply use the GGUF Unet loader found under the `bootleg` category. Place the .g
 
 LoRA loading is experimental but it should work with just the built-in LoRA loader node(s).
 
-Pre-quantized models:
+Pre-quantized models (🍴 icon on ones added by this fork):
 
 - [flux1-dev GGUF](https://huggingface.co/city96/FLUX.1-dev-gguf)
 - [flux1-schnell GGUF](https://huggingface.co/city96/FLUX.1-schnell-gguf)
 - [stable-diffusion-3.5-large GGUF](https://huggingface.co/city96/stable-diffusion-3.5-large-gguf)
 - [stable-diffusion-3.5-large-turbo GGUF](https://huggingface.co/city96/stable-diffusion-3.5-large-turbo-gguf)
+- [Krea 2 (Both Turbo and Raw)](https://huggingface.co/molbal/krea2-gguf) 🍴
+- [Ideogram 4](https://huggingface.co/molbal/ideogram-4-gguf) 🍴
+
+
+> [!IMPORTANT]  
+> Please note, that this fork does not support _K quants on diffusion models, only on text encoders. They may or may not load, but inference speed may be very slow. There may be other forks, or other custom nodes with better support for these quantization types.
 
 Initial support for quantizing T5 has also been added recently, these can be used using the various `*CLIPLoader (gguf)` nodes which can be used inplace of the regular ones. For the CLIP model, use whatever model you were using before for CLIP. The loader can handle both types of files - `gguf` and regular `safetensors`/`bin`.
 
 - [t5_v1.1-xxl GGUF](https://huggingface.co/city96/t5-v1_1-xxl-encoder-gguf)
+- [Qwen3-VL-4B-Instruct-GGUF](https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct-GGUF)🍴
 
 See the instructions in the [tools](https://github.com/city96/ComfyUI-GGUF/tree/main/tools) folder for how to create your own quants.
+
+## Converting Krea 2 and Ideogram 4 models
+
+The converter detects supported Krea 2 and Ideogram 4 checkpoints directly.
+Provide an existing `.safetensors`, `.ckpt`, `.pt`, `.pth`, or `.bin` diffusion
+model file; no model-specific conversion script is required.
+
+Run these commands from the `ComfyUI-GGUF` directory, replacing the source and
+destination paths with your model names:
+
+```bash
+# Compact standard GGUF
+python tools/convert.py --src /path/to/krea2_or_ideogram.safetensors \
+  --dst /path/to/model-Q4_0.gguf --quant-type Q4_0
+
+# Higher-quality standard GGUF
+python tools/convert.py --src /path/to/krea2_or_ideogram.safetensors \
+  --dst /path/to/model-Q8_0.gguf --quant-type Q8_0
+
+# Recommended for RTX 30-series NVIDIA GPUs
+python tools/convert.py --src /path/to/krea2_or_ideogram.safetensors \
+  --dst /path/to/model-Q8_CR.gguf --quant-type Q8_CR
+```
+
+For the portable Windows distribution, use its embedded Python executable:
+
+```bat
+.\python_embeded\python.exe .\ComfyUI\custom_nodes\ComfyUI-GGUF\tools\convert.py ^
+  --src C:\path\to\krea2_or_ideogram.safetensors ^
+  --dst C:\path\to\model-Q8_CR.gguf --quant-type Q8_CR
+```
+
+Place the resulting GGUF in `ComfyUI/models/unet` or
+`ComfyUI/models/diffusion_models`, then load it with **Unet Loader (GGUF)**.
+
+## Supported conversion formats
+
+The standard GGUF formats use the package's normal GGML loader path. `Q8_CR`
+is a custom ComfyUI-native INT8 layout for eligible Linear weights.
+
+| Format | Storage / execution | Recommended use |
+| --- | --- | --- |
+| `F16` | FP16 GGUF | Maximum compatibility with half-precision storage. |
+| `BF16` | BF16 GGUF | Preserve BF16 source models where the target supports BF16. |
+| `Q8_0` | Standard GGML 8-bit quantization | Excellent general-quality 8-bit GGUF; portable and straightforward to offload. |
+| `Q5_1` | Standard GGML 5-bit quantization | Lower storage with a quality-oriented 5-bit format. |
+| `Q5_0` | Standard GGML 5-bit quantization | Lower storage alternative to `Q5_1`. |
+| `Q4_1` | Standard GGML 4-bit quantization | Smaller files when VRAM or RAM is constrained. |
+| `Q4_0` | Standard GGML 4-bit quantization | Smallest supported standard diffusion-model format; expect the largest quality trade-off. |
+| `Q8_CR` | Per-row INT8 ConvRot through ComfyUI native ops | **Maintainer recommendation for NVIDIA RTX 30-series systems.** |
+
+
+> [!WARNING]
+> `_K` quant formats are not supported for diffusion models; they are supported only for text encoders.
+
+## Native weight-only quantization
+
+The converter supports one custom global quantization mode for DiT/transformer
+UNets: `Q8_CR`.
+
+### What Q8_CR is, why it exists, and how it works
+
+`Q8_CR` is an INT8 weight-only format designed for transformer-style diffusion
+models. Its purpose is to reduce GGUF model storage and VRAM pressure while
+preserving the fast native INT8 Linear operations available on supported NVIDIA
+GPUs. It is especially useful when the full diffusion model does not fit in
+VRAM and ComfyUI needs to offload weights to CPU memory.
+
+During conversion, the converter:
+
+1. Selects eligible 2-D Linear weights. One-dimensional tensors, small
+   tensors, and architecture-designated sensitive tensors stay FP32; Conv2d
+   weights stay FP16.
+2. Applies the compatible ConvRot/Hadamard rotation to each eligible weight
+   matrix.
+3. Quantizes the rotated weights to INT8 using an FP32 scale for every output
+   row.
+4. Stores the INT8 payload, scales, and ConvRot metadata in the GGUF file.
+
+During loading, the GGUF loader recognizes this metadata and passes the raw
+INT8 weights and row scales to ComfyUI's `TensorWiseINT8Layout`. On supported
+CUDA systems, ComfyUI executes the native INT8/ConvRot Linear path directly;
+it does not first expand the weight matrix to FP16 or BF16. The ordinary GGUF
+container still provides memory-mapped loading and CPU offload behavior.
+
+- `Q8_CR` stores eligible 2-D Linear weights as per-row INT8 ConvRot. It uses
+  ComfyUI's native `TensorWiseINT8Layout` path, so weights remain INT8 during
+  inference.
+
+Q8_CR keeps 1-D, small, and architecture-designated high-precision tensors
+in FP32. Conv2d weights remain FP16 because these modes accelerate Linear
+matrix multiplication only.
+
+### Maintainer recommendation: Q8_CR on RTX 30-series
+
+For NVIDIA RTX 30-series systems, the maintainer recommends `Q8_CR` for Krea 2
+and Ideogram 4. It combines:
+
+- Fast native INT8 operations on these GPUs through ComfyUI's ConvRot backend.
+- GGUF's convenient CPU offload and memory-mapped model storage behavior.
+- The generally excellent image quality expected from 8-bit GGUF
+  quantization, while retaining selected sensitive tensors in higher precision.
+
+Use `Q8_0` instead when you need the conventional portable GGML 8-bit format.
+Use `Q4_0` primarily when the smaller model footprint matters more than
+quality or sampling speed.
+
+### Q8_CR platform support
+
+Q8_CR does not require CUDA. It uses ComfyUI's `comfy_kitchen` layout backend:
+
+- NVIDIA CUDA uses ComfyUI's optimized native INT8 backend when available.
+- Linux and non-CUDA environments use the `comfy_kitchen` eager backend.
+- CPU Q8_CR loading and inference are supported, but naturally slower than
+  optimized CUDA inference.
+
+Q4_PT is retired from conversion and loading until a performant W4A16 backend
+is available. Its experimental implementation remains in the source for future
+work, but it is no longer selectable or loadable.
+
+Reconvert any `Q8_CR` GGUF created before ConvRot weights were marked as
+pre-rotated. Older files load safely with native non-rotated INT8 instead.

__init__.py

@@ -1,3 +1,5 @@
+WEB_DIRECTORY = "./web"
+
 # only import if running as a custom node
 try:
     import comfy.utils
@@ -6,4 +8,4 @@
 else:
     from .nodes import NODE_CLASS_MAPPINGS
     NODE_DISPLAY_NAME_MAPPINGS = {k:v.TITLE for k,v in NODE_CLASS_MAPPINGS.items()}
-    __all__ = ['NODE_CLASS_MAPPINGS', 'NODE_DISPLAY_NAME_MAPPINGS']
+    __all__ = ['NODE_CLASS_MAPPINGS', 'NODE_DISPLAY_NAME_MAPPINGS', 'WEB_DIRECTORY']

dequant.py

@@ -18,6 +18,9 @@ def dequantize_tensor(tensor, dtype=None, dequant_dtype=None):
 
     if qtype in TORCH_COMPATIBLE_QTYPES:
         return tensor.to(dtype)
+    elif qtype == gguf.GGMLQuantizationType.BF16:
+        tensor = torch.Tensor(tensor.data.view(torch.bfloat16).reshape(oshape))
+        return tensor if dtype is None or dtype == torch.bfloat16 else tensor.to(dtype)
     elif qtype in dequantize_functions:
         dequant_dtype = dtype if dequant_dtype == "target" else dequant_dtype
         return dequantize(tensor.data, qtype, oshape, dtype=dequant_dtype).to(dtype)

editor.md

@@ -0,0 +1,60 @@
+# Prompt Canvas Editor
+
+A single-file browser editor for building Ideogram-style structured JSON prompts with canvas-based bounding boxes. The app lives entirely in `ui.html`; there is no build step, package manager, or local server requirement.
+
+## Features
+
+- Set canvas width and height with sliders or by double-clicking the displayed values.
+- Draw, move, resize, delete, and edit bounding boxes directly on the canvas.
+- Cycle through selected boxes with the `<-` and `->` controls when boxes overlap.
+- Edit global prompt fields, including high-level description, aesthetics, lighting, medium, style/photo mode, background, and color palette.
+- Edit per-box mode, description, optional text content, and per-box color palette.
+- Generate formatted JSON from the current canvas and form state.
+- Paste existing prompt JSON into the JSON box and load it back into the editable canvas.
+
+## Usage
+
+Open `ui.html` directly in a modern browser.
+
+The Tailwind design system is loaded from the Tailwind CDN, so the page needs internet access for styling. The editor logic itself is plain HTML, CSS, and JavaScript.
+
+## Basic Workflow
+
+1. Set the canvas size.
+2. Draw boxes on the canvas by clicking and dragging.
+3. Select a box and edit its properties in the right panel.
+4. Fill in the global prompt settings.
+5. Click `Generate JSON` to write the prompt JSON into the textarea.
+6. Copy or save the generated JSON wherever your workflow needs it.
+
+To edit an existing prompt, paste the JSON into the textarea and click `Load JSON`. The editor will rebuild the canvas boxes and form fields from the prompt.
+
+## JSON Shape
+
+The editor expects prompt JSON in this general form:
+
+```json
+{
+  "high_level_description": "",
+  "style_description": {
+    "aesthetics": "",
+    "lighting": "",
+    "medium": "",
+    "art_style": "",
+    "color_palette": []
+  },
+  "compositional_deconstruction": {
+    "background": "",
+    "elements": [
+      {
+        "type": "obj",
+        "bbox": [0, 0, 1000, 1000],
+        "desc": "",
+        "color_palette": []
+      }
+    ]
+  }
+}
+```
+
+Bounding boxes use normalized coordinates from `0` to `1000` in `[y1, x1, y2, x2]` order. The editor converts those coordinates to the current canvas size when loading JSON, then converts them back to normalized coordinates when generating JSON.