feat: load ConvRot int4 and per-channel int8 safetensors

docs/external_quantization.md

@@ -0,0 +1,99 @@
+# Externally quantized safetensors
+
+Some checkpoints ship weights that were already quantized by another toolchain,
+storing the payload as raw bytes plus sidecar tensors that describe how to
+dequantize it. Such files are loaded directly: each weight is repacked once, at
+load time, into a native ggml type, so inference uses the ordinary kernels and no
+backend-specific support is required.
+
+| On-disk form | Repacked to | Lossless? |
+| --- | --- | --- |
+| ConvRot `convrot_w4a4` (int4) | `Q4_0` | yes, apart from F32→F16 on the scale |
+| ComfyUI `int8_tensorwise`, ConvRot with `linear_dtype: int8` | `Q8_0` | yes, apart from F32→F16 on the scale |
+| bitsandbytes NF4 | `F32` | yes (the codebook is non-uniform, so no 4-bit ggml type matches it) |
+
+Because NF4 decodes to F32, pass `--type` to requantize it to something compact,
+e.g. `--type q4_K`. The int4/int8 paths are already compact and need no `--type`.
+
+## ConvRot
+
+[ConvRot](https://arxiv.org/abs/2512.03673) is a 4-bit scheme for diffusion
+transformers. It multiplies each weight by a fixed orthogonal rotation before
+quantizing, which spreads outliers across the group and makes 4-bit weights far
+more accurate.
+
+The consequence for inference is that the stored weight is `W·H`, not `W`. Since
+
+```
+y = x·Wᵀ = x·(W'·H)ᵀ = (x·H)·W'ᵀ
+```
+
+the activation must be rotated by the same `H` before the matmul. Skipping this
+does not degrade quality gracefully — it produces noise.
+
+`H` is a block-diagonal regular-Hadamard transform over `convrot_groupsize` input
+channels (256 by default; the group size must be a power of four). It is built
+from the radix-4 core
+
+```
+M = [[ 1,  1,  1, -1],
+     [ 1,  1, -1,  1],
+     [ 1, -1,  1,  1],
+     [-1,  1,  1,  1]]
+```
+
+Kronecker-composed `log4(groupsize)` times and scaled by `1/sqrt(groupsize)`.
+That makes `H` symmetric as well as orthogonal, so no transpose bookkeeping is
+needed. The rotation is applied as a single matmul against a shared constant
+matrix, costing `groupsize / out_features` extra FLOPs — under 5% for a
+6144-wide layer.
+
+Note that activations stay in F32 here, so this is W4A16 rather than the paper's
+W4A4: lower throughput than a dedicated 4-bit-activation kernel, but higher
+accuracy.
+
+A layer is only rotated when the checkpoint marks it and `in_features` divides
+the group size; ConvRot leaves the remaining layers unrotated and omits the
+marker, which is honoured.
+
+**LoRA is not supported on ConvRot layers.** LoRA deltas are trained against the
+unrotated weight and cannot be mixed into a rotated activation, so adapters are
+skipped on those layers with a warning rather than applied incorrectly.
+
+## Recognized layouts
+
+ComfyUI-style, read from the safetensors `__metadata__` key
+`_quantization_metadata`:
+
+```json
+{"format_version": "1.0",
+ "layers": {"model.diffusion_model.blocks.0.attn.wq":
+            {"format": "convrot_w4a4", "convrot_groupsize": 256}}}
+```
+
+with `<layer>.weight` holding the packed payload and `<layer>.weight_scale`
+holding one F32 scale per output channel.
+
+bitsandbytes NF4 is detected from its own sidecars —
+`<layer>.weight.quant_state.bitsandbytes__nf4` (which carries the logical shape
+the packed tensor has lost), `.absmax`, `.quant_map`, and the nested pair used
+when the absmax is itself double-quantized.
+
+Byte tensors that no configuration claims are dropped, so leftover metadata blobs
+do not reach the model.
+
+## Example
+
+Krea-2 Turbo quantized to ConvRot int4:
+
+```
+sd-cli --diffusion-model Krea2_INT8.safetensors \
+       --llm Qwen3-VL-4B-Instruct-Q4_K_M.gguf \
+       --vae wan_2.1_vae.safetensors \
+       -p "a lovely cat holding a sign says 'krea2.cpp'" \
+       --diffusion-fa -v
+```
+
+At ~4.5 bits per weight the diffusion model is roughly a third the size of Q8_0.
+If it still does not fit, cap what stays resident with `-ngl` (see
+[backend.md](backend.md)).

docs/krea2.md

@@ -10,6 +10,8 @@ Krea2 uses a Krea2 diffusion transformer, the Wan2.1 VAE, and Qwen3-VL 4B as the
 - Download Krea2 Turbo
     - safetensors: https://huggingface.co/krea/Krea-2-Turbo/tree/main
     - gguf: https://huggingface.co/realrebelai/KREA-2_GGUFs/tree/main/TURBO
+- Krea2 also loads ConvRot int4 safetensors directly (roughly a third the size of
+  Q8_0); see [external_quantization.md](external_quantization.md)
 - Download vae
     - safetensors: https://huggingface.co/Comfy-Org/Wan_2.1_ComfyUI_repackaged/blob/main/split_files/vae/wan_2.1_vae.safetensors
 - Download Qwen3-VL 4B

src/convert.cpp

@@ -60,6 +60,17 @@ static bool collect_tensors_for_export(ModelLoader& model_loader,
     tensors.reserve(model_loader.get_tensor_storage_map().size());
     for (const auto& kv : model_loader.get_tensor_storage_map()) {
         const TensorStorage& tensor_storage = kv.second;
+        // A ConvRot weight is stored in a rotated basis and is only correct when
+        // the runtime rotates the activation to match. No output container here
+        // can carry that marker, so exporting would produce a file that loads
+        // cleanly and generates noise.
+        if (tensor_storage.convrot_groupsize() > 0) {
+            LOG_ERROR(
+                "cannot convert '%s': ConvRot-rotated weights would lose their rotation marker "
+                "and the result would be silently wrong. Use the source model directly.",
+                tensor_storage.name.c_str());
+            return false;
+        }
         TensorExportInfo info;
         info.storage = tensor_storage;
         info.type    = get_export_tensor_type(model_loader, tensor_storage, type, tensor_type_rules);

src/core/ggml_extend.hpp

@@ -14,6 +14,7 @@
 #include <iterator>
 #include <map>
 #include <memory>
+#include <mutex>
 #include <optional>
 #include <random>
 #include <regex>
@@ -3316,6 +3317,65 @@ class Identity : public UnaryBlock {
     }
 };
 
+// ConvRot (arXiv 2512.03673) stores W*H rather than W, where H is a block-diagonal
+// rotation over `group` input channels. Recovering y = x*W^T therefore means
+// rotating the activation by the same H before the matmul.
+//
+// H is the regular-Hadamard radix-4 butterfly: the 4x4 core below Kronecker-composed
+// log4(group) times, scaled by 1/sqrt(group). That construction makes H both
+// orthogonal and symmetric, so the mul_mat orientation here needs no transpose.
+// `group` must be a power of four (the butterfly has no other factorization).
+__STATIC_INLINE__ const std::vector<ggml_fp16_t>& sd_convrot_matrix(int group) {
+    static std::map<int, std::vector<ggml_fp16_t>> cache;
+    static std::mutex cache_mutex;
+    std::lock_guard<std::mutex> lock(cache_mutex);
+
+    auto it = cache.find(group);
+    if (it != cache.end()) {
+        return it->second;
+    }
+
+    static const int core[4][4] = {{1, 1, 1, -1}, {1, 1, -1, 1}, {1, -1, 1, 1}, {-1, 1, 1, 1}};
+    const float norm            = 1.0f / std::sqrt((float)group);
+
+    std::vector<ggml_fp16_t> h((size_t)group * group);
+    for (int i = 0; i < group; i++) {
+        for (int j = 0; j < group; j++) {
+            int sign = 1;
+            for (int a = i, b = j; a > 0 || b > 0; a >>= 2, b >>= 2) {
+                sign *= core[a & 3][b & 3];
+            }
+            // Row-major: element (row i, col k) lands at i*group + k, which is the
+            // [k, i] layout ggml_mul_mat contracts over.
+            h[(size_t)i * group + j] = ggml_fp32_to_fp16((float)sign * norm);
+        }
+    }
+    return cache.emplace(group, std::move(h)).first->second;
+}
+
+__STATIC_INLINE__ ggml_tensor* ggml_ext_convrot_rotate(GGMLRunnerContext* ctx, ggml_tensor* x, int group) {
+    const std::string name = "ggml_runner_build_in_tensor:convrot_h" + std::to_string(group);
+
+    ggml_tensor* h = ggml_get_tensor(ctx->ggml_ctx, name.c_str());
+    if (h == nullptr) {
+        h = ggml_new_tensor_2d(ctx->ggml_ctx, GGML_TYPE_F16, group, group);
+        ggml_set_name(h, name.c_str());
+        // The cache entry has static lifetime, so the pointer stays valid until the
+        // runner copies it into the backend buffer after allocation.
+        ctx->bind_backend_tensor_data(h, sd_convrot_matrix(group).data());
+    }
+
+    const int64_t ne0 = x->ne[0], ne1 = x->ne[1], ne2 = x->ne[2], ne3 = x->ne[3];
+    if (!ggml_is_contiguous(x)) {
+        x = ggml_cont(ctx->ggml_ctx, x);
+    }
+    // Rotation is block-diagonal, so folding every group into the row dimension
+    // applies it to all of them with one matmul.
+    x = ggml_reshape_2d(ctx->ggml_ctx, x, group, ggml_nelements(x) / group);
+    x = ggml_mul_mat(ctx->ggml_ctx, h, x);
+    return ggml_reshape_4d(ctx->ggml_ctx, x, ne0, ne1, ne2, ne3);
+}
+
 class Linear : public UnaryBlock {
 protected:
     int64_t in_features;
@@ -3325,12 +3385,21 @@ class Linear : public UnaryBlock {
     bool force_prec_f32;
     bool allow_weight_scale;
     bool has_weight_scale = false;
+    int convrot_groupsize = 0;
     float scale;
     std::string prefix;
 
     void init_params(ggml_context* ctx, const String2TensorStorage& tensor_storage_map = {}, const std::string prefix = "") override {
-        this->prefix         = prefix;
-        has_weight_scale     = false;
+        this->prefix        = prefix;
+        has_weight_scale    = false;
+        convrot_groupsize   = 0;
+        auto weight_storage = tensor_storage_map.find(prefix + "weight");
+        if (weight_storage != tensor_storage_map.end()) {
+            const int group = weight_storage->second.convrot_groupsize();
+            if (group > 0 && in_features % group == 0) {
+                convrot_groupsize = group;
+            }
+        }
         enum ggml_type wtype = get_type(prefix + "weight", tensor_storage_map, GGML_TYPE_F32);
         if (in_features % ggml_blck_size(wtype) != 0 || force_f32) {
             wtype = GGML_TYPE_F32;
@@ -3378,7 +3447,20 @@ class Linear : public UnaryBlock {
         }
         ggml_tensor* linear_bias = has_weight_scale ? nullptr : b;
         ggml_tensor* out         = nullptr;
-        if (ctx->weight_adapter) {
+        if (convrot_groupsize > 0) {
+            // The stored weight lives in the rotated basis, so the activation has
+            // to be rotated to match. LoRA deltas are trained against the
+            // unrotated weight and cannot be mixed in here, so the adapter is
+            // deliberately bypassed rather than applied to a rotated input.
+            if (ctx->weight_adapter) {
+                static std::once_flag warned;
+                std::call_once(warned, []() {
+                    LOG_WARN("LoRA is not supported on ConvRot-quantized layers; ignoring it for those layers");
+                });
+            }
+            x   = ggml_ext_convrot_rotate(ctx, x, convrot_groupsize);
+            out = ggml_ext_linear(ctx->ggml_ctx, x, w, linear_bias, force_prec_f32, scale);
+        } else if (ctx->weight_adapter) {
             WeightAdapter::ForwardParams forward_params;
             forward_params.op_type               = WeightAdapter::ForwardParams::op_type_t::OP_LINEAR;
             forward_params.linear.force_prec_f32 = force_prec_f32;