feat(phi3): end-to-end Phi-3 / Phi-3.5 architecture support

[unamedkr]

Summary

Adds full inference support for Phi-3 family GGUFs, validated end-to-end against bartowski/Phi-3.5-mini-instruct-Q4_K_M.gguf. Phi-3.5-mini is now the recommended best speed + quality model in the registry — vocab 32K + 3.8B params makes the lm_head matmul the fastest of any model we ship.

Validation

Coherent multi-paragraph output on plain prompts AND chat template:

```
$ phi3_infer_test ~/.cache/quantcpp/Phi-3.5-mini-instruct-Q4_K_M.gguf
The capital of France is Paris.
The Eiffel Tower, located in the city center, stands as a symbolic
landmark for both locals and tourists alike. The tower's iconic
silhouette is visible from various points around the city, offering
a panoramic view of Parisian life and its vibrant culture.
The Seine River meanders through this historic metropolis...
```

```
$ phi3_infer_test ... "What is 2+2?"
<|user|>
What is 2+2?<|end|>
<|assistant|>
The sum of 2 + 2 equals to four.
```

Regression checks:

• ctest --test-dir build → 35/35 passed
• Llama-3.2-1B end-to-end → still coherent
• SmolLM2-135M end-to-end → still coherent
• Full build clean (no new warnings)

Why Phi-3 is hard

Phi-3 ships fused weight tensors instead of llama-style separate ones, plus a long-context RoPE variant:

Tensor	Shape	What's inside
`blk.N.attn_qkv.weight`	`[hidden, 3*hidden]`	Q ‖ K ‖ V along the output axis
`blk.N.ffn_up.weight`	`[hidden, 2*ff]`	gate ‖ up along the output axis
`rope_factors_short`	`[head_dim/2]`	LongRoPE per-pair rescaling
`rope_factors_long`	`[head_dim/2]`	LongRoPE per-pair rescaling

Before this PR: load reported `0 self_attn` and the forward pass ran against zero-initialized attention weights → garbage tokens.

Implementation

Loader (`tq_load_gguf`)

• Detects `attn_qkv.weight` → `gguf_w_qkv` field, marks layer as attention
• Detects `ffn_up.weight` with `shape[1] == 2 * intermediate_dim` AND no separate `ffn_gate.weight` → `gguf_w_up_gate`
• Reads `phi3.rope.scaling.original_context_length` and `attn_factor`
• Locates `rope_factors_short`/`rope_factors_long` global tensors
• Hard-fail path from PR feat(feedback): Quick Wins from 2026-04-12 external user report #59 now correctly identifies Phi-3 as supported (32 attn layers detected)

Forward (`self_attn_forward` + `tq_forward` dispatcher)

• New `if (layer->gguf_w_qkv)` branch: one matmul into a temp buffer of size `q_dim + 2*kv_dim`, memcpy splits into `s->q/s->k/s->v`
• New `if (layer->gguf_w_up_gate)` FFN branch: one matmul of size `2*inter`, memcpy second half to `s->hb2`. Layout is `[gate | up]` (HuggingFace convention)
• The dispatcher in `tq_forward`'s layer loop ALSO calls `self_attn_forward` when `gguf_w_qkv` is set — without this the new branches were unreachable
• Same for FFN dispatcher: accept `gguf_w_up_gate` as a valid gate/up source

LongRoPE (NeoX-style)

• New branch in full RoPE: `factor[i] = (pos < orig_ctx_len) ? short[i] : long[i]`, `freq[i] = 1 / (rope_base^(2i/head_dim) * factor[i])`
• Uses NeoX-style pair layout `(q[i], q[i + half])` rather than the interleaved `(q[2i], q[2i+1])` that quant.cpp's existing `tq_rope` uses for Llama
• Why NeoX: llama.cpp's GGUF converter pre-permutes separate Q/K weights so interleaved RoPE works for Llama-family models. The fused `attn_qkv` is NOT permuted, so we apply rotation in its native NeoX form.
• `rope_attn_factor` multiplied into Q only when `pos >= orig_ctx_len`

State allocation

• `max_dim` bumped to cover `q_dim + 2*kv_dim` (fused QKV temp buffer reuses `s->xb2`)
• `s->hb` bumped to `2 * inter` when fused gate||up is detected

Tokenizer BOS

• `tq_encode` adds `` to the BOS lookup chain alongside `` and `<|im_start|>`
• `quant_generate` enables `add_bos=1` when vocab has ``. Phi-3 specifically degrades into garbage without it; Llama-3 behavior is unchanged because Llama-3 uses `<|begin_of_text|>`.

Debugging journey (TIL)

1. First implementation: garbage output. Loader correct, forward path wasn't being hit.
2. The dispatcher's `if (layer->gguf_wq)` check guarded the call to `self_attn_forward`. Phi-3 has `gguf_wq == NULL`. Fixed.
3. Still garbage. Forward branch firing but output is fragmented English.
4. Tried NeoX vs GPT-J RoPE — both garbage but different.
5. Verified Q5_K dequant + matmul produces bit-perfect output against an FP32 reference.
6. Verified tokenizer produces correct token IDs (`The capital of France is` → 5 correct tokens).
7. Tried `[gate | up]` vs `[up | gate]` for the fused FFN. `[gate | up]` was the answer. Output went from "Block...attacker...non...meteor..." to "Paris. The Eiffel Tower..." instantly.

The bug was that I had the FFN split order backwards. HuggingFace's `gate, up = gate_up_proj(x).chunk(2, dim=-1)` produces `[gate, up]` along the output axis, which is what the GGUF preserves. I had assumed `[up, gate]` based on a half-remembered llama.cpp comment.

New tools

• `tools/gguf_inspect.c` — dumps tensor names, shapes, types, and metadata from any GGUF. Used to verify Phi-3.5's layout before writing loader code. General-purpose, kept for future architecture work.
• `tools/phi3_infer_test.c` — minimal end-to-end inference test. Doubles as a smoke test for any future Phi-3 changes.

Registry / docs updates

• `Phi-3.5-mini` added to `_MODEL_REGISTRY` with bartowski's Q4_K_M variant (~2.4 GB)
• New CLI aliases: `phi3.5`, `phi3.5:mini`, `phi-3.5`, `phi-3.5-mini`
• `docs/supported_models.md` updated: phi3 now ✅ Fully supported. Recommended-models table puts Phi-3.5-mini at the top.
• `docs/spikes/2026-04-12_phi3_support.md` updated with the inspection findings + conclusions that drove implementation.

Test plan

• Unit tests pass (35/35)
• Phi-3.5-mini end-to-end inference produces coherent paragraphs
• Phi-3 chat template works ("What is 2+2?" → "...equals to four")
• Llama-3.2-1B regression — still coherent
• SmolLM2-135M regression — still coherent
• Manual: `quantcpp run phi-3.5-mini` after wheel build
• Manual: long-context test (>4096 tokens) to exercise the LongRoPE `pos >= orig_ctx_len` branch

🤖 Generated with Claude Code