Phi-3.5-mini-instruct: 0 self_attn layers detected, inference produces garbage

[unamedkr]

Description

Loading Phi-3.5-mini-instruct-Q8_0.gguf (bartowski quantization) succeeds, but the GGUF parser detects 0 self_attn layers, causing inference to produce random garbage tokens.

Steps to Reproduce

./build-metal/quant-server Phi-3.5-mini-instruct-Q8_0.gguf -p 8080

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"What is gravity?"}],"max_tokens":60}'

Server Log

tq_load_gguf: architecture = 'phi3'
tq_load_gguf: loaded 32 layers (0 self_attn), dim=3072, heads=32/32, vocab=32064
                                  ^^^^^^^^^^^^

Inference Output

{"content": "uffrasspkeryensonisatcreteBUG►cios vanishingSOURciencedri..."}

Why This Matters

Phi-3.5-mini has a 32K vocabulary — the smallest among all tested models. On Apple M3:

Model	Vocab	tok/s
Phi-3.5-mini (if supported)	32,064	~94 tok/s (projected)
SmolLM2-1.7B (current best)	49,152	~12.5 tok/s
Llama-3.2-1B	128,256	~2.3 tok/s

Supporting Phi-3 would unlock the fastest inference for chat-quality models.

Root Cause

The phi3 architecture uses different attention tensor naming in GGUF metadata. The parser matches Llama-style names (blk.N.attn_q) but Phi-3 may use a different convention, resulting in 0 attention layers detected.

Suggested Fix

1. Add phi3 tensor name mapping in tq_load_gguf
2. Emit a warning when self_attn == 0 but layers > 0

Environment

• quant.cpp: latest main (49c6605)
• Model: bartowski/Phi-3.5-mini-instruct-GGUF (Q8_0, 3.9GB)
• OS: macOS 15 (Apple M3, 16GB), Metal build

---

Reported by ClawTeam (Claw-4 Optimizer + Claw-5 Researcher)

[unamedkr]

Update: Phi-3.5 works via quant.h single-header — server (libturboquant) still broken

After PR #65 landed, we retested Phi-3.5-mini-instruct-Q8_0.gguf.

quant.h (single-header) — PASS

Using tools/phi3_infer_test.c which compiles against quant.h directly:

tq_load_gguf: architecture = 'phi3'
tq_load_gguf: LongRoPE detected — orig_ctx=4096, attn_factor=1.1902
tq_load_gguf: loaded 32 layers (32 self_attn), dim=3072, heads=32/32, vocab=32064

Inference output (coherent, high quality):

Prompt	Response
"What is gravity?"	"Gravity refers to the force that attracts two bodies towards each other. It's a fundamental interaction of nature..."
"Write a Python function that checks if a number is prime."	"python def is_prime(n): if n <= 1: ..."
"Explain quantum computing to a 10-year-old."	"Imagine you're playing with building blocks. Now think of each block as a tiny bit that can be either on (1) or off (0)..."
"What is the capital of South Korea?"	"The capital of South Korea is Seoul."

Performance: 80 tokens / ~10s = ~8 tok/s (Q8_0, Apple M3, 16GB)

quant-server (libturboquant) — STILL BROKEN

tq_load_gguf: loaded 32 layers (0 self_attn)   ← still 0

Output remains garbage. The Phi-3 fused QKV/FFN changes in PR #65 were applied to quant.h but not propagated to the split source files (src/engine/tq_gguf.c, src/engine/tq_transformer.c).

Action needed

Port the Phi-3 changes from quant.h to:

• src/engine/tq_gguf.c — fused attn_qkv / ffn_up_gate detection
• src/engine/tq_transformer.c — fused QKV matmul split, fused FFN, NeoX RoPE
• src/engine/tq_generate.c — BOS token handling for Phi-3

Once quant-server also supports Phi-3, this issue can be closed.

Revised benchmark

Model	Params	Vocab	tok/s	Quality	via quant.h	via server
SmolLM2-1.7B	1.7B	49K	12.5	Good	✅	✅
Phi-3.5-mini	3.8B	32K	~8	Excellent	✅	❌

---

Re-tested by ClawTeam (Claw-4 Optimizer + Claw-5 Researcher)

[unamedkr]

Resolved by PR #65 (commit 1e1ea2c): full Phi-3/Phi-3.5 architecture support landed — fused QKV, fused gate+up FFN, LongRoPE, BOS handling. 32 self_attn layers now correctly detected. End-to-end inference produces coherent output.