Skip to content

gemma4/gguf: accept a scalar attention.head_count_kv - #64

Open
Rhycecomley wants to merge 1 commit into
FlashML-org:mainfrom
Rhycecomley:fix/gemma4-gguf-scalar-head-count-kv
Open

gemma4/gguf: accept a scalar attention.head_count_kv#64
Rhycecomley wants to merge 1 commit into
FlashML-org:mainfrom
Rhycecomley:fix/gemma4-gguf-scalar-head-count-kv

Conversation

@Rhycecomley

Copy link
Copy Markdown

Fixes #63.

parse_gguf_config treats gemma4.attention.head_count_kv as a per-layer array and indexes
it directly. llama.cpp collapses that array to a single value when every block shares it,
which the official google/gemma-4-E2B-it-qat-q4_0-gguf
checkpoint hits (head_count_kv == 1 across all 35 blocks), so ft serve dies during
_adjust_config with:

TypeError: 'int' object is not subscriptable

This broadcasts a scalar to one entry per block before the SWA/full lookups. Arrays are
unaffected — the isinstance guard only fires on the scalar form.

Verification

Before: ft serve --model gemma-4-E2B_q4_0-it.gguf raises the TypeError above and the
frontend shuts the API server down.

After: config parsing completes and startup proceeds past this point.

Note my machine can't finish a full generation run on this checkpoint for unrelated reasons
(4 GB VRAM), so I've verified the parse fix rather than end-to-end inference — worth a second
look from someone with a supported card.

llama.cpp writes gemma4.attention.head_count_kv as a single value when every
block shares it, so parse_gguf_config raised "'int' object is not subscriptable"
on the official google/gemma-4-E2B-it-qat-q4_0-gguf checkpoint. Broadcast the
scalar to one entry per block before the per-layer lookups.

Fixes FlashML-org#63

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Rhycecomley

Copy link
Copy Markdown
Author

Scope note: as detailed in #63 (comment), this makes a scalar head_count_kv parse, but it does not make the E2B checkpoint loadable — the next line hits a per-layer feed_forward_length that ModelConfig.intermediate_size can't represent. Please treat this purely as a robustness fix for valid-but-uniform GGUF metadata, not as E-series support.

@mkornreich

Copy link
Copy Markdown

Does this relate to my PR? See #59

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Gemma-4 E2B QAT GGUF fails to load: scalar attention.head_count_kv indexed as a per-layer list

2 participants