SentencePiece BPE produces incorrect output on Gemma 3

[chonknick]

Reproducer

import kitoken
from tokenizers import Tokenizer
from huggingface_hub import hf_hub_download

path = hf_hub_download("google/gemma-3-4b-it", "tokenizer.json")
ki = kitoken.Kitoken.from_tokenizers_file(path)
hf = Tokenizer.from_file(path)

text = "In [[political philosophy]], the concept of [[limited government]]"

ki_ids = ki.encode(text)
hf_ids = hf.encode(text, add_special_tokens=False).ids

print(f"kitoken: {len(ki_ids)} tokens → {ki_ids[:15]}")
print(f"HF:      {len(hf_ids)} tokens → {hf_ids[:15]}")
print(f"Match: {ki_ids == hf_ids}")

Output:

kitoken: 20 tokens → [878, 6655, 34773, 27688, 7016, 2318, 1955, 235265, 3178, 1055, 6654, 3153, 576, 6655, 17268, 3838, 7016, 2318, 1955, 235265]
HF:      16 tokens → [878, 6655, 34773, 27688, 7016, 2318, 1955, 235265, 3178, 138, 6654, 3153, 576, 6655, 17268, 3838, 7016, 2318, 1955, 235265]
Match: False

Details

kitoken produces ~13% more tokens than HuggingFace on enwik8 with Gemma 3's SentencePiece BPE tokenizer (3,575 vs 3,156 tokens on 10KB). The first divergence is at token index 9, where HF produces token 138 but kitoken produces 3153.

This suggests some merges are being skipped or merge ranks are being applied incorrectly for SentencePiece-style BPE models.

Tested with kitoken 0.10.1, tokenizers 0.25.0.

[rjpower]

Thanks for writing kitoken! FWIW, I addressed all of the divergences vs HF which I could fin, with the exception of some metaspace handling, here: https://github.com/marin-community/kitoken. I confirmed I get identical results with HF for ~10B tokens on web data as well.

I have no particular love for the HF semantics, but adhering to them closer makes it easy to fuzz and address most of the issues we found. There's a bunch of random cruft on that repo from the testing, if you would find it helpful, I'm happy to prepare a clean PR, or feel free to just adopt whatever changes you like directly.