build_corpus: exact tokens for small integer literals#678
Merged
Conversation
Every numeric literal tokenizes to a single TOK_NUM, so the value is erased: the stream cannot tell `[1,2,3]` from `[0,0,0]`. Measured on the 2026-07 ecosystem corpus, that erasure is costly twice over. 1. A specification written as assertions carries no information -- `(f of 4) == 1` and `(f of 3) == 0` reach a consumer as the identical token sequence, so a model graded on whether it satisfies the assertions has nothing to learn from them. Integers 0..31 are 76% of all numeric-literal occurrences. 2. It manufactures degenerate repetition. 5.3% of the corpus sits inside a short literal cycle repeated >=3x -- the `0,0,0,0` / `"s","s"` shape -- and essentially all of it routes through the collapsed literal tokens, because every integer 0..31 becomes one `0` token (7.7% of the whole corpus). The optional 7th arg int_count=N gives exact tokens to integers [0, N), directly after the slot region. Non-integer, negative, and >=N literals keep the lossy TOK_NUM fallback -- the goal is breaking the high-frequency collision, not full losslessness. Opt-in: 0 leaves the stream byte-identical, so slot- and frequency-mode callers are unaffected. Measured effect on the real corpus at int_count=32: degenerate-cycle content drops 5.29% -> 3.62%, a 31.6% relative reduction, with genuine zeros-arrays and (still-lossy) string runs making up the remainder. `[1,2,3,4]` stops looking like a cycle; `[0,0,0]` correctly stays one repeated token. test_corpus_ints.sh (5 checks): 1..5 encode as 5 distinct tokens (collision broken); a zeros run stays one repeated token (real repetition preserved); out-of-range and non-integer literals stay lossy; no 7th arg -> int_count 0. Full suite 3113/3113 release, 3117/3117 ASan+leaks. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds an opt-in 7th arg
int_count=Ntobuild_corpusthat gives exact tokens to integer literals[0, N). Everything else (non-integer, negative,>= N) keeps the lossyTOK_NUMfallback.int_count=0(the default, and every existing caller) leaves the stream byte-identical.Why
Every numeric literal currently tokenizes to one
TOK_NUM, erasing the value. On the 2026-07 ecosystem corpus that costs us twice:(f of 4) == 1and(f of 3) == 0become the same token sequence. A model graded on satisfying assertions learns nothing from them. Integers 0..31 are 76% of numeric-literal occurrences.0,0,0,0/"s","s"shape), and essentially all of it routes through the collapsed literal tokens — every integer 0..31 is one0token, 7.7% of the whole corpus.Measured effect (real corpus,
int_count=32)Degenerate-cycle content drops 5.29% → 3.62%, a 31.6% relative reduction.
[1,2,3,4]stops looking like a cycle;[0,0,0]correctly stays one repeated token. The remainder is genuine zeros-arrays plus still-lossy string runs (integers alone can't reach the string half — deliberately scoped).Tests
test_corpus_ints.sh(5 checks): collision broken (1..5 → 5 distinct tokens), genuine repetition preserved (zeros run → one token), out-of-range and non-integer stay lossy, opt-in default. Full suite 3113/3113 release, 3117/3117 ASan+leaks.🤖 Generated with Claude Code