Skip to content

Pull requests: huggingface/tokenizers

New pull request New
128 Open 918 Closed
128 Open 918 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who’s assigned
Assigned to nobody Loading
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Pull requests list

Fix malformed BPE merges panicking during build
#2219 opened Jul 19, 2026 by 0xzr Loading…
1
Preserve backslashes in generated Python stub docstrings
#2218 opened Jul 18, 2026 by cupkk Loading…
1
5
Add a ParityBpeTrainer example and list it in the trainers API docs
#2217 opened Jul 18, 2026 by cimeister Contributor Loading…
Fix XSS in EncodingVisualizer HTML attributes
#2216 opened Jul 18, 2026 by varneeth01 Loading…
1
1
Fix dead wikitext-103 S3 link in quicktour.mdx
#2215 opened Jul 18, 2026 by RudrenduPaul Loading…
1
Fix token_to_sequence returning Some(0) for out-of-bounds index
#2214 opened Jul 17, 2026 by Osamaali313 Loading…
1
feat: multi threaded pipeline
#2213 opened Jul 17, 2026 by McPatate Member • Draft
1
Fix Metaspace decoder dropping internal spaces in the first token
#2212 opened Jul 17, 2026 by sohumt123 Loading…
Add missing return type hints to BaseTokenizer.save, save_model, to_str
#2211 opened Jul 17, 2026 by RudrenduPaul Loading…
Atomnorm
#2209 opened Jul 16, 2026 by ArthurZucker Collaborator Loading…
1
chore(deps-dev): bump websocket-driver from 0.7.4 to 0.7.5 in /tokenizers/examples/unstable_wasm/www dependencies Pull requests that update a dependency file javascript Pull requests that update Javascript code
#2208 opened Jul 15, 2026 by dependabot Bot Loading…
1
Pipeline-only tk-encode + tk-train on the pipeline (thin deps) — replaces #2200
#2207 opened Jul 15, 2026 by ArthurZucker Collaborator • Draft
fix: skip pre-tokenizer when is_pretokenized=True
#2204 opened Jul 15, 2026 by pctablet505 • Draft
1
Upgrade hf-hub to 1.0.0
#2203 opened Jul 15, 2026 by assafvayner • Draft
1
perf(atomsplit): fuse o200k letter case-split + 32-wide ASCII classify
#2201 opened Jul 15, 2026 by ArthurZucker Collaborator Loading…
1
perf: reduce BPE tokenizer load-time memory
#2199 opened Jul 15, 2026 by xrl1 Loading…
perf: reduce BPE tokenizer load-time memory
#2197 opened Jul 14, 2026 by xrl1 Loading…
2
Upgrade hf-hub to 1.0
#2196 opened Jul 14, 2026 by assafvayner Loading…
1
Clarify byte-level pretokenizer behavior
#2195 opened Jul 14, 2026 by michaelalbada Loading…
2
(wip) faster bpe run-pipeline-bench
#2194 opened Jul 13, 2026 by SBrandeis Contributor • Draft
[POC] fast encode: FlatCache + MPHF RankStore + incremental merge run-pipeline-bench
#2190 opened Jul 12, 2026 by ArthurZucker Collaborator • Draft
4
fix: anchor ptrhash revision
#2189 opened Jul 11, 2026 by SBrandeis Contributor • Draft
4
feat: Use BucketVocabStore in PipelineBPE
#2188 opened Jul 11, 2026 by SBrandeis Contributor Loading…
1
Emit UserWarning from Split.pre_tokenize_str() on zero matches
#2186 opened Jul 10, 2026 by RUSHI-KOLLA Loading…
1
tk-encode: wire post-processor as pipeline STAGE_POST
#2182 opened Jul 10, 2026 by ArthurZucker Collaborator Loading…
3
ProTip! no:milestone will show everything without a milestone.