Skip to content

Pull requests: huggingface/tokenizers

New pull request New
27 Open 736 Closed
27 Open 736 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who’s assigned
Assigned to nobody Loading
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Pull requests list

DOCS: add add_prefix_space to processors.ByteLevel
#1878 opened Oct 21, 2025 by CloseChoice Loading…
1
RFC for now, save the special_tokens_mapping
#1874 opened Oct 7, 2025 by ArthurZucker • Draft
Mark Python tests that need network access
#1872 opened Oct 2, 2025 by gordonmessmer Loading…
Update indicatif dependency
#1867 opened Sep 22, 2025 by gordonmessmer Loading…
Add setter to mutable properties
#1865 opened Sep 13, 2025 by cyyever Loading…
Add a multithreaded tokenizer test and 3.13t CI
#1864 opened Sep 12, 2025 by ngoldbaum Loading…
3
feat: allow BPETrainer to be seeded with a set of initial tokens
#1862 opened Sep 6, 2025 by henrycharlesworth Loading…
Mark immutable pyclasses as frozen
#1861 opened Sep 5, 2025 by ngoldbaum Loading…
Fix unsigned integer underflow issue with truncation
#1859 opened Sep 1, 2025 by maxdebayser Loading…
5
feat: add cli for tokenizer and training Feature Request
#1842 opened Aug 6, 2025 by b00f Loading…
6
feat: whitespace optimize
#1841 opened Aug 6, 2025 by b00f Loading…
1
Unused Unicode Character Filter
#1832 opened Jul 23, 2025 by sanderland Loading…
2
Add enforce_utf8_boundaries option to BpeTrainer
#1830 opened Jul 22, 2025 by sanderland Loading…
10
Faster Whitespace PreTokenizer (Drop-in Replacement)
#1822 opened Jul 7, 2025 by 8ria Loading…
1
Track lockfile
#1806 opened Jun 22, 2025 by sftse Loading…
6
Adding multiprocessing for sentencepiece_extractor
#1804 opened Jun 19, 2025 by AamodThakur Loading…
Expose Encoding attributes via the buffer protocol interface
#1789 opened Jun 4, 2025 by mariosasko Loading…
3
add group capture to replace
#1788 opened Jun 3, 2025 by cboseak Loading…
1
Add Truncate pre-tokenizer
#1783 opened May 27, 2025 by ArthurZucker • Draft
Add benchmark for deserializing large added vocab + optimizations
#1782 opened May 27, 2025 by ArthurZucker • Draft
1
Make unigram cache optional
#1763 opened Apr 18, 2025 by wangrunji0408 Loading…
3
Implement Append normalizer
#1755 opened Mar 24, 2025 by austinleedavis Loading…
4
Pre-tokenizers that support multi-word/non-whitespace BPE in single pass
#1753 opened Mar 22, 2025 by mjbommar Loading…
2
Does windows aarch work ?
#1719 opened Jan 10, 2025 by Narsil Loading…
1
Draft backtrack
#1712 opened Jan 3, 2025 by ArthurZucker • Draft
2
ProTip! Type g p on any issue or pull request to go back to the pull request listing page.