Mojobag

An implementation of a BPE tokenizer in Mojo. Train your own tokenizer to produce a bag of tokens.

Important

Only tested on Linux x86. Does not work on Apple Silicon yet due to a pcre2 dependency issue.

Getting Started

Clone this repository
Install pixi
Run pixi shell. This will set up your environment with all required dependencies.

I've included a compressed wikitext-103-raw-v1 in data/. To build training, test and validation datasets, run pixi unzip-wikitext.