Description
To make it effortless for researchers and developers to test UniqToken without local installation, we need a clean, interactive Google Colab notebook.
Scope of Work
Create a new notebook at notebooks/quickstart.ipynb demonstrating:
- Installing UniqToken and building the Rust native core.
- Training a custom Unigram tokenizer on a sample multilingual dataset.
- Visualizing token compression (Bytes/Token) vs standard Tiktoken.
- Exporting the trained model directly to Hugging Face
tokenizer.json.
Getting Started
- Folder:
notebooks/
- Add an "Open in Colab" badge to
README.md.
Description
To make it effortless for researchers and developers to test UniqToken without local installation, we need a clean, interactive Google Colab notebook.
Scope of Work
Create a new notebook at
notebooks/quickstart.ipynbdemonstrating:tokenizer.json.Getting Started
notebooks/README.md.