Description
LLaMA.cpp and local LLM runners consume tokenizers in the .gguf format (tokenizer.ggml.tokens, tokenizer.ggml.scores, tokenizer.ggml.token_type).
Scope of Work
- Add an export method to
hf_exporter.py or a new gguf_exporter.py.
- Convert UniqToken log-probabilities and vocab tables into standard GGUF metadata key-value pairs.
- Add a unit test verifying round-trip score extraction.
Getting Started
- Files to edit:
hf_exporter.py, test_tokenizer.py
Description
LLaMA.cpp and local LLM runners consume tokenizers in the
.ggufformat (tokenizer.ggml.tokens,tokenizer.ggml.scores,tokenizer.ggml.token_type).Scope of Work
hf_exporter.pyor a newgguf_exporter.py.Getting Started
hf_exporter.py,test_tokenizer.py