Skip to content

feat(export): Add GGUF tokenizer metadata export for llama.cpp integration #5

Description

@umran666

Description

LLaMA.cpp and local LLM runners consume tokenizers in the .gguf format (tokenizer.ggml.tokens, tokenizer.ggml.scores, tokenizer.ggml.token_type).

Scope of Work

  1. Add an export method to hf_exporter.py or a new gguf_exporter.py.
  2. Convert UniqToken log-probabilities and vocab tables into standard GGUF metadata key-value pairs.
  3. Add a unit test verifying round-trip score extraction.

Getting Started

  • Files to edit: hf_exporter.py, test_tokenizer.py

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions