Skip to content
This repository has been archived by the owner on Jun 24, 2024. It is now read-only.

How do I use Huggingface tokenization to use a model on Huggingace in MODEL_PATH instead of my local machine? #427

Open
balqaasem opened this issue Sep 14, 2023 · 1 comment

Comments

@balqaasem
Copy link

No description provided.

@LLukas22
Copy link
Contributor

I don't think i understand your question completely, but if you want to use an external tokenizer you can simply provide the path or model-name to the tokenizer you want to use. The tokenizers library will resolve it and if necessary download the tokenizer's config file.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants