Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Experimental] Add GPTNeoXTokenizer support to HFTokenizerConverter #678

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from
Draft
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
Improve handling of missing vocab_file attribute in HFTokenizerConv…
…erter

This commit updates `HFTokenizerConverter` to handle cases where the `hf_tokenizer` object might not have a `vocab_file` attribute.

Changes:

* Uses `getattr` to retrieve the `vocab_file` attribute for flexibility
* Stores the retrieved value in a separate variable `vocab_file` for clarity
* Checks if `vocab_file` is `None` before checking its existence

This ensures the converter works correctly even with tokenizers that don't define a `vocab_file` attribute.
kazssym committed Mar 24, 2024

Verified

This commit was signed with the committer’s verified signature.
dsanders11 David Sanders
commit 087aa1e5984041199676d513711b953263f163a7
5 changes: 3 additions & 2 deletions onnxruntime_extensions/_hf_cvt.py
Original file line number Diff line number Diff line change
@@ -43,10 +43,11 @@ def convert_json_vocab(hf_tokenizer):
f"{hf_tokenizer.__name__}: vocab_files_names is not found")

tokenizer_file = filenames["tokenizer_file"]
if (hf_tokenizer.vocab_file is None) or (not os.path.exists(hf_tokenizer.vocab_file)):
vocab_file = getattr(hf_tokenizer, "vocab_file", None)
if (vocab_file is None) or (not os.path.exists(vocab_file)):
model_dir = hf_tokenizer.name_or_path
else:
model_dir = os.path.dirname(hf_tokenizer.vocab_file)
model_dir = os.path.dirname(vocab_file)
tokenizer_json = json.load(
open(os.path.join(model_dir, tokenizer_file), "r", encoding="utf-8"))
# get vocab object from json file