[SS-JIA]

Stack from ghstack (oldest at bottom):

• [executorch][native-ggml] Enable CUDA graphs in CMake builds #24849
• -> [executorch][native] Load Hugging Face tokenizer.json in the LLM runner #24848
• [executorch][native-ggml] Size the scheduler hash for graph leafs too #24847
• [executorch][native-ggml] Leave zero-frequency RoPE dims unrotated #24846
• [executorch][native-ggml] Accept GGUF q5_k weights #24845

Ulterior Motive

Running Gemma 4 31B through ExecuTorch Native's GGML engine on CUDA, to find why it decodes slower than llama.cpp on the same GGUF.

Rationale

What: --tokenizer also takes a Hugging Face tokenizer.json, or a directory holding one.

Why: The runner loaded only Llama 3 TikToken models, and Gemma 4 ships its tokenizer as a tokenizer.json.

Details

• Tokenizer: a path ending in .json, or a directory, loads tokenizers::HFTokenizer; any other path loads the Llama 3 TikToken model as before.
• BOS:
  • Before: TikToken's encode prepended --num_bos BOS tokens, at most 127.
  • After: the runner prepends --num_bos copies of the tokenizer's BOS itself, because a tokenizer.json adds BOS only if its post-processor does. TikToken prompts encode to the same tokens as before.
• llm_runner depends on //pytorch/tokenizers:hf_tokenizer.

Authored with Claude Code.

Differential Revision: D130721992

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24848

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (4 Unrelated Failures)

As of commit 8630f11 with merge base 4529a94 ():

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Test QNN Backend / test-qnn / test-backend-linux (qnn, models) / linux-job (gh) (trunk failure)
  curl: (22) The requested URL returned error:
• Test QNN Backend / test-qnn / test-backend-linux (qnn, operators) / linux-job (gh) (trunk failure)
  curl: (22) The requested URL returned error:
• Test QNN Windows MSVC build / build-qnn-windows-arm64 (gh) (trunk failure)
• Test QNN Windows MSVC build / build-qnn-windows-x64 / windows-job (gh) (trunk failure)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[JakeStevens]

Review automatically exported from Phabricator review in Meta.