Skip to content

Try local_files_only first in load() to skip avoidable HF Hub round-trips - #1571

Closed
Vlor999 wants to merge 1 commit into
ml-explore:mainfrom
Vlor999:fix/load-offline-first
Closed

Try local_files_only first in load() to skip avoidable HF Hub round-trips#1571
Vlor999 wants to merge 1 commit into
ml-explore:mainfrom
Vlor999:fix/load-offline-first

Conversation

@Vlor999

@Vlor999 Vlor999 commented Jul 15, 2026

Copy link
Copy Markdown

Closes #1570.

Summary

_download() (used by both load() for model weights and load_tokenizer() for tokenizer files) called huggingface_hub.snapshot_download() without local_files_only=True. Even with a complete local cache, snapshot_download performs an API round-trip (repo_info/model_info) to check for updates before touching the cache. Since load() calls _download() twice per invocation (weights + tokenizer files), this cost was paid twice on every single call — for the CLI (mlx_lm.generate, mlx_lm.chat, mlx_lm.server) and every library user, not just first-time downloads.

This PR makes _download() try snapshot_download(..., local_files_only=True) first (near-instant when the cache is complete) and only fall back to the network-enabled call on LocalEntryNotFoundError (the genuine first-download / incomplete-cache case). ModelScope users are unaffected — left on the existing always-network path since I couldn't validate local_files_only semantics for that backend.

Measurements (M4 Pro, 5 runs/config, mean ± stdev)

Model before after change
Qwen2.5-0.5B-Instruct-4bit 0.355 ± 0.030 s 0.178 ± 0.003 s -49.8%
Llama-3.2-1B-Instruct-4bit 0.592 ± 0.042 s 0.412 ± 0.002 s -30.5%
gpt-oss-20b-MXFP4-Q8 3.765 ± 0.062 s 3.544 ± 0.052 s -5.9% (weight loading dominates at this size)

For an isolated cache hit, snapshot_download(..., local_files_only=True) on these repos takes 0.4-1.6ms, confirming the ~200-230ms per call removed was pure avoidable network latency rather than any real cache-validation work.

Note on load_tokenizer: after this fix, the new dominant cost of load() (profiled with cProfile) is transformers.AutoTokenizer.from_pretrained itself (Tokenizer.__setstate__ + a copy.deepcopy of the tokenizer object, ~280ms combined for Llama-3.2-1B). That's upstream transformers/tokenizers code, out of scope for this PR, but worth flagging for anyone looking at load() latency next.

Test plan

  • python -m pytest tests/test_utils.py tests/test_generate.py tests/test_tokenizers.py tests/test_prompt_cache.py — 62 passed, 6 subtests passed, no changes needed.
  • black --check mlx_lm/utils.py — clean on the diff (an unrelated pre-existing block elsewhere in the file would reformat under a newer black, untouched here).
  • Verified the fallback path still works for a repo requiring an actual download (raises LocalEntryNotFoundError on cache miss, falls through to the network-enabled call).

…-trips

_download() (used by both load() for model weights and load_tokenizer() for
tokenizer files) calls huggingface_hub.snapshot_download() without
local_files_only=True. Even when a repo is fully cached locally,
snapshot_download makes a network round-trip (a repo_info/model_info API
call) to check for updates before touching the cache. Since load() calls
_download() twice per invocation (once for weights, once for tokenizer
files), this costs ~400-450ms on every single load() call for an
already-cached model - paid by every CLI invocation (mlx_lm.generate,
mlx_lm.chat) and every library user, not just first-time downloads.

This tries snapshot_download(..., local_files_only=True) first (near-instant
when the cache is complete) and only falls back to the network-enabled call
on LocalEntryNotFoundError (the genuine first-download/incomplete-cache
case). ModelScope users are unaffected (different backend, left on the
existing always-network path since I can't validate local_files_only
semantics there).

Measured on an M4 Pro (5 runs/config, mean +/- stdev):
- Qwen2.5-0.5B-Instruct-4bit: 0.355s -> 0.178s (-49.8%)
- Llama-3.2-1B-Instruct-4bit: 0.592s -> 0.412s (-30.5%)
- gpt-oss-20b-MXFP4-Q8:       3.765s -> 3.544s (-5.9%, weight loading dominates)

Full benchmark methodology, raw data, and plots in the companion issue/PR.
@Vlor999

Vlor999 commented Aug 3, 2026

Copy link
Copy Markdown
Author

Superseded by #1650, which rebuilds this on current main and adds a measurement (147 ms -> 0.26 ms per call) plus tests. Closing to keep the queue clean.

@Vlor999 Vlor999 closed this Aug 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

load() pays two avoidable Hugging Face Hub network round-trips even when the model is fully cached locally

1 participant