Skip to content

Serve fully cached repos without a Hub round-trip - #1650

Open
Vlor999 wants to merge 1 commit into
ml-explore:mainfrom
Vlor999:fix/load-local-files-only
Open

Serve fully cached repos without a Hub round-trip#1650
Vlor999 wants to merge 1 commit into
ml-explore:mainfrom
Vlor999:fix/load-local-files-only

Conversation

@Vlor999

@Vlor999 Vlor999 commented Aug 3, 2026

Copy link
Copy Markdown

Proposed changes

Fixes #1649.

_download() called snapshot_download() without local_files_only, so a repo
that is already fully cached still paid a Hub API round-trip to check for
updates before serving from the cache. load() calls _download() twice — once
for weights, once for tokenizer files — so every invocation paid it twice, for
the CLI and for library users alike, not only on first download.

This tries local_files_only=True first and falls back to the existing online
call on LocalEntryNotFoundError, so behaviour is unchanged whenever the cache
cannot satisfy the request.

Why

huggingface_hub 1.26.0, warm cache, median of 5 calls on an already-cached repo:

per call per load() (2 calls)
before (network check) 147.2 ms ~294 ms
after (local first) 0.26 ms ~0.5 ms
565x

Reproducer is in #1649. The absolute number is latency to the Hub, so it grows
on a slow link — and on a flaky one the current code blocks until the check
times out, where this returns immediately.

How

  • Try the cache first, fall back on LocalEntryNotFoundError only.
  • Gated on _supports_local_files_only, because ModelScope's snapshot_download
    takes no such argument. On ModelScope the behaviour is byte-for-byte unchanged.
  • hf_repo_to_path() in the same file already uses local_files_only=True, so
    this follows an established pattern rather than introducing one.

Alternative rejected: catching a broader exception. LocalEntryNotFoundError
is what a cache miss raises (verified against 1.26.0); catching more would
silently swallow real Hub errors and turn them into a redundant second attempt.

Validation

python -m unittest tests.test_utils      # 11 passed
pre-commit run --files mlx_lm/utils.py tests/test_utils.py   # black, isort

Two tests added:

  • test_cached_repo_skips_network — a cached repo results in exactly one
    snapshot_download call, made with local_files_only=True.
  • test_falls_back_to_network_when_not_cached — a LocalEntryNotFoundError
    produces a second call without local_files_only.

I checked the tests are not vacuous by flipping the new argument to False:
both fail. With the fix in place, all 11 pass.


Supersedes #1571, which was opened against an older main. This is the same fix
rebuilt on current main, with the measurement and tests added.

_download() called snapshot_download() without local_files_only, so even a
fully cached repo paid an API round-trip to check for updates. load() calls
_download() twice (weights and tokenizer files), so every invocation paid it
twice.

Try local_files_only=True first and fall back to the online path on
LocalEntryNotFoundError. ModelScope has no equivalent, so the fast path is
gated on the Hugging Face backend.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

load() pays an avoidable Hub round-trip per call even when the model is fully cached

1 participant