AudioCodecModel speaker encoder is fetched from a hardcoded HuggingFace URL, bypassing the HF cache and crashing model construction on HTTP 429

[janbernloehr]

Describe the bug

AudioCodecModel.__init__ loads the pretrained ResNet speaker encoder from a hardcoded HuggingFace URL, which makes model construction fail intermittently with FileNotFoundError whenever HuggingFace rate-limits the request (HTTP 429).

In nemo/collections/tts/models/audio_codec.py, under the use_scl_loss branch:

self.speaker_encoder.load_checkpoint(
    "https://huggingface.co/Edresson/Speaker_Encoder_H_ASP/resolve/main/pytorch_model.bin", strict=False
)

load_checkpoint -> load_fsspec treats a URL as remote and calls fsspec.open(path, "rb"). As a result the fetch:

1. bypasses the HuggingFace cache - load_fsspec only short-circuits to torch.load when os.path.isfile(path), so the checkpoint (44,610,930 bytes) is re-downloaded on every model construction even when HF_HOME points at a warm shared cache;
2. is unauthenticated - HF_TOKEN/HF_HUB_TOKEN are not used, so requests are subject to the anonymous per-IP quota, which is easily exhausted on a shared egress IP or in CI;
3. has no retry/backoff, and fsspec converts the HTTP 429 into a bare FileNotFoundError, which hides the real cause and defeats 429-aware handling. from_config_dict then retries instantiation 3 more times within ~3 seconds, adding load to an already-throttled endpoint.

Notably, other HuggingFace artifacts loaded by the same model do go through huggingface_hub and are served from the cache; this one call is the outlier, which is why a warm cache does not protect it.

This affects any model that instantiates an AudioCodecModel with use_scl_loss: true in its config - including loading the published nvidia/magpie_tts_multilingual_357m checkpoint, whose nested codec config sets it.

Worth noting: this checkpoint is only needed for the speaker-consistency training loss. get_speaker_embedding is called solely from training_step/validation_step, and state_dict/load_state_dict explicitly delete speaker_encoder.* keys. So for pure inference it is downloaded, frozen, and never used - gating the fetch on training mode would remove the external dependency from the inference path entirely.

Steps/Code to reproduce bug

from nemo.collections.tts.models import MagpieTTSModel
# Any AudioCodecModel config with use_scl_loss: true reaches the same line.
model = MagpieTTSModel.from_pretrained("nvidia/magpie_tts_multilingual_357m")

Whenever huggingface.co answers 429 for Edresson/Speaker_Encoder_H_ASP/resolve/main/pytorch_model.bin, construction fails:

aiohttp.client_exceptions.ClientResponseError: 429, message='Too Many Requests',
  url='https://huggingface.co/Edresson/Speaker_Encoder_H_ASP/resolve/main/pytorch_model.bin'
The above exception was the direct cause of the following exception:
  File "nemo/collections/tts/models/audio_codec.py", line 185, in __init__
    self.speaker_encoder.load_checkpoint(
  File "nemo/collections/tts/modules/audio_codec_modules.py", line 460, in load_checkpoint
    state = load_fsspec(checkpoint_path, map_location=torch.device("cpu"))
  File "nemo/collections/tts/modules/audio_codec_modules.py", line 211, in load_fsspec
    with fsspec.open(path, "rb") as f:
  File ".../fsspec/implementations/http.py", line 440, in _info
    raise FileNotFoundError(url) from exc
FileNotFoundError: https://huggingface.co/Edresson/Speaker_Encoder_H_ASP/resolve/main/pytorch_model.bin

To force the failure deterministically without waiting for a real 429, warm the HF cache and then block egress to huggingface.co: cached artifacts still resolve, but this fetch fails - demonstrating it does not use the cache.

Expected behavior

The speaker-encoder checkpoint resolves through the HuggingFace cache and honours HF_TOKEN, HF_HOME/HF_HUB_CACHE, and HF_HUB_OFFLINE, with retry/backoff on transient HTTP errors - like every other HF artifact NeMo loads. It should be downloaded at most once per cache, and a rate-limit response should surface as a retryable rate-limit error rather than FileNotFoundError. Ideally the fetch is skipped entirely when the model is not training, since the speaker encoder is unused at inference.

Suggested fix

from huggingface_hub import hf_hub_download

speaker_encoder_checkpoint = hf_hub_download(
    repo_id="Edresson/Speaker_Encoder_H_ASP",
    filename="pytorch_model.bin",
)
self.speaker_encoder.load_checkpoint(speaker_encoder_checkpoint, strict=False)

hf_hub_download returns a local path, which load_fsspec loads via torch.load, so restored weights and strict=False semantics are unchanged. Two related improvements: make load_fsspec preserve the underlying HTTP error instead of collapsing it into FileNotFoundError, and add backoff between the from_config_dict instantiation retries.

Validation

Applied to NeMo v2.7.3 in a container and ran the previously-failing text-to-speech benchmark across four configurations; all now build the model and complete with metrics. Verified from HTTP traces that the checkpoint is downloaded once into the shared cache and served locally on subsequent runs (later runs issue only a HEAD revalidation and no download). Benchmark throughput is unchanged within run-to-run noise (-1.0% ... +1.0%).

Environment overview (please complete the following information)

• Environment location: Docker, on-prem Slurm cluster (8xH100 node, single GPU used)
• Method of NeMo install: from source - pip install -e ".[common]" at commit 1d4ee423806d461f9146ae982f9da8eb32495ae7 (v2.7.3), plus requirements_tts.txt, requirements_asr.txt, requirements_audio.txt
• NeMo revision: 1d4ee423806d461f9146ae982f9da8eb32495ae7; the same code is present on main as of 2026-07-29 (commit 2c6305594c6609d86b0dd382f89dff5db511ae1d)
• Base image: NVIDIA PyTorch container; the affected code path is framework-independent

Environment details

NVIDIA docker image is used. Python 3.12; fsspec 2024.12.0 with the aiohttp HTTP backend; huggingface_hub 1.15.0 is already an installed NeMo dependency, so the suggested fix adds no new requirement.

Additional context

GPU model: NVIDIA H100. The failure is not GPU- or platform-specific - it is a host-network/HTTP failure during model construction, observed on both A100 and H100 hosts. It is intermittent and correlates with shared-IP request volume: concurrent unrelated jobs on the same egress IP were throttled in the same window, and sibling jobs minutes apart on identical images differ only in whether HuggingFace answered 200 or 429.

---

This issue was drafted with assistance from the opus AI model.