fix(tts): use Hugging Face cache for speaker encoder

[janbernloehr]

What does this PR do ?

Resolve the pretrained speaker-encoder checkpoint through hf_hub_download
before loading it. This allows AudioCodecModel to use Hugging Face caching,
authentication, offline-mode, and retry behavior instead of fetching a
hardcoded URL on every model construction.

Collection: TTS

Changelog

• Resolve Edresson/Speaker_Encoder_H_ASP through hf_hub_download.
• Load the returned local checkpoint path with the existing strict=False
  behavior.
• Add a unit test verifying that the cache-aware resolver and local checkpoint
  path are used.

Usage

No public API or configuration changes.

Before your PR is "Ready for review"

Pre checks:

• Read and followed the contributor guidelines.
• Added a regression test.
• No documentation changes are necessary.
• No optional-install impact; huggingface_hub is an existing dependency.

PR Type:

• Bugfix

Testing

• pytest tests/collections/tts/models/test_audio_codec.py -m "not pleasefixme" -v
  — 5 passed
• isort and Black 24.10.0 checks passed

Additional Information

Fixes #15985

Question for maintainers: offline CI behavior

The Magpie functional tests run with HF_HUB_OFFLINE=1 and use a pre-populated Hugging Face cache. The new hf_hub_download() call correctly consults that cache, but Edresson/Speaker_Encoder_H_ASP/pytorch_model.bin is not currently present.
The failure occurs when FrechetCodecDistance restores the codec with use_scl_loss=True, although FCD only uses the codec embeddings and does not use the speaker encoder. The regular Magpie inference loader already overrides use_scl_loss=False.

Which approach would you prefer?

1. Add this checkpoint to the runner-managed Hugging Face cache.
2. Disable use_scl_loss when FCD restores the codec, matching the existing inference path.
3. Defer loading the speaker-encoder checkpoint until speaker embeddings are first requested.

Option 2 appears to be the smallest behavioral change; cache provisioning may still be appropriate for CI that intentionally exercises speaker-consistency loss.

[Edresson]

Hi @janbernloehr ,

Thank you very much for the PR. The proposed change makes sense, and I think we should work toward getting it merged. It should also allow us to remove the fsspec dependency entirely.

I will add the Edresson/Speaker_Encoder_H_ASP model to the Hugging Face cache in our test environment so that the CI tests can pass.

Would you be willing to also remove the remaining fsspec references from the following files?

nemo/collections/tts/modules/audio_codec_modules.py
pyproject.toml
examples/voice_agent/environment.yaml

Otherwise, I am happy to make those changes and push a commit directly to this PR.