Fix Lhotse training limits for prompt ASR models - #16019
Conversation
Signed-off-by: SiavashShams <siavashshams.ac@gmail.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6d41e2e90e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| and hasattr(self._train_dl, 'dataset') | ||
| and isinstance(self._train_dl.dataset, torch.utils.data.IterableDataset) |
There was a problem hiding this comment.
When either prompt model uses a non-Lhotse Hugging Face streaming dataset (streaming: true with hf_data_cfg), the dataset factory returns HFIterableAudioToBPEDataset, whose __len__ explicitly raises NotImplementedError (hf_audio_to_text.py:483-487). This broadened IterableDataset condition therefore reaches len(self._train_dl.dataset) during setup and prevents training from starting; the previous is_tarred guard skipped this supported path. Restrict the conversion to sized/tarred iterable datasets, and apply the same correction to the mirrored hybrid implementation.
Useful? React with 👍 / 👎.
Signed-off-by: SiavashShams <siavashshams.ac@gmail.com>
svcnvidia-nemo-ci
added
the
waiting-on-maintainers
2 participants
Important
The
Update branchbutton must only be pressed in very rare occassions.An outdated branch is never blocking the merge of a PR.
Please reach out to the automation team before pressing that button.
What does this PR do ?
Prevent prompt-conditioned RNNT and hybrid ASR models from treating Lhotse or unsized Hugging Face streaming loaders as native tarred iterable datasets when normalizing
limit_train_batches.Collection: ASR
Changelog
limit_train_batchesvalues unchanged for Lhotse prompt training loaders.IterableDatasetloaders.Usage
trainer.limit_train_batches=1.0can be used with finite prompt-conditioned Lhotse training manifests without overriding the dataset configuration.GitHub Actions CI
The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.
The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".
Local validation:
pytest -p no:cacheprovider -m unit -v tests/collections/asr/test_asr_rnnt_encoder_model_bpe_prompt.py tests/collections/asr/test_asr_hybrid_rnnt_ctc_model_bpe_prompt.py -k "setup_training_data and limit_train_batches"— 6 passed.pytest -p no:cacheprovider -m unit -v tests/collections/asr/test_asr_rnnt_encoder_model_bpe_prompt.py tests/collections/asr/test_asr_hybrid_rnnt_ctc_model_bpe_prompt.py— 33 passed.isort, Black, andsetup.py stylechecks passed.pytest -p no:cacheprovider tests/collections/asr -m "not pleasefixme" --cpu -vwas stopped after 140.75 seconds with 212 passed and 18 skipped; unrelated failures required unavailable model data, a writable external model cache, ormatplotlib.Before your PR is "Ready for review"
Pre checks:
PR Type:
If you haven't finished some of the above items you can still open "Draft" PR.
Who can review?
Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.
Additional Information