Skip to content

Lhotse error when trainer.limit_train_batches=1.0 #16004

Description

@advait-bm

Describe the bug

When trainer.limit_train_batches=1.0, I get TypeError: object of type 'LhotseSpeechToTextBpeDatasetWithPromptIndex' has no len()

OneLogger: Setting error_handling_strategy to DISABLE_QUIETLY_AND_REPORT_METRIC_ERROR for rank (rank=0) with OneLogger disabled. To override: explicitly set error_handling_strategy parameter.
No exporters were provided. This means that no telemetry data will be collected.
Using bfloat16 Automatic Mixed Precision (AMP)
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
HPU available: False, using: 0 HPUs
`Trainer(limit_train_batches=1.0)` was configured so 100% of the batches per epoch will be used..
Error executing job with overrides: ['+init_from_pretrained_model=nvidia/nemotron-3.5-asr-streaming-0.6b', '++model.train_ds.manifest_filepath=/workspace/nemo/manifest/train_manifest.json', '++model.validation_ds.manifest_filepath=/workspace/nemo/manifest/test_manifest.json', '++model.optim.sched.d_model=1024', '++trainer.devices=1', '++trainer.max_epochs=5', '++trainer.max_steps=100000', '++trainer.limit_train_batches=1.0', '++trainer.precision=bf16', '++model.optim.lr=1e-4', '++model.optim.weight_decay=0.001', '++exp_manager.version=train', '++exp_manager.use_datetime_version=False', '++exp_manager.exp_dir=/workspace/nemo/checkpoints']
Traceback (most recent call last):
  File "/workspace/nemo/NeMo/examples/asr/speech_to_text_finetune.py", line 217, in main
    asr_model = setup_dataloaders(asr_model, cfg)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/nemo/NeMo/examples/asr/speech_to_text_finetune.py", line 191, in setup_dataloaders
    asr_model.setup_training_data(cfg.model.train_ds)
  File "/workspace/nemo/NeMo/nemo/collections/asr/models/rnnt_bpe_models_prompt.py", line 273, in setup_training_data
    * ceil((len(self._train_dl.dataset) / self.world_size) / train_data_config['batch_size'])
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: object of type 'LhotseSpeechToTextBpeDatasetWithPromptIndex' has no len()

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

Steps/Code to reproduce bug
Set trainer.limit_train_batches=1.0

Expected behavior

No errors.

Environment overview (please complete the following information)

  • Environment location: 1 H100 (runpod.io)
  • Method of NeMo install: From source as in README.md

Environment details

  • OS version: Ubuntu 24.04
  • PyTorch version: 2.12.0+cu132
  • Python version: 3.12.3

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions