`audio_to_text` doesn't support duplicate files with different offsets
This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days. This issue was closed because it has been inactive for 7 days since being marked as stale. Hi, I'm facing the same problem in NeMo 2.0. It took me a long time to see that the '-sub1', 'sub2'.... don't work. This kind a huge issue since with audio lasting up to 1 hour, you don't want to divide them into every segment you have. I believe you'll be able to avoid these issues by switching to lhotse dataloaders. For ASR recipes,  just add `train_ds.use_lhotse: True`. > I believe you'll be able to avoid these issues by switching to lhotse dataloaders. For ASR recipes, just add `train_ds.use_lhotse: True`.

Hi, thanks for the tip but when I try to use lhotse dataloaders I have the following error:

```
[NeMo I 2024-11-27 10:24:19 nemo_logging:381] Creating a Lhotse DynamicCutSampler (bucketing is disabled, (max_batch_duration=None max_batch_size=1)
Error executing job with overrides: []
Traceback (most recent call last):
  File "NeMo/nemo/collections/asr/models/ctc_models.py", line 385, in setup_training_data
    * ceil((len(self._train_dl.dataset) / self.world_size) / train_data_config['batch_size'])
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: object of type 'IterableDatasetWrapper' has no len()
```

I have switched to using sharded manifests as it seems to be required for that type of dataloader. I don't know if this is normal and I need to change something else? but what?

I have the following config for my train_ds:
```
train_ds:
    manifest_filepath:
    - - bucket11/sharded_manifests/manifest__OP_0..23_CL_.json
    - - bucket12/sharded_manifests/manifest__OP_0..23_CL_.json
    tarred_audio_filepaths:
    - - bucket11/audio__OP_0..23_CL_.tar
    - - bucket12/audio__OP_0..23_CL_.tar
    sample_rate: ${model.sample_rate}
    batch_size: 1 # must be 1 if using bucketing_batch_size
    shuffle: true
    num_workers: 8
    use_lhotse: true
    pin_memory: true
    max_duration: 30.1
    min_duration: 0.1
    shuffle_n: 2048
    # tarred datasets
    is_tarred: true
    bucketing_strategy: "synced_randomized" # "fixed_order" "synced_randomized" "fully_randomized"
    bucketing_weights: [1, 1]
    bucketing_batch_size: 8
    shard_strategy: scatter
```
@pzelasko 
When building tarred datasets with `convert_to_tarred_dataset` the script treats duplicate files with concatenating `-sub` to the file name. This allows the manifest to track number of samples while avoiding too many duplicate files in the tar balls. For `audio_to_label` data, this `-sub` is picked up by the class and used to duplicate audio streams.

For `audio_to_text` data, there is not support for this concatenation, the `_loop_offset` method still assumes unique files. As such, anytime you use multiple offsets from the same audio file, the dataloader conflicts with the audio stream, causing hanging.

**Steps/Code to reproduce bug**
- Build a manifest file with multiple offsets from same file
- Run convert_to_tarred
- Run training over tarred script
- Watch run hang

**Expected behavior**

- The `audio_to_text` data file should duplicate audio streams for each instance of `-sub` in the manifest. This can be done by simply duplicating the text in `audio_to_label`

**Environment overview (please complete the following information)**

 - Environment location: [Docker container on cs-oci run]
 - Method of NeMo install: source from main

