ZeroDivisionError: integer division or modulo by zero || Ambernet fine-tuning
**Describe the bug**

I am trying to fine-tune the pre-trained ambernet-based language identification using [Mozzila Common Voice dataset ](https://commonvoice.mozilla.org/) by freezing the encoder and fine-tune only the classifier module.
` /NeMo/NeMo_1.12.0/ambernet/16d76d1aa14af9b770fabdac12fcf8e0/ambernet.nemo`

I have `65226 files loaded accounting to # 4 labels` train dataset and `774 files loaded accounting to # 4 labels` validation dataset. 

I am using the default training [configuration ](https://github.com/NVIDIA/NeMo/tree/main/examples/asr/conf/lang_id) with below changes:
```
init_from_nemo_model:
  model:
   path: ./.cache/torch/NeMo/NeMo_1.12.0/ambernet/16d76d1aa14af9b770fabdac12fcf8e0/ambernet.nemo

 decoder:
   _target_: nemo.collections.asr.modules.SpeakerDecoder
   feat_in: *enc_feat_out
   num_classes: 4
   pool_mode: 'xvector' # 'attention'
   emb_sizes: 512
```

**Expected behavior**

All the parameters are loaded correctly on the pre-trained model and the sanity check are passed. The training starts good and it is expected to proceed with iterations/epochs to fine-tune the model instead, I am getting the below error during the training.  

```
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
`Trainer(val_check_interval=1.0)` was configured so validation will run at the end of the training epoch..
[NeMo I 2022-11-21 14:03:57 collections:297] Filtered duration for loading collection is 0.000000.
[NeMo I 2022-11-21 14:03:57 collections:301] # 65226 files loaded accounting to # 4 labels
[NeMo W 2022-11-21 14:03:57 label_models:194] Total number of 4 found in all the manifest files.
[NeMo I 2022-11-21 14:03:57 collections:297] Filtered duration for loading collection is 0.000000.
[NeMo I 2022-11-21 14:03:57 collections:301] # 65226 files loaded accounting to # 4 labels
[NeMo I 2022-11-21 14:03:57 collections:297] Filtered duration for loading collection is 0.000000.
[NeMo I 2022-11-21 14:03:57 collections:301] # 774 files loaded accounting to # 4 labels
[NeMo I 2022-11-21 14:03:57 collections:297] Filtered duration for loading collection is 0.000000.
[NeMo I 2022-11-21 14:03:57 collections:301] # 774 files loaded accounting to # 4 labels
[NeMo I 2022-11-21 14:03:57 cross_entropy:55] Weighted Cross Entropy loss with weight tensor([0.7663, 1.0014, 1.3230, 1.0631])
[NeMo W 2022-11-21 14:03:57 nemo_logging:349] /home/thayabaran.kathiresan/anaconda3/envs/nemo_LID/lib/python3.9/site-packages/torchmetrics/utilities/prints.py:36: UserWarning: Torchmetrics v0.9 introduced a new argument class property called `full_state_update` that has
                    not been set for this class (TopKClassificationAccuracy). The property determines if `update` by
                    default needs access to the full metric state. If this is not the case, significant speedups can be
                    achieved and we recommend setting this to `False`.
                    We provide an checking function
                    `from torchmetrics.utilities import check_forward_full_state_property`
                    that can be used to check if the `full_state_update=True` (old and potential slower behaviour,
                    default for now) or if `full_state_update=False` can be used safely.
                    
      warnings.warn(*args, **kwargs)
    
[NeMo I 2022-11-21 14:03:57 features:225] PADDING: 16
Model encoder has been frozen
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [7]
[NeMo I 2022-11-21 14:03:59 modelPT:597] Optimizer config = SGD (
    Parameter Group 0
        dampening: 0
        differentiable: False
        foreach: None
        lr: 0.001
        maximize: False
        momentum: 0
        nesterov: False
        weight_decay: 0.001
    )
[NeMo I 2022-11-21 14:03:59 lr_scheduler:910] Scheduler "<nemo.core.optim.lr_scheduler.CosineAnnealing object at 0x7fd62f330a90>" 
    will be used during training (effective maximum steps = 25500) - 
    Parameters : 
    (warmup_ratio: 0.1
    min_lr: 0.0001
    max_steps: 25500
    )

  | Name              | Type                              | Params
------------------------------------------------------------------------
0 | loss              | CrossEntropyLoss                  | 0     
1 | eval_loss         | CrossEntropyLoss                  | 0     
2 | _accuracy         | TopKClassificationAccuracy        | 0     
3 | preprocessor      | AudioToMelSpectrogramPreprocessor | 0     
4 | encoder           | ConvASREncoder                    | 19.4 M
5 | decoder           | SpeakerDecoder                    | 3.1 M 
6 | _macro_accuracy   | Accuracy                          | 0     
7 | spec_augmentation | SpectrogramAugmentation           | 0     
------------------------------------------------------------------------
3.1 M     Trainable params
19.4 M    Non-trainable params
22.5 M    Total params
90.009    Total estimated model params size (MB)

Sanity Checking: 0it [00:00, ?it/s]
Sanity Checking:   0%|          | 0/2 [00:00<?, ?it/s]
Sanity Checking DataLoader 0:   0%|          | 0/2 [00:00<?, ?it/s]
Sanity Checking DataLoader 0:  50%|█████     | 1/2 [00:01<00:01,  1.00s/it]
Sanity Checking DataLoader 0: 100%|██████████| 2/2 [00:01<00:00,  1.40it/s]
                                                                           

Training: 0it [00:00, ?it/s]
Training:   0%|          | 0/517 [00:00<?, ?it/s]
Epoch 0:   0%|          | 0/517 [00:00<?, ?it/s] [NeMo W 2022-11-21 14:04:03 nemo_logging:349] /home/thayabaran.kathiresan/anaconda3/envs/nemo_LID/lib/python3.9/site-packages/pytorch_lightning/trainer/connectors/logger_connector/result.py:231: UserWarning: You called `self.log('global_step', ...)` in your `training_step` but the value needs to be floating point. Converting it to torch.float32.
      warning_cache.warn(
    

Epoch 0:   0%|          | 1/517 [00:01<16:14,  1.89s/it]
Epoch 0:   0%|          | 1/517 [00:01<16:14,  1.89s/it, loss=1.67]
Epoch 0:   0%|          | 2/517 [00:02<09:49,  1.14s/it, loss=1.67]
Epoch 0:   0%|          | 2/517 [00:02<09:49,  1.14s/it, loss=1.63]
Epoch 0:   1%|          | 3/517 [00:02<07:38,  1.12it/s, loss=1.63]
Epoch 0:   1%|          | 3/517 [00:02<07:38,  1.12it/s, loss=1.66]
Epoch 0:   1%|          | 4/517 [00:03<06:33,  1.30it/s, loss=1.66]
Epoch 0:   1%|          | 4/517 [00:03<06:33,  1.30it/s, loss=1.63]
Epoch 0:   1%|          | 5/517 [00:03<05:51,  1.46it/s, loss=1.63]
Epoch 0:   1%|          | 5/517 [00:03<05:51,  1.46it/s, loss=1.63]
Epoch 0:   1%|          | 6/517 [00:03<05:26,  1.57it/s, loss=1.63]
Epoch 0:   1%|          | 6/517 [00:03<05:26,  1.57it/s, loss=1.63]
Epoch 0:   1%|▏         | 7/517 [00:04<05:07,  1.66it/s, loss=1.63]
Epoch 0:   1%|▏         | 7/517 [00:04<05:07,  1.66it/s, loss=1.62]
Epoch 0:   2%|▏         | 8/517 [00:04<04:52,  1.74it/s, loss=1.62]
Epoch 0:   2%|▏         | 8/517 [00:04<04:52,  1.74it/s, loss=1.62]
Epoch 0:   2%|▏         | 9/517 [00:04<04:41,  1.81it/s, loss=1.62]
Epoch 0:   2%|▏         | 9/517 [00:04<04:41,  1.81it/s, loss=1.63]
Epoch 0:   2%|▏         | 10/517 [00:05<04:30,  1.87it/s, loss=1.63]
Epoch 0:   2%|▏         | 10/517 [00:05<04:30,  1.87it/s, loss=1.63]
Epoch 0:   2%|▏         | 11/517 [00:05<04:22,  1.93it/s, loss=1.63]
Epoch 0:   2%|▏         | 11/517 [00:05<04:22,  1.93it/s, loss=1.63]
Epoch 0:   2%|▏         | 12/517 [00:06<04:31,  1.86it/s, loss=1.63]
Epoch 0:   2%|▏         | 12/517 [00:06<04:31,  1.86it/s, loss=1.64]
Epoch 0:   3%|▎         | 13/517 [00:06<04:23,  1.91it/s, loss=1.64]
Epoch 0:   3%|▎         | 13/517 [00:06<04:23,  1.91it/s, loss=1.63]
Epoch 0:   3%|▎         | 14/517 [00:07<04:18,  1.95it/s, loss=1.63]
Epoch 0:   3%|▎         | 14/517 [00:07<04:18,  1.95it/s, loss=1.64]
Epoch 0:   3%|▎         | 15/517 [00:07<04:12,  1.99it/s, loss=1.64]
Epoch 0:   3%|▎         | 15/517 [00:07<04:12,  1.99it/s, loss=1.63]
Epoch 0:   3%|▎         | 16/517 [00:07<04:07,  2.02it/s, loss=1.63]
Epoch 0:   3%|▎         | 16/517 [00:07<04:08,  2.02it/s, loss=1.63]
Epoch 0:   3%|▎         | 17/517 [00:08<04:05,  2.04it/s, loss=1.63]
Epoch 0:   3%|▎         | 17/517 [00:08<04:05,  2.04it/s, loss=1.63]
Epoch 0:   3%|▎         | 18/517 [00:08<04:02,  2.06it/s, loss=1.63]
Epoch 0:   3%|▎         | 18/517 [00:08<04:02,  2.06it/s, loss=1.63]
Epoch 0:   4%|▎         | 19/517 [00:10<04:30,  1.84it/s, loss=1.63]
Epoch 0:   4%|▎         | 19/517 [00:10<04:30,  1.84it/s, loss=1.64]
Epoch 0:   4%|▍         | 20/517 [00:10<04:26,  1.87it/s, loss=1.64]
Epoch 0:   4%|▍         | 20/517 [00:10<04:26,  1.87it/s, loss=1.64]
Epoch 0:   4%|▍         | 21/517 [00:11<04:27,  1.85it/s, loss=1.64]
Epoch 0:   4%|▍         | 21/517 [00:11<04:27,  1.85it/s, loss=1.63]
Epoch 0:   4%|▍         | 22/517 [00:11<04:27,  1.85it/s, loss=1.63]
Epoch 0:   4%|▍         | 22/517 [00:11<04:27,  1.85it/s, loss=1.63]
Epoch 0:   4%|▍         | 23/517 [00:12<04:23,  1.87it/s, loss=1.63]
Epoch 0:   4%|▍         | 23/517 [00:12<04:23,  1.87it/s, loss=1.62]
Epoch 0:   5%|▍         | 24/517 [00:12<04:19,  1.90it/s, loss=1.62]
Epoch 0:   5%|▍         | 24/517 [00:12<04:19,  1.90it/s, loss=1.62]
Epoch 0:   5%|▍         | 25/517 [00:13<04:15,  1.92it/s, loss=1.62]
Epoch 0:   5%|▍         | 25/517 [00:13<04:15,  1.92it/s, loss=1.62]
Epoch 0:   5%|▌         | 26/517 [00:13<04:12,  1.94it/s, loss=1.62]
Epoch 0:   5%|▌         | 26/517 [00:13<04:12,  1.94it/s, loss=1.62]
Epoch 0:   5%|▌         | 27/517 [00:13<04:09,  1.96it/s, loss=1.62]
Epoch 0:   5%|▌         | 27/517 [00:13<04:09,  1.96it/s, loss=1.63]
Epoch 0:   5%|▌         | 28/517 [00:14<04:07,  1.98it/s, loss=1.63]
Epoch 0:   5%|▌         | 28/517 [00:14<04:07,  1.98it/s, loss=1.63]
Epoch 0:   6%|▌         | 29/517 [00:14<04:04,  2.00it/s, loss=1.63]
Epoch 0:   6%|▌         | 29/517 [00:14<04:04,  2.00it/s, loss=1.63]
Epoch 0:   6%|▌         | 30/517 [00:14<04:01,  2.02it/s, loss=1.63]
Epoch 0:   6%|▌         | 30/517 [00:14<04:01,  2.02it/s, loss=1.63]
Epoch 0:   6%|▌         | 31/517 [00:15<03:59,  2.03it/s, loss=1.63]
Epoch 0:   6%|▌         | 31/517 [00:15<03:59,  2.03it/s, loss=1.62]
Epoch 0:   6%|▌         | 32/517 [00:15<03:56,  2.05it/s, loss=1.62]
Epoch 0:   6%|▌         | 32/517 [00:15<03:56,  2.05it/s, loss=1.62]
Epoch 0:   6%|▋         | 33/517 [00:15<03:54,  2.06it/s, loss=1.62]
Epoch 0:   6%|▋         | 33/517 [00:15<03:54,  2.06it/s, loss=1.62]
Epoch 0:   7%|▋         | 34/517 [00:16<03:52,  2.08it/s, loss=1.62]
Epoch 0:   7%|▋         | 34/517 [00:16<03:52,  2.08it/s, loss=1.62]
Epoch 0:   7%|▋         | 35/517 [00:16<03:51,  2.09it/s, loss=1.62]
Epoch 0:   7%|▋         | 35/517 [00:16<03:51,  2.09it/s, loss=1.62]
Epoch 0:   7%|▋         | 36/517 [00:17<03:49,  2.10it/s, loss=1.62]
Epoch 0:   7%|▋         | 36/517 [00:17<03:49,  2.10it/s, loss=1.62]
Epoch 0:   7%|▋         | 37/517 [00:17<03:48,  2.10it/s, loss=1.62]
Epoch 0:   7%|▋         | 37/517 [00:17<03:48,  2.10it/s, loss=1.62]
Epoch 0:   7%|▋         | 38/517 [00:17<03:46,  2.12it/s, loss=1.62]
Epoch 0:   7%|▋         | 38/517 [00:17<03:46,  2.12it/s, loss=1.62]
Epoch 0:   8%|▊         | 39/517 [00:18<03:44,  2.13it/s, loss=1.62]
Epoch 0:   8%|▊         | 39/517 [00:18<03:44,  2.13it/s, loss=1.63]
Traceback (most recent call last):
  File "/home/thayabaran.kathiresan/nemo_LID/TL_Scripts/TL_Nemo_Training.py", line 32, in <module>
    trainer.fit(ambernet_model)
  File "/home/thayabaran.kathiresan/anaconda3/envs/nemo_LID/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 696, in fit
    self._call_and_handle_interrupt(
  File "/home/thayabaran.kathiresan/anaconda3/envs/nemo_LID/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 650, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/home/thayabaran.kathiresan/anaconda3/envs/nemo_LID/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 735, in _fit_impl
    results = self._run(model, ckpt_path=self.ckpt_path)
  File "/home/thayabaran.kathiresan/anaconda3/envs/nemo_LID/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1166, in _run
    results = self._run_stage()
  File "/home/thayabaran.kathiresan/anaconda3/envs/nemo_LID/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1252, in _run_stage
    return self._run_train()
  File "/home/thayabaran.kathiresan/anaconda3/envs/nemo_LID/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1283, in _run_train
    self.fit_loop.run()
  File "/home/thayabaran.kathiresan/anaconda3/envs/nemo_LID/lib/python3.9/site-packages/pytorch_lightning/loops/loop.py", line 200, in run
    self.advance(*args, **kwargs)
  File "/home/thayabaran.kathiresan/anaconda3/envs/nemo_LID/lib/python3.9/site-packages/pytorch_lightning/loops/fit_loop.py", line 271, in advance
    self._outputs = self.epoch_loop.run(self._data_fetcher)
  File "/home/thayabaran.kathiresan/anaconda3/envs/nemo_LID/lib/python3.9/site-packages/pytorch_lightning/loops/loop.py", line 200, in run
    self.advance(*args, **kwargs)
  File "/home/thayabaran.kathiresan/anaconda3/envs/nemo_LID/lib/python3.9/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 174, in advance
    batch = next(data_fetcher)
  File "/home/thayabaran.kathiresan/anaconda3/envs/nemo_LID/lib/python3.9/site-packages/pytorch_lightning/utilities/fetching.py", line 184, in __next__
    return self.fetching_function()
  File "/home/thayabaran.kathiresan/anaconda3/envs/nemo_LID/lib/python3.9/site-packages/pytorch_lightning/utilities/fetching.py", line 263, in fetching_function
    self._fetch_next_batch(self.dataloader_iter)
  File "/home/thayabaran.kathiresan/anaconda3/envs/nemo_LID/lib/python3.9/site-packages/pytorch_lightning/utilities/fetching.py", line 277, in _fetch_next_batch
    batch = next(iterator)
  File "/home/thayabaran.kathiresan/anaconda3/envs/nemo_LID/lib/python3.9/site-packages/pytorch_lightning/trainer/supporters.py", line 557, in __next__
    return self.request_next_batch(self.loader_iters)
  File "/home/thayabaran.kathiresan/anaconda3/envs/nemo_LID/lib/python3.9/site-packages/pytorch_lightning/trainer/supporters.py", line 569, in request_next_batch
    return apply_to_collection(loader_iters, Iterator, next)
  File "/home/thayabaran.kathiresan/anaconda3/envs/nemo_LID/lib/python3.9/site-packages/pytorch_lightning/utilities/apply_func.py", line 99, in apply_to_collection
    return function(data, *args, **kwargs)
  File "/home/thayabaran.kathiresan/anaconda3/envs/nemo_LID/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 628, in __next__
    data = self._next_data()
  File "/home/thayabaran.kathiresan/anaconda3/envs/nemo_LID/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1333, in _next_data
    return self._process_data(data)
  File "/home/thayabaran.kathiresan/anaconda3/envs/nemo_LID/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1359, in _process_data
    data.reraise()
  File "/home/thayabaran.kathiresan/anaconda3/envs/nemo_LID/lib/python3.9/site-packages/torch/_utils.py", line 543, in reraise
    raise exception
ZeroDivisionError: Caught ZeroDivisionError in DataLoader worker process 7.
Original Traceback (most recent call last):
  File "/home/thayabaran.kathiresan/anaconda3/envs/nemo_LID/lib/python3.9/site-packages/torch/utils/data/_utils/worker.py", line 302, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/thayabaran.kathiresan/anaconda3/envs/nemo_LID/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 61, in fetch
    return self.collate_fn(data)
  File "/home/thayabaran.kathiresan/anaconda3/envs/nemo_LID/lib/python3.9/site-packages/nemo/collections/asr/data/audio_to_label.py", line 436, in fixed_seq_collate_fn
    return _fixed_seq_collate_fn(self, batch)
  File "/home/thayabaran.kathiresan/anaconda3/envs/nemo_LID/lib/python3.9/site-packages/nemo/collections/asr/data/audio_to_label.py", line 133, in _fixed_seq_collate_fn
    repeat = fixed_length // sig_len
ZeroDivisionError: integer division or modulo by zero
```

**Environment overview**
 - Method of NeMo install: [pip install or from source]. 
```
apt-get update && apt-get install -y libsndfile1 ffmpeg
pip install Cython
pip install nemo_toolkit[all]
```
**Additional Information**

- Train and Validation manifest have been checked for any bad-audio/corrupted files but, the data set is clean and according to the input standard
- The training is crashing at different iterations every time but, never completed 1 Epoch 
- Tried running in CPU and it produces the same crash.
