wav2vec training - regression ? 
This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days. This issue was closed because it has been inactive for 7 days since being marked as stale. Not sure why this one was closed? From what I see `wav2vec` does not work
@titu1994 @tbartley94 
 @orena1  Sorry, the CI bot closes it if there's no activity. On the docket to look at. This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days. This issue was closed because it has been inactive for 7 days since being marked as stale.
**Describe the bug**

Hi,
I am trying to train wav2vec as in this ticket:
https://github.com/NVIDIA/NeMo/issues/4362

but I get an error, this is the script that I use:
```
python speech_to_text_ctc.py \
--config-path='../conf/wav2vec_ctc'  \
--config-name=wav2vecCTC \
model.train_ds.manifest_filepath=/mnt/NVM.SGPU01/oren/libri_mini/train_clean_5.json  \
model.train_ds.batch_size=10 \
model.validation_ds.batch_size=10 \
model.validation_ds.manifest_filepath=/mnt/NVM.SGPU01/oren//libri_mini/dev_clean_2.json  \
model.tokenizer.dir=/home/oren.amsalem/.cache/torch/hub/torchaudio/tutorial-assets/spm_unigram_1023.model \
```

I get this error:
```
Error executing job with overrides: ['model.train_ds.manifest_filepath=/mnt/NVM.SGPU01/oren/libri_mini/train_clean_5.json', 'model.train_ds.batch_size=10', 'model.validation_ds.batch_size=10', 'model.validation_ds.manifest_filepath=/mnt/NVM.SGPU01/oren//libri_mini/dev_clean_2.json', 'model.tokenizer.dir=/home/oren.amsalem/.cache/torch/hub/torchaudio/tutorial-assets/spm_unigram_1023.model']
Traceback (most recent call last):
  File "/home/oren.amsalem/cloned/NeMo/examples/asr/asr_ctc/speech_to_text_ctc.py", line 91, in main
    trainer.fit(asr_model)
  File "/mnt/NVM/oren/anaconda3_10/envs/nemo/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 532, in fit
    call._call_and_handle_interrupt(
  File "/mnt/NVM/oren/anaconda3_10/envs/nemo/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 42, in _call_and_handle_interrupt
    return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
  File "/mnt/NVM/oren/anaconda3_10/envs/nemo/lib/python3.10/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 93, in launch
    return function(*args, **kwargs)
  File "/mnt/NVM/oren/anaconda3_10/envs/nemo/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 571, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "/mnt/NVM/oren/anaconda3_10/envs/nemo/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 980, in _run
    results = self._run_stage()
  File "/mnt/NVM/oren/anaconda3_10/envs/nemo/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1023, in _run_stage
    self.fit_loop.run()
  File "/mnt/NVM/oren/anaconda3_10/envs/nemo/lib/python3.10/site-packages/pytorch_lightning/loops/fit_loop.py", line 202, in run
    self.advance()
  File "/mnt/NVM/oren/anaconda3_10/envs/nemo/lib/python3.10/site-packages/pytorch_lightning/loops/fit_loop.py", line 355, in advance
    self.epoch_loop.run(self._data_fetcher)
  File "/mnt/NVM/oren/anaconda3_10/envs/nemo/lib/python3.10/site-packages/pytorch_lightning/loops/training_epoch_loop.py", line 133, in run
    self.advance(data_fetcher)
  File "/mnt/NVM/oren/anaconda3_10/envs/nemo/lib/python3.10/site-packages/pytorch_lightning/loops/training_epoch_loop.py", line 219, in advance
    batch_output = self.automatic_optimization.run(trainer.optimizers[0], kwargs)
  File "/mnt/NVM/oren/anaconda3_10/envs/nemo/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 188, in run
    self._optimizer_step(kwargs.get("batch_idx", 0), closure)
  File "/mnt/NVM/oren/anaconda3_10/envs/nemo/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 266, in _optimizer_step
    call._call_lightning_module_hook(
  File "/mnt/NVM/oren/anaconda3_10/envs/nemo/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 145, in _call_lightning_module_hook
    output = fn(*args, **kwargs)
  File "/mnt/NVM/oren/anaconda3_10/envs/nemo/lib/python3.10/site-packages/pytorch_lightning/core/module.py", line 1270, in optimizer_step
    optimizer.step(closure=optimizer_closure)
  File "/mnt/NVM/oren/anaconda3_10/envs/nemo/lib/python3.10/site-packages/pytorch_lightning/core/optimizer.py", line 161, in step
    step_output = self._strategy.optimizer_step(self._optimizer, closure, **kwargs)
  File "/mnt/NVM/oren/anaconda3_10/envs/nemo/lib/python3.10/site-packages/pytorch_lightning/strategies/ddp.py", line 257, in optimizer_step
    optimizer_output = super().optimizer_step(optimizer, closure, model, **kwargs)
  File "/mnt/NVM/oren/anaconda3_10/envs/nemo/lib/python3.10/site-packages/pytorch_lightning/strategies/strategy.py", line 231, in optimizer_step
    return self.precision_plugin.optimizer_step(optimizer, model=model, closure=closure, **kwargs)
  File "/mnt/NVM/oren/anaconda3_10/envs/nemo/lib/python3.10/site-packages/pytorch_lightning/plugins/precision/precision_plugin.py", line 116, in optimizer_step
    return optimizer.step(closure=closure, **kwargs)
  File "/mnt/NVM/oren/anaconda3_10/envs/nemo/lib/python3.10/site-packages/torch/optim/lr_scheduler.py", line 68, in wrapper
    return wrapped(*args, **kwargs)
  File "/mnt/NVM/oren/anaconda3_10/envs/nemo/lib/python3.10/site-packages/torch/optim/optimizer.py", line 373, in wrapper
    out = func(*args, **kwargs)
  File "/mnt/NVM/oren/anaconda3_10/envs/nemo/lib/python3.10/site-packages/torch/optim/optimizer.py", line 76, in _use_grad
    ret = func(self, *args, **kwargs)
  File "/mnt/NVM/oren/anaconda3_10/envs/nemo/lib/python3.10/site-packages/torch/optim/adamw.py", line 161, in step
    loss = closure()
  File "/mnt/NVM/oren/anaconda3_10/envs/nemo/lib/python3.10/site-packages/pytorch_lightning/plugins/precision/precision_plugin.py", line 103, in _wrap_closure
    closure_result = closure()
  File "/mnt/NVM/oren/anaconda3_10/envs/nemo/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 142, in __call__
    self._result = self.closure(*args, **kwargs)
  File "/mnt/NVM/oren/anaconda3_10/envs/nemo/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/mnt/NVM/oren/anaconda3_10/envs/nemo/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 128, in closure
    step_output = self._step_fn()
  File "/mnt/NVM/oren/anaconda3_10/envs/nemo/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 315, in _training_step
    training_step_output = call._call_strategy_hook(trainer, "training_step", *kwargs.values())
  File "/mnt/NVM/oren/anaconda3_10/envs/nemo/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 293, in _call_strategy_hook
    output = fn(*args, **kwargs)
  File "/mnt/NVM/oren/anaconda3_10/envs/nemo/lib/python3.10/site-packages/pytorch_lightning/strategies/ddp.py", line 330, in training_step
    return self.model(*args, **kwargs)
  File "/mnt/NVM/oren/anaconda3_10/envs/nemo/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/mnt/NVM/oren/anaconda3_10/envs/nemo/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/mnt/NVM/oren/anaconda3_10/envs/nemo/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1515, in forward
    inputs, kwargs = self._pre_forward(*inputs, **kwargs)
  File "/mnt/NVM/oren/anaconda3_10/envs/nemo/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1409, in _pre_forward
    if torch.is_grad_enabled() and self.reducer._rebuild_buckets():
RuntimeError: It looks like your LightningModule has parameters that were not used in producing the loss returned by training_step. If this is intentional, you must enable the detection of unused parameters in DDP, either by setting the string value `strategy='ddp_find_unused_parameters_true'` or by setting the flag in the strategy with `strategy=DDPStrategy(find_unused_parameters=True)`.

```


I've tried using `accelerator=cpu` but I get an error about the device - here:
https://github.com/NVIDIA/NeMo/pull/8084
When I fix that error as in the pull request I get this error:

```
377.513   Total estimated model params size (MB)
Epoch 0:   0%|                                                                                                                                                       | 0/152 [00:00<?, ?it/s]
double free or corruption (out)
Aborted (core dumped)
```

**Environment overview (please complete the following information)** 
**Environment details**
torch 2.1.2
python 3.10

