CUDA out of memory after 44 epochs
**Describe the bug**

Hi
I am trying to run QuartzNet 10x5 model on my dataset of almost 200 hours audio. But the strange thing is I got this error:

```
RuntimeError: CUDA out of memory. Tried to allocate 130.00 MiB (GPU 0; 11.91 GiB total capacity; 10.99 GiB already allocated; 50.38 MiB free; 11.19 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
```

I can't understand why I got out of memory error after 44 epochs!

**Steps/Code to reproduce bug**

TRAIN_DS  config:
```
  batch_size: 4
  trim_silence: true
  max_duration: 62.0
  normalize_transcripts: false
  shuffle: true
  num_workers: 8
  pin_memory: true
  is_tarred: false
  tarred_audio_filepaths: null
  shuffle_n: 2048
  bucketing_strategy: synced_randomized
  bucketing_batch_size: null
  augmentor:
    white_noise:
      prob: 0.4
      min_level: -90
      max_level: -46
    gain:
      prob: 0.4
      min_gain_dbfs: 0
      max_gain_dbfs: 50
    speed:
      prob: 0.3
      sr: 16000
      resample_type: kaiser_best
      min_speed_rate: 0.5
      max_speed_rate: 2.0
      num_rates: -1
    time_stretch:
      prob: 0.3
      min_speed_rate: 0.5
      max_speed_rate: 2.0
      num_rates: -1
    noise:
      prob: 0.45
      manifest_path: ./noise_manifest.json
      min_snr_db: -4
      max_snr_db: 10
      max_gain_db: 300.0
```
VALIDATION_DS config:

```
  batch_size: 4
  normalize_transcripts: false
  shuffle: false
  num_workers: 8
  pin_memory: true
  trim_silence: true
```

TRAINER config:
```
trainer = ptl.Trainer(devices=-1,
                      accelerator='gpu',
                      max_epochs=EPOCHS,
                      accumulate_grad_batches=1,
                      enable_checkpointing=False,
                      logger=False,
                      log_every_n_steps=5,
                      check_val_every_n_epoch=1,
                      strategy="ddp")
```

**Environment overview (please complete the following information)**

 - Environment location: Bare-metal
 - Method of NeMo install: pip install nemo_toolkit['all']
 

**Environment details**

- OS version : Ubuntu 20.04 LTS
- PyTorch version:   1.12.1
- Python version: 3.8.0

**Additional context**

I ran my model on 2 GPU's : 
TITAN X 12288MiB and
GTX 1080 Ti 11264MiB
