[Speaker Recognition] "RuntimeError: CUDA out of memory" with small batch size
**Describe the bug**
I m training the speaker model with CN-celeb2 dataset, but no matter what batch_size is used, it still appear CUDA out of memory. Even set batch_size=2, GPU memory will continuously increase until out of memory.
Is any parameter that should be adjusted?

bug info:
`RuntimeError: CUDA out of memory. Tried to allocate 6.31 GiB (GPU 0; 15.78 GiB total capacity; 9.07 GiB already allocated; 3.26 GiB free; 11.37 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF`

speaker_reco.py
```
import os
import pytorch_lightning as pl
from omegaconf import OmegaConf
from pytorch_lightning import seed_everything
from nemo.collections.asr.models import EncDecSpeakerLabelModel
from nemo.core.config import hydra_runner
from nemo.utils import logging
from nemo.utils.exp_manager import exp_manager

seed_everything(42)

@hydra_runner(config_path="conf", config_name="ecapa_tdnn.yaml")
def main(cfg):

    logging.info(f'Hydra config: {OmegaConf.to_yaml(cfg)}')
    trainer = pl.Trainer(**cfg.trainer)
    log_dir = exp_manager(trainer, cfg.get("exp_manager", None))
    speaker_model = EncDecSpeakerLabelModel(cfg=cfg.model, trainer=trainer)
    trainer.fit(speaker_model)
    if not trainer.fast_dev_run:
        model_path = os.path.join(log_dir, '..', 'spkr.nemo')
        speaker_model.save_to(model_path)

    if hasattr(cfg.model, 'test_ds') and cfg.model.test_ds.manifest_filepath is not None:
        trainer = pl.Trainer(devices=1, accelerator=cfg.trainer.accelerator)
        if speaker_model.prepare_test(trainer):
            trainer.test(speaker_model)


if __name__ == '__main__':
    main()
```
ecapa_tdnn.yaml
```
name: "ECAPA_TDNN_cn2_300"

model:

  sample_rate: 16000

  train_ds:
    manifest_filepath: /home/ec5017b/media-lab/nemo/NeMo/examples/speaker_tasks/diarization/data/voxconverse/train.json
    sample_rate: ${model.sample_rate}
    labels: null
    batch_size: 16
    shuffle: True
    augmentor:
      # noise:
      #   manifest_path: null
      #   prob: 0.5
      #   min_snr_db: 0
      #   max_snr_db: 15

      speed:
        prob: 0.5
        sr: ${model.sample_rate}
        resample_type: 'kaiser_fast'
        min_speed_rate: 0.95
        max_speed_rate: 1.05

  validation_ds:
    manifest_filepath: /home/ec5017b/media-lab/nemo/NeMo/examples/speaker_tasks/diarization/data/voxconverse/dev.json
    sample_rate: ${model.sample_rate}
    labels: null
    batch_size: 16
    shuffle: False

  preprocessor:
    _target_: nemo.collections.asr.modules.AudioToMelSpectrogramPreprocessor
    normalize: "per_feature"
    window_size: 0.025
    sample_rate: ${model.sample_rate}
    window_stride: 0.01
    window: "hann"
    features: 80
    n_fft: 512
    frame_splicing: 1
    dither: 0.00001
    stft_conv: false

  spec_augment:
    _target_: nemo.collections.asr.modules.SpectrogramAugmentation
    freq_masks: 3
    freq_width: 4
    time_masks: 5
    time_width: 0.03


  encoder:
    _target_: nemo.collections.asr.modules.ECAPAEncoder
    feat_in: ${model.preprocessor.features}
    filters: [1024,1024,1024,1024,3072]
    kernel_sizes: [5,3,3,3,1]
    dilations: [1,1,1,1,1]
    scale: 8


  decoder:
    _target_: nemo.collections.asr.modules.SpeakerDecoder
    feat_in: 3072
    num_classes: 7205
    pool_mode: 'attention' #xvector,tap or attention
    emb_sizes: 192
    angular: True 

  loss:
    scale: 30
    margin: 0.2

  optim:
    name: sgd
    lr: 0.08
    weight_decay: 0.0002

    # scheduler setup
    sched:
      name: CosineAnnealing
      warmup_ratio: 0.1
      min_lr: 0.0001

trainer:
  devices: 1 # number of gpus (trained on four nodes - each node has 8 gpus)
  max_epochs: 10
  max_steps: -1 # computed at runtime if not set
  num_nodes: 1
  accelerator: gpu
  strategy: ddp
  accumulate_grad_batches: 1
  deterministic: False
  enable_checkpointing: False
  logger: False
  log_every_n_steps: 1  # Interval of logging.
  val_check_interval: 1.0  # Set to 0.25 to check 4 times per epoch, or an int for number of iterations

exp_manager:
  exp_dir: null
  name: ${name}
  create_tensorboard_logger: True
  create_checkpoint_callback: True
```
 
**Environment details**

- Ubuntu 16.04.7 LTS
- PyTorch version: 1.12.1
- Python version: 3.8.0
- GPU: Tesla V100 PCIe 16GB
