ASR finetuning/training 7x slower on v1.8.2 
**Describe the bug**

I recently updated from NeMo 1.7.2 to 1.8.2
When running a training session, the speed is now `4.04s/it` whereas it used to be `2.97it/s`.
Note it's taking ~4 **seconds per iteration** whereas it used to be ~3 **iterations per second**!

Here's my trainer config since it's the only things that's different from the NeMo examples config for CitriNet-1024 with SPE tokenizer:

```yaml
trainer:
  devices: 8
  accelerator: gpu
  max_epochs: 100
  max_steps: -1
  num_nodes: 1
  strategy: ddp
  accumulate_grad_batches: 4
  enable_checkpointing: false  # Provided by exp_manager
  logger: false                # Provided by exp_manager
  log_every_n_steps: 100
  val_check_interval: 1.0
  precision: 16
```

**Steps/Code to reproduce bug**

Run any training example script.

**Expected behavior**

Speed should not be significantly affected.

**Environment overview (please complete the following information)**

 - Environment location: Docker (tried `nvcr.io/nvidia/pytorch:22.03-py3` and `nvcr.io/nvidia/pytorch:22.02-py3`)
 - Method of NeMo install: `pip install nemo_toolkit[all]==1.8.2`
 - If method of install is [Docker], provide `docker pull` & `docker run` commands used
 ```
docker run --rm -it --gpus all --ipc=host --env-file .env train
```

**Additional context**

Running this on an AWS p3dn.24xlarge instance (8xV100)
