TitaNet validation step too slow
Hello, I'm training TitaNet from scratch using voxceleb 1 and 2 with another private dataset, while the training is fast 6it/s on a dual 3090 setup the valitadion step takes 2200s/it. It happens using all the labels and also with a limited set of labels (1024), or even with an4 dataset.

**Steps/Code to reproduce bug**

Followed this tutorial: https://github.com/NVIDIA/NeMo/blob/stable/tutorials/speaker_tasks/Speaker_Identification_Verification.ipynb and changed the data path.


 - Environment location: Local machine AMD® Ryzen 9 3900x  80GB 2x 3090
 - Method of NeMo install: from source, using this commit https://github.com/NVIDIA/NeMo/commit/c1abc047d1d5962019f7789b20d8092c7e55b690 tested with python 3.7 and 3.9 using conda enviroments

**Environment details**

- OS version: Ubuntu 22.04.2 LTS
- PyTorch version 11.7 11.8 slow with both versions
- Python version 3.7 3.9 slow with both versions

**Additional context**

It is slow with 1 GPU and 2GPUs, while in validation step the usage jumps from 10% to 20%.

Any other network like QuartzNetor Conformer works just fine.

¿Any ideas?