The training time for SpeakerNet is too long

I found that setting `num_sanity_val_steps: 0` and` limit_val_batches: 0.0` allows the training stage to run without the `Sanity Checking DataLoader"`and `Validation DataLoader`. However, I still want to know if there is a better way to handle this, as I believe a time requirement of 66452 hours is not a normal value. Hi, its not a good idea to have same manifest for training and validation. The requirement is to have subset of speaker classes from training manifest. So prepare validation manifest accordingly. 

Training time depends on multiple factors: dataset size, batchsize, GPU, number of GPUs, utterance length of each sample etc.  Hi, @nithinraok 

Thank you for letting me know about the validation requirement. 

Although the training time depends on many factors, my training stage takes about 1 day, while the validation stage takes 66,452 hours. 

I believe there is a problem here, but I don't know how to deal with it. 

Besides my current practice of skipping the validation stage, do you have any other suggestions? My suggestion is to have 10% of training hrs of speakers in validation set.  ok, thanks
Hi,
 I am training a SpeakerNet, and my process is as follows:
Step 1: In the `SpeakerNet_verification_3x2x256.yaml` file,
   - Set `train_ds manifest_filepath` to `path/to/my/trainset`.
   - Set `validation_ds manifest_filepath` to `path/to/my/trainset`.
   It's worth noting that `train_ds` and `validation_ds` have the same `manifest_filepath` because I found that not doing so results in errors.

Step 2: Execute `speaker_reco.py`.

I noticed that the program spends a considerable amount of time in the `Sanity Checking DataLoader` and also in the `Validation DataLoader` stage. The program estimates that it will take 66452 hours to complete one epoch. I believe there is some issue, and I'm wondering if anyone can provide me with advice on this matter.

Tool version
python 3.10.13
NeMo 1.20.0
pytorch-lightning 1.9.4
torch 2.1.0
