Error trying to finetune HifiGan: stack expects each tensor to be equal size
**Describe the bug**

I am trying to finetune the HifiGan vocoder using a set of my own audio data file. It is based on the FastPitch notebook in the tutorial.
https://github.com/NVIDIA/NeMo/blob/main/tutorials/tts/FastPitch_Finetuning.ipynb
Actually the first spec model of FastPitch is trained successfully. But the training of HifiGan failed. Feel a little strange because the same procedure was used on another set of data a few days ago and got a success.

**Steps/Code to reproduce bug**

Command I use to run the finetuning:
`hifigan_finetune.py --config-name=hifigan.yaml model.train_ds.dataloader_params.batch_size=32 model.max_steps=1000 model.optim.lr=0.00001 ~model.optim.sched train_dataset=./hifigan_train_ft.json validation_datasets=./hifigan_train_ft.json exp_manager.exp_dir=hifigan_ft +init_from_pretrained_model=tts_hifigan trainer.check_val_every_n_epoch=10 model/train_ds=train_ds_finetune model/validation_ds=val_ds_finetune`

Here are the first ten lines of hifigan_train_ft.json:
{"audio_filepath": "./wavs/SPEAKER_target/0_0.wav", "text": " Hi everyone, this is Kevin Chiang from Experian Datalab.", "mel_filepath": "data_mels/mel_0_0.npy"}
{"audio_filepath": "./wavs/SPEAKER_target/0_1.wav", "text": " This is what happens when machine meets world.", "mel_filepath": "data_mels/mel_0_1.npy"}
{"audio_filepath": "./wavs/SPEAKER_target/10_0.wav", "text": " It is a very wide scope,", "mel_filepath": "data_mels/mel_10_0.npy"}
{"audio_filepath": "./wavs/SPEAKER_target/10_1.wav", "text": " obvious because everybody is much more familiar", "mel_filepath": "data_mels/mel_10_1.npy"}
{"audio_filepath": "./wavs/SPEAKER_target/10_2.wav", "text": " with credit bureau, but it really covered a lot of areas.", "mel_filepath": "data_mels/mel_10_2.npy"}
{"audio_filepath": "./wavs/SPEAKER_target/10_3.wav", "text": " So you asked about Datalab.", "mel_filepath": "data_mels/mel_10_3.npy"}
{"audio_filepath": "./wavs/SPEAKER_target/10_4.wav", "text": " Datalab is actually the R&D arm of Experian.", "mel_filepath": "data_mels/mel_10_4.npy"}
{"audio_filepath": "./wavs/SPEAKER_target/10_5.wav", "text": " We are founded about 10, 11 years ago.", "mel_filepath": "data_mels/mel_10_5.npy"}
{"audio_filepath": "./wavs/SPEAKER_target/10_6.wav", "text": " And then starting from that point,", "mel_filepath": "data_mels/mel_10_6.npy"}
{"audio_filepath": "./wavs/SPEAKER_target/10_7.wav", "text": " we have grown into four labs globally.", "mel_filepath": "data_mels/mel_10_7.npy"}

Here is how the mel file path look like:
[
![Screenshot from 2023-01-13 01-00-33](https://user-images.githubusercontent.com/7971146/212257571-42ed282a-382d-4ae8-8790-1e76d80c10b9.png)
](url)

**Expected behavior**
Error in training step:
[
![Screenshot from 2023-01-13 00-52-05](https://user-images.githubusercontent.com/7971146/212256373-ff8df136-b04c-4bbb-91ec-8b2cffb54674.png)
](url)

**Environment overview (please complete the following information)**
python 3.8.15
nemo installed with 'pip install --upgrade nemo_toolkit[all]'


**Additional context**

GPU model: Nvidia P5200


Thank you!
