Cannot select speaker while inferencing on models trained on latest FastPitch_Finetuning.ipynb version
Hi all.

I'm trying to generate audio with Inference_DurationPitchControl.ipynb. I've trained the FastPitch model with both LJSpeech and a custom dataset on the latest release (r1.8.2) with FastPitch_Finetuning (https://github.com/NVIDIA/NeMo/blob/main/tutorials/tts/FastPitch_Finetuning.ipynb). Also, I've models previously trained on a previous version of this script (r1.6.2).

The problem is using the new models, the voice is always the voice of the LJSpeech woman, regardless the speaker selection I choose. Nevertheless, there is still a subtle difference in the generated audio if I choose one speaker or another.

If I load one of the 'old' models, I can select the speaker without a problem.

Here I paste the code that I've used for training the new models on the latest FastPitch_Finetuning script (I've changed batch size to 16):

```
!(python fastpitch_finetune.py --config-name=fastpitch_align_v1.05.yaml \
train_dataset=./train.json \
validation_datasets=./val.json \
sup_data_path=./fastpitch_sup_data \
phoneme_dict_path=tts_dataset_files/cmudict-0.7b_nv22.01 \
heteronyms_path=tts_dataset_files/heteronyms-030921 \
whitelist_path=tts_dataset_files/lj_speech.tsv \
exp_manager.exp_dir=$OUTPUT_CHEKPOINTS \
+init_from_nemo_model=./tts_en_fastpitch_align.nemo \
+trainer.max_steps=$max_steps ~trainer.max_epochs \
trainer.check_val_every_n_epoch=25 \
model.train_ds.dataloader_params.batch_size=16 model.validation_ds.dataloader_params.batch_size=16 \
model.n_speakers=2 model.pitch_mean=121.9 model.pitch_std=20 \
model.pitch_fmin=65 model.pitch_fmax=512 model.optim.lr=2e-4 \
~model.optim.sched model.optim.name=adam trainer.devices=1 trainer.strategy=null \
+model.text_tokenizer.add_blank_at=true \
)
```

Here is the code of the old models training:

```
!(python fastpitch_finetune.py --config-name=fastpitch_align.yaml \
train_dataset=./train.json \
validation_datasets=./val.json \
+init_from_nemo_model=./tts_en_fastpitch_align.nemo \
+trainer.max_steps=$max_steps ~trainer.max_epochs trainer.check_val_every_n_epoch=25 \
prior_folder=./PriorsCustom model.train_ds.dataloader_params.batch_size=24 \
model.validation_ds.dataloader_params.batch_size=24 \
exp_manager.exp_dir=$OUTPUT_CHEKPOINTS \
model.n_speakers=2 model.pitch_avg=90 model.pitch_std=20 \
model.pitch_fmin=65 model.pitch_fmax=512 model.optim.lr=2e-4 \
~model.optim.sched model.optim.name=adam trainer.devices=1 trainer.strategy=null \
)
```

Thank you very much.