pretrain chinese TTS generate a bed audio
**Describe the bug**

I follow the sample code in [NGC](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/tts_zh_fastpitch_hifigan_sfspeech) to generate chinese TTS audio
but get a bed result

I also try other models (e.g. english, german and spanish), but it works fine

**Steps/Code to reproduce bug**

```
# Load spectrogram generator
from nemo.collections.tts.models import FastPitchModel
spec_generator = FastPitchModel.from_pretrained(model_name="tts_zh_fastpitch_sfspeech")

# Load Vocoder
from nemo.collections.tts.models import HifiGanModel
model = HifiGanModel.from_pretrained(model_name="tts_zh_hifigan_sfspeech")

# Generate audio
import soundfile as sf
import torch
with torch.no_grad():
    parsed = spec_generator.parse("这些新一代的CPU不只效能惊人。")
    spectrogram = spec_generator.generate_spectrogram(tokens=parsed)
    audio = model.convert_spectrogram_to_audio(spec=spectrogram)
    if isinstance(audio, torch.Tensor):
        audio = audio.to('cpu').numpy()

# Save the audio to disk in a file called speech.wav
sf.write("speech.wav", audio.T, 22050, format='WAV')
```

**Environment details**

OS: ubuntu 20.04
NeMo : brench 1.20.0
python : 3.8
pytorch : 2.0.1

