Erratic learning rate regarding FastSpeech2 (NoamAnnealing) and HiFi-GAN
**Describe the bug**

Given the default configuration of fastspeech2 and HifiGAN, their implemented learning rates deviate severely from what's reported in their respective original papers.

**Steps/Code to reproduce bug**

Preprocess LJSpeech as specified by the official NeMo example, 
python fastspeech2.py (or hifigan.py)

**Expected behavior**
![expected](https://user-images.githubusercontent.com/37045193/122874113-95059880-d36d-11eb-8f87-7f35e06fb232.png)


**What is observed**
FastSpeech2
![observed](https://user-images.githubusercontent.com/37045193/122874437-f4fc3f00-d36d-11eb-8f53-5a76ca1e46f1.png)

HiFiGAN
![observed_hifigan](https://user-images.githubusercontent.com/37045193/122874450-f88fc600-d36d-11eb-85ab-a4e94f24462f.png)

We believe the error comes from the missing scaler, which needs to be 1000. given the default model dimension 256.
mult = self._normalize * min(step ** (-0.5), step * (self.warmup_steps ** (-1.5)))
mult = 1000. * self._normalize * min(step ** (-0.5), step * (self.warmup_steps ** (-1.5)))
