Problems with WaveGlow
Hi all,

I am trying to train a TTS model with NeMo implementations of **Tacotron2** and **WaveGlow** architectures. The dataset is my own and includes 15 hours of speech in Slovenian language (sample rate 22050, 16-bit, mono). Label set has 83 symbols, all within 0-255 ASCII. 

The dataset has been tested and should not be problematic – I used it before for training a TTS model with Tacotron2/WaveGlow architectures from the **DeepLeaningExamples** (DLE). The model developed with DLE produced rather clear speech. 

With the NeMo implementation of the Tacotron2 and WaveGlow I am however not able to develop a model that would produce speech of at least a moderate quality. According to the encoder/decoder alignment it seems that Tacotron2 learns ok, while WaveGlow never learns as it should. The NeMo version I use is 1.5.0.

Here is what I have tried so far:

1.	I trained with 16 and 32 precisions
2.	I trained with batch sizes of 10, 16 and 20.
3.	I trained 1000 and 1500 epochs
4.	I trained on different HW settings: I used one node with 4 x A100 GPU and even 8 nodes with 4 x A100 GPU

No matter what I try, results are disappointing. Here are also some figures:

This is what the model produces after 1000 epochs (fig above) and what it should produce (fig below) (bs=20):
<img width="452" alt="WaveGlow predicted" src="https://user-images.githubusercontent.com/16853936/151710024-4e28a9a6-148c-459a-947f-af2d70988f46.png">
<img width="452" alt="WaveGlow target" src="https://user-images.githubusercontent.com/16853936/151710040-1190ab0c-e2ac-430c-a58c-5e3b1a840458.png">

I know that val loss is not very informative for WaveGlow, but nevertheless, here is what is shows:
<img width="452" alt="loss" src="https://user-images.githubusercontent.com/16853936/151710461-f4948328-6da3-4ebd-b9a3-c63e55289772.png">

And last but not least: I trained HifiGan instead of WaveGlow and results seem ok. So I am wondering what am I doing wrong with WaveGlow. Any help/hint will be appreciated.