NaN error while training Conformer-Transducer with automatic mixed precesion
Hi, thanks for the great toolkit!

I'm training the conformer-transducer-medium on librispeech-960hr from scratch, following the config in `examples/asr/conf/conformer/conformer_transducer_bpe.yaml`

I use a sentencepiece unigram tokenizer with a vocab size of 1024 generated using the script in NeMo. The batch size is set at 32 on each GPU for a global batch size of 256 (8 A100 GPUs are used). I use automatic mixed precision with amp_level O1. I tried a few different settings for the learning rate and weight decay to adjust for the batch size of 256, which is smaller than the default 2k batch size, and here are the results:

1. Learning rate = 2, weight decay = 1e-6. The training went well and got 2.8/7.6 WER on the dev-clean/dev-other sets at epoch 144. But suddenly got NaN error at epoch 145. I tried resuming the training from checkpoint but got NaN error again at the end of epoch 145.

2. Learning rate = 5, weight decay = 1e-3 (the default setting). The training ran for 25 epoch and got NaN error at epoch 26.

3. Learning rate = 2, weight decay = 1e-4. The training ran for 58 epochs and got NaN error at epoch 59.

Given that the NaN error occurred so late in the training, I doubt that the error was caused by some training example. So, any suggestions to find the root causes would be really appreciated!

Thank you!