Gradient overflow
**Describe the bug**

I'm receiving these warnings and errors when I'm training Quartznet using spanish dataset at A100:

***Warnings:***
- Could not import torchaudio. Some features might not work.  warnings.warn('Could not import torchaudio. Some features might not work.')libibverbs: 
- Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0

***Error***:
- Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 1.0

**Environment overview (please complete the following information)**

 - Docker image: https://ngc.nvidia.com/catalog/containers/nvidia:nemo
 - Using Enroot to start the instance
 - Following the steps of: Spanish__Cross_Language.ipynb
 - NeMo Parameters:
     - nproc_per_node=8
     - batch_size=32
     - amp_opt_level = 'O1'
 - Dataset: Mozilla Common Voice - Spanish Version 6.1
 - Running at a DGX A100 80G


