using skip_nan_grad with gradient accumulation for ASR
This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days. This issue was closed because it has been inactive for 7 days since being marked as stale.
Hi, I notice that while training the ASR fastconformer model on my data, some after a few epochs there can be a batch that cause NaN error. So, I tried setting skip_nan_grad = True, which allow the train_loss curve to look normal (without spiking to NaN), but I notice in the error log that after the message "detected inf or nan values in gradients! Setting gradients to zero.", the all prediction in the following validation rounds are "??" and the validation wer increased to 1. I set accumulate_grad_batches to 4 for the training so I wonder whether this might be the culprit. Does setting skip_nan_grad = True can still work with gradient accumulation. I'd really appreciate any comment you can give on the issue!