loss equals to nan when I use rir and noise augmentor
I have the same issue. When I am trying to train with Background Noises, the ctc-loss becomes nan. Also i checked spectrogram, it contains nans. So, I guess, the bug comes from preprocessor or comes from augmentation script.
@amejri Do you find the problem?  I found the bug. In my case, It comes from the data. When the voice energy is high and the noise energy is very small. The multiplication gain becomes very large which raises nan's in mel spectrogram. Did you manually inspect the spectrograms? Or a pass through the data post spectrogram generation to read the min max ? Yes, I manually set NaN checker. @naymaraq could you share with us how you fix the problem please ? I didn't solve it yet.
Hi guys,
I am trying to finetune a quartznet model using rir noise augmentor. During the training step, the loss become nan and at the end when I save the model, I have a file with clearly nothing on it ( Size: 132K instead of the usual 69 Mo size).
Does anyone have an idea about it?
Thanks in advance.
With kind regards.