Fastconformer loss reduction method for gradient accumulation
In some tutorials I've used grad accumulation 8 without modifying any LR etc and the model is able to trim and converge fine. You need a decently large global batch size, otherwise model will not begin to train unless you start with a pretrained encoder  yes, I used batch size of 256 with 8 accumulation steps for a total of 2,048. I got good result for the fastconformer CTC model but not yet for the fastconformer transducer. Perhaps there's some other issues. Thanks! When you get batch size of 256, not necessary to have such large grad accumulation but it does not hurt, just maybe slow down convergence a little bit at the beginning. 
Have you tried hybrid models? 
What do you mean by not good results? You mean wer is worse than CTC? I haven't tried the hybrid models. I couldn't get the fastconformer transducer's wer close to the Conformer-L Transducer's wer yet. But I'm working with Japanese, which have longer average acoustic length for each character (especially kanji), leading to higher loss when using mean_volume reduction than in other languages. I guess that I might need to tune the learning rate. This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.
Hi, I see that the fastconformer model is trained with mean_volume loss reduction (both for ctc and transducer loss). However, when using gradient accumulation (such as 8), the mean_volume loss is aggregated for each single batch rather for all of the 8 batches. I haven't successfully trained fastconformer transducer so I suspect that this is a possible issue. Do you have any suggestion on changing the reduction method or changing the learning rate when using gradient accumulation? 