How the gradients are updated when using grad_accumulation
Hello, 

The paper of CitriNet said they used 32 V100 GPUs, with a batch size of 32 per GPU.
then, the gradients of each GPU are accumulated and updated together?

When I trained CitriNet-384 model with single gpu, with batch size of 32, then grad_accumulation=32 will be same condition with the paper?

Also, Im training Multilingual ASR model which includes English (LibriSpeech 960h, wpe 256) and Korean (KsponSpeech 1000h, bpe 1641) with CitriNet-384. I made the hyperparameters same with paper (NovoGrad optimizer with learning rate (LR) of 0.05, β1 = 0.8, β2 = 0.25 and weight decay of 0.001). However, validation_loss seem to not decrease until epoch 18. 

is there any tips for this? Im testing with lr=0.01 now. 

Thank you.