Gradient Noise Convolution (GNC): Smoothing Loss Function for
  Distributed Large-Batch SGD
  Large-batch stochastic gradient descent (SGD) is widely used for training in
distributed deep learning because of its training-time efficiency, however,
extremely large-batch SGD leads to poor generalization and easily converges to
sharp minima, which prevents naive large-scale data-parallel SGD (DP-SGD) from
converging to good minima. To overcome this difficulty, we propose gradient
noise convolution (GNC), which effectively smooths sharper minima of the loss
function. For DP-SGD, GNC utilizes so-called gradient noise, which is induced
by stochastic gradient variation and convolved to the loss function as a
smoothing effect. GNC computation can be performed by simply computing the
stochastic gradient on each parallel worker and merging them, and is therefore
extremely easy to implement. Due to convolving with the gradient noise, which
tends to spread along a sharper direction of the loss function, GNC can
effectively smooth sharp minima and achieve better generalization, whereas
isotropic random noise cannot. We empirically show this effect by comparing GNC
with isotropic random noise, and show that it achieves state-of-the-art
generalization performance for large-scale deep neural network optimization.
