On the Noisy Gradient Descent that Generalizes as SGD
  The gradient noise of SGD is considered to play a central role in the
observed strong generalization abilities of deep learning. While past studies
confirm that the magnitude and the covariance structure of gradient noise are
critical for regularization, it remains unclear whether or not the class of
noise distributions is important. In this work we provide negative results by
showing that noises in classes different from the SGD noise can also
effectively regularize gradient descent. Our finding is based on a novel
observation on the structure of the SGD noise: it is the multiplication of the
gradient matrix and a sampling noise that arises from the mini-batch sampling
procedure. Moreover, the sampling noises unify two kinds of gradient
regularizing noises that belong to the Gaussian class: the one using (scaled)
Fisher as covariance and the one using the gradient covariance of SGD as
covariance. Finally, thanks to the flexibility of choosing noise class, an
algorithm is proposed to perform noisy gradient descent that generalizes well,
the variant of which even benefits large batch SGD training without hurting
generalization.
