An Empirical Study of Large-Batch Stochastic Gradient Descent with
  Structured Covariance Noise
  The choice of batch-size in a stochastic optimization algorithm plays a
substantial role for both optimization and generalization. Increasing the
batch-size used typically improves optimization but degrades generalization. To
address the problem of improving generalization while maintaining optimal
convergence in large-batch training, we propose to add covariance noise to the
gradients. We demonstrate that the learning performance of our method is more
accurately captured by the structure of the covariance matrix of the noise
rather than by the variance of gradients. Moreover, over the convex-quadratic,
we prove in theory that it can be characterized by the Frobenius norm of the
noise matrix. Our empirical studies with standard deep learning
model-architectures and datasets shows that our method not only improves
generalization performance in large-batch training, but furthermore, does so in
a way where the optimization performance remains desirable and the training
duration is not elongated.
