On the Generalization Benefit of Noise in Stochastic Gradient Descent
  It has long been argued that minibatch stochastic gradient descent can
generalize better than large batch gradient descent in deep neural networks.
However recent papers have questioned this claim, arguing that this effect is
simply a consequence of suboptimal hyperparameter tuning or insufficient
compute budgets when the batch size is large. In this paper, we perform
carefully designed experiments and rigorous hyperparameter sweeps on a range of
popular models, which verify that small or moderately large batch sizes can
substantially outperform very large batches on the test set. This occurs even
when both models are trained for the same number of iterations and large
batches achieve smaller training losses. Our results confirm that the noise in
stochastic gradients can enhance generalization. We study how the optimal
learning rate schedule changes as the epoch budget grows, and we provide a
theoretical account of our observations based on the stochastic differential
equation perspective of SGD dynamics.
