Stochastic Training is Not Necessary for Generalization
  It is widely believed that the implicit regularization of SGD is fundamental
to the impressive generalization behavior we observe in neural networks. In
this work, we demonstrate that non-stochastic full-batch training can achieve
comparably strong performance to SGD on CIFAR-10 using modern architectures. To
this end, we show that the implicit regularization of SGD can be completely
replaced with explicit regularization even when comparing against a strong and
well-researched baseline. Our observations indicate that the perceived
difficulty of full-batch training may be the result of its optimization
properties and the disproportionate time and effort spent by the ML community
tuning optimizers and hyperparameters for small-batch training.
