SGD: The Role of Implicit Regularization, Batch-size and Multiple-epochs
  Multi-epoch, small-batch, Stochastic Gradient Descent (SGD) has been the
method of choice for learning with large over-parameterized models. A popular
theory for explaining why SGD works well in practice is that the algorithm has
an implicit regularization that biases its output towards a good solution.
Perhaps the theoretically most well understood learning setting for SGD is that
of Stochastic Convex Optimization (SCO), where it is well known that SGD learns
at a rate of $O(1/\sqrt{n})$, where $n$ is the number of samples. In this
paper, we consider the problem of SCO and explore the role of implicit
regularization, batch size and multiple epochs for SGD. Our main contributions
are threefold:
  (a) We show that for any regularizer, there is an SCO problem for which
Regularized Empirical Risk Minimzation fails to learn. This automatically rules
out any implicit regularization based explanation for the success of SGD.
  (b) We provide a separation between SGD and learning via Gradient Descent on
empirical loss (GD) in terms of sample complexity. We show that there is an SCO
problem such that GD with any step size and number of iterations can only learn
at a suboptimal rate: at least $\widetilde{\Omega}(1/n^{5/12})$.
  (c) We present a multi-epoch variant of SGD commonly used in practice. We
prove that this algorithm is at least as good as single pass SGD in the worst
case. However, for certain SCO problems, taking multiple passes over the
dataset can significantly outperform single pass SGD.
  We extend our results to the general learning setting by showing a problem
which is learnable for any data distribution, and for this problem, SGD is
strictly better than RERM for any regularization function. We conclude by
discussing the implications of our results for deep learning, and show a
separation between SGD and ERM for two layer diagonal neural networks.
