Towards Theoretical Understanding of Large Batch Training in Stochastic
  Gradient Descent
  Stochastic gradient descent (SGD) is almost ubiquitously used for training
non-convex optimization tasks. Recently, a hypothesis proposed by Keskar et al.
[2017] that large batch methods tend to converge to sharp minimizers has
received increasing attention. We theoretically justify this hypothesis by
providing new properties of SGD in both finite-time and asymptotic regimes. In
particular, we give an explicit escaping time of SGD from a local minimum in
the finite-time regime and prove that SGD tends to converge to flatter minima
in the asymptotic regime (although may take exponential time to converge)
regardless of the batch size. We also find that SGD with a larger ratio of
learning rate to batch size tends to converge to a flat minimum faster,
however, its generalization performance could be worse than the SGD with a
smaller ratio of learning rate to batch size. We include numerical experiments
to corroborate these theoretical findings.
