Minimization of Stochastic First-order Oracle Complexity of Adaptive
  Methods for Nonconvex Optimization
  Numerical evaluations have definitively shown that, for deep learning
optimizers such as stochastic gradient descent, momentum, and adaptive methods,
the number of steps needed to train a deep neural network halves for each
doubling of the batch size and that there is a region of diminishing returns
beyond the critical batch size. In this paper, we determine the actual critical
batch size by using the global minimizer of the stochastic first-order oracle
(SFO) complexity of the optimizer. To prove the existence of the actual
critical batch size, we set the lower and upper bounds of the SFO complexity
and prove that there exist critical batch sizes in the sense of minimizing the
lower and upper bounds. This proof implies that, if the SFO complexity fits the
lower and upper bounds, then the existence of these critical batch sizes
demonstrates the existence of the actual critical batch size. We also discuss
the conditions needed for the SFO complexity to fit the lower and upper bounds
and provide numerical results that support our theoretical results.
