Stopping Criteria for, and Strong Convergence of, Stochastic Gradient
  Descent on Bottou-Curtis-Nocedal Functions
  Stopping criteria for Stochastic Gradient Descent (SGD) methods play
important roles from enabling adaptive step size schemes to providing rigor for
downstream analyses such as asymptotic inference. Unfortunately, current
stopping criteria for SGD methods are often heuristics that rely on asymptotic
normality results or convergence to stationary distributions, which may fail to
exist for nonconvex functions and, thereby, limit the applicability of such
stopping criteria. To address this issue, in this work, we rigorously develop
two stopping criteria for SGD that can be applied to a broad class of nonconvex
functions, which we term Bottou-Curtis-Nocedal functions. Moreover, as a
prerequisite for developing these stopping criteria, we prove that the gradient
function evaluated at SGD's iterates converges strongly to zero for
Bottou-Curtis-Nocedal functions, which addresses an open question in the SGD
literature. As a result of our work, our rigorously developed stopping criteria
can be used to develop new adaptive step size schemes or bolster other
downstream analyses for nonconvex functions.
