Escaping Saddle Points with Stochastically Controlled Stochastic
  Gradient Methods
  Stochastically controlled stochastic gradient (SCSG) methods have been proved
to converge efficiently to first-order stationary points which, however, can be
saddle points in nonconvex optimization. It has been observed that a stochastic
gradient descent (SGD) step introduces anistropic noise around saddle points
for deep learning and non-convex half space learning problems, which indicates
that SGD satisfies the correlated negative curvature (CNC) condition for these
problems. Therefore, we propose to use a separate SGD step to help the SCSG
method escape from strict saddle points, resulting in the CNC-SCSG method. The
SGD step plays a role similar to noise injection but is more stable. We prove
that the resultant algorithm converges to a second-order stationary point with
a convergence rate of $\tilde{O}( \epsilon^{-2} log( 1/\epsilon))$ where
$\epsilon$ is the pre-specified error tolerance. This convergence rate is
independent of the problem dimension, and is faster than that of CNC-SGD. A
more general framework is further designed to incorporate the proposed CNC-SCSG
into any first-order method for the method to escape saddle points. Simulation
studies illustrate that the proposed algorithm can escape saddle points in much
fewer epochs than the gradient descent methods perturbed by either noise
injection or a SGD step.
