Stochastic algorithms with geometric step decay converge linearly on
  sharp functions
  Stochastic (sub)gradient methods require step size schedule tuning to perform
well in practice. Classical tuning strategies decay the step size polynomially
and lead to optimal sublinear rates on (strongly) convex problems. An
alternative schedule, popular in nonconvex optimization, is called
\emph{geometric step decay} and proceeds by halving the step size after every
few epochs. In recent work, geometric step decay was shown to improve
exponentially upon classical sublinear rates for the class of \emph{sharp}
convex functions. In this work, we ask whether geometric step decay similarly
improves stochastic algorithms for the class of sharp nonconvex problems. Such
losses feature in modern statistical recovery problems and lead to a new
challenge not present in the convex setting: the region of convergence is
local, so one must bound the probability of escape. Our main result shows that
for a large class of stochastic, sharp, nonsmooth, and nonconvex problems a
geometric step decay schedule endows well-known algorithms with a local linear
rate of convergence to global minimizers. This guarantee applies to the
stochastic projected subgradient, proximal point, and prox-linear algorithms.
As an application of our main result, we analyze two statistical recovery
tasks---phase retrieval and blind deconvolution---and match the best known
guarantees under Gaussian measurement models and establish new guarantees under
heavy-tailed distributions.
