Escape saddle points faster on manifolds via perturbed Riemannian
  stochastic recursive gradient
  In this paper, we propose a variant of Riemannian stochastic recursive
gradient method that can achieve second-order convergence guarantee and escape
saddle points using simple perturbation. The idea is to perturb the iterates
when gradient is small and carry out stochastic recursive gradient updates over
tangent space. This avoids the complication of exploiting Riemannian geometry.
We show that under finite-sum setting, our algorithm requires
$\widetilde{\mathcal{O}}\big( \frac{ \sqrt{n}}{\epsilon^2} + \frac{\sqrt{n}
}{\delta^4} + \frac{n}{\delta^3}\big)$ stochastic gradient queries to find a
$(\epsilon, \delta)$-second-order critical point. This strictly improves the
complexity of perturbed Riemannian gradient descent and is superior to
perturbed Riemannian accelerated gradient descent under large-sample settings.
We also provide a complexity of $\widetilde{\mathcal{O}} \big(
\frac{1}{\epsilon^3} + \frac{1}{\delta^3 \epsilon^2} + \frac{1}{\delta^4
\epsilon} \big)$ for online optimization, which is novel on Riemannian manifold
in terms of second-order convergence using only first-order information.
