Variance reduction for Riemannian non-convex optimization with batch
  size adaptation
  Variance reduction techniques are popular in accelerating gradient descent
and stochastic gradient descent for optimization problems defined on both
Euclidean space and Riemannian manifold. In this paper, we further improve on
existing variance reduction methods for non-convex Riemannian optimization,
including R-SVRG and R-SRG/R-SPIDER with batch size adaptation. We show that
this strategy can achieve lower total complexities for optimizing both general
non-convex and gradient dominated functions under both finite-sum and online
settings. As a result, we also provide simpler convergence analysis for R-SVRG
and improve complexity bounds for R-SRG under finite-sum setting. Specifically,
we prove that R-SRG achieves the same near-optimal complexity as R-SPIDER
without requiring a small step size. Empirical experiments on a variety of
tasks demonstrate effectiveness of proposed adaptive batch size scheme.
