Adaptivity of Stochastic Gradient Methods for Nonconvex Optimization
  Adaptivity is an important yet under-studied property in modern optimization
theory. The gap between the state-of-the-art theory and the current practice is
striking in that algorithms with desirable theoretical guarantees typically
involve drastically different settings of hyperparameters, such as step-size
schemes and batch sizes, in different regimes. Despite the appealing
theoretical results, such divisive strategies provide little, if any, insight
to practitioners to select algorithms that work broadly without tweaking the
hyperparameters. In this work, blending the "geometrization" technique
introduced by Lei & Jordan 2016 and the \texttt{SARAH} algorithm of Nguyen et
al., 2017, we propose the Geometrized \texttt{SARAH} algorithm for non-convex
finite-sum and stochastic optimization. Our algorithm is proved to achieve
adaptivity to both the magnitude of the target accuracy and the
Polyak-\L{}ojasiewicz (PL) constant if present. In addition, it achieves the
best-available convergence rate for non-PL objectives simultaneously while
outperforming existing algorithms for PL objectives.
