Inexact SARAH Algorithm for Stochastic Optimization
  We develop and analyze a variant of the SARAH algorithm, which does not
require computation of the exact gradient. Thus this new method can be applied
to general expectation minimization problems rather than only finite sum
problems. While the original SARAH algorithm, as well as its predecessor, SVRG,
require an exact gradient computation on each outer iteration, the inexact
variant of SARAH (iSARAH), which we develop here, requires only stochastic
gradient computed on a mini-batch of sufficient size. The proposed method
combines variance reduction via sample size selection and iterative stochastic
gradient updates. We analyze the convergence rate of the algorithms for
strongly convex and non-strongly convex cases, under smooth assumption with
appropriate mini-batch size selected for each case. We show that with an
additional, reasonable, assumption iSARAH achieves the best known complexity
among stochastic methods in the case of non-strongly convex stochastic
functions.
