Variance Reduction on General Adaptive Stochastic Mirror Descent
  In this work, we investigate the idea of variance reduction by studying its
properties with general adaptive mirror descent algorithms in nonsmooth
nonconvex finite-sum optimization problems. We propose a simple yet generalized
framework for variance reduced adaptive mirror descent algorithms named SVRAMD
and provide its convergence analysis in both the nonsmooth nonconvex problem
and the P-L conditioned problem. We prove that variance reduction reduces the
SFO complexity of adaptive mirror descent algorithms and thus accelerates their
convergence. In particular, our general theory implies that variance reduction
can be applied to algorithms using time-varying step sizes and self-adaptive
algorithms such as AdaGrad and RMSProp. Moreover, the convergence rates of
SVRAMD recover the best existing rates of non-adaptive variance reduced mirror
descent algorithms without complicated algorithmic components. Extensive
experiments in deep learning validate our theoretical findings.
