Sampling and Update Frequencies in Proximal Variance Reduced Stochastic
  Gradient Methods
  Variance reduced stochastic gradient methods have gained popularity in recent
times. Several variants exist with different strategies for the storing and
sampling of gradients. In this work we focus on the analysis of the interaction
of these two aspects. We present and analyze a general proximal variance
reduced gradient method under strong convexity assumptions. Special cases of
the algorithm include SAGA, L-SVRG and their proximal variants. Our analysis
sheds light on epoch-length selection and the need to balance the convergence
of the iterates and how often gradients are stored. The analysis improves on
other convergence rates found in literature and produces a new and faster
converging sampling strategy for SAGA. Problem instances for which the
predicted rates are the same as the practical rates are presented together with
problems based on real world data.
