Convergence of Distributed Stochastic Variance Reduced Methods without
  Sampling Extra Data
  Stochastic variance reduced methods have gained a lot of interest recently
for empirical risk minimization due to its appealing run time complexity. When
the data size is large and disjointly stored on different machines, it becomes
imperative to distribute the implementation of such variance reduced methods.
In this paper, we consider a general framework that directly distributes
popular stochastic variance reduced methods in the master/slave model, by
assigning outer loops to the parameter server, and inner loops to worker
machines. This framework is natural and friendly to implement, but its
theoretical convergence is not well understood. We obtain a comprehensive
understanding of algorithmic convergence with respect to data homogeneity by
measuring the smoothness of the discrepancy between the local and global loss
functions. We establish the linear convergence of distributed versions of a
family of stochastic variance reduced algorithms, including those using
accelerated and recursive gradient updates, for minimizing strongly convex
losses. Our theory captures how the convergence of distributed algorithms
behaves as the number of machines and the size of local data vary. Furthermore,
we show that when the data are less balanced, regularization can be used to
ensure convergence at a slower rate. We also demonstrate that our analysis can
be further extended to handle nonconvex loss functions.
