Delayed Projection Techniques for Linearly Constrained Problems:
  Convergence Rates, Acceleration, and Applications
  In this work, we study a novel class of projection-based algorithms for
linearly constrained problems (LCPs) which have a lot of applications in
statistics, optimization, and machine learning. Conventional primal
gradient-based methods for LCPs call a projection after each (stochastic)
gradient descent, resulting in that the required number of projections equals
that of gradient descents (or total iterations). Motivated by the recent
progress in distributed optimization, we propose the delayed projection
technique that calls a projection once for a while, lowering the projection
frequency and improving the projection efficiency. Accordingly, we devise a
series of stochastic methods for LCPs using the technique, including a variance
reduced method and an accelerated one. We theoretically show that it is
feasible to improve projection efficiency in both strongly convex and generally
convex cases. Our analysis is simple and unified and can be easily extended to
other methods using delayed projections. When applying our new algorithms to
federated optimization, a newfangled and privacy-preserving subfield in
distributed optimization, we obtain not only a variance reduced federated
algorithm with convergence rates better than previous works, but also the first
accelerated method able to handle data heterogeneity inherent in federated
optimization.
