Cooperative SGD: A unified Framework for the Design and Analysis of
  Communication-Efficient SGD Algorithms
  Communication-efficient SGD algorithms, which allow nodes to perform local
updates and periodically synchronize local models, are highly effective in
improving the speed and scalability of distributed SGD. However, a rigorous
convergence analysis and comparative study of different communication-reduction
strategies remains a largely open problem. This paper presents a unified
framework called Cooperative SGD that subsumes existing communication-efficient
SGD algorithms such as periodic-averaging, elastic-averaging and decentralized
SGD. By analyzing Cooperative SGD, we provide novel convergence guarantees for
existing algorithms. Moreover, this framework enables us to design new
communication-efficient SGD algorithms that strike the best balance between
reducing communication overhead and achieving fast error convergence with low
error floor.
