Decentralized gradient methods: does topology matter?
  Consensus-based distributed optimization methods have recently been advocated
as alternatives to parameter server and ring all-reduce paradigms for large
scale training of machine learning models. In this case, each worker maintains
a local estimate of the optimal parameter vector and iteratively updates it by
averaging the estimates obtained from its neighbors, and applying a correction
on the basis of its local dataset. While theoretical results suggest that
worker communication topology should have strong impact on the number of epochs
needed to converge, previous experiments have shown the opposite conclusion.
This paper sheds lights on this apparent contradiction and show how sparse
topologies can lead to faster convergence even in the absence of communication
delays.
