Exponential Graph is Provably Efficient for Decentralized Deep Training
  Decentralized SGD is an emerging training method for deep learning known for
its much less (thus faster) communication per iteration, which relaxes the
averaging step in parallel SGD to inexact averaging. The less exact the
averaging is, however, the more the total iterations the training needs to
take. Therefore, the key to making decentralized SGD efficient is to realize
nearly-exact averaging using little communication. This requires a skillful
choice of communication topology, which is an under-studied topic in
decentralized optimization.
  In this paper, we study so-called exponential graphs where every node is
connected to $O(\log(n))$ neighbors and $n$ is the total number of nodes. This
work proves such graphs can lead to both fast communication and effective
averaging simultaneously. We also discover that a sequence of $\log(n)$
one-peer exponential graphs, in which each node communicates to one single
neighbor per iteration, can together achieve exact averaging. This favorable
property enables one-peer exponential graph to average as effective as its
static counterpart but communicates more efficiently. We apply these
exponential graphs in decentralized (momentum) SGD to obtain the
state-of-the-art balance between per-iteration communication and iteration
complexity among all commonly-used topologies. Experimental results on a
variety of tasks and models demonstrate that decentralized (momentum) SGD over
exponential graphs promises both fast and high-quality training. Our code is
implemented through BlueFog and available at
https://github.com/Bluefog-Lib/NeurIPS2021-Exponential-Graph.
