Communication optimization strategies for distributed deep neural
  network training: A survey
  Recent trends in high-performance computing and deep learning have led to the
proliferation of studies on large-scale deep neural network training. However,
the frequent communication requirements among computation nodes drastically
slows the overall training speeds, which causes bottlenecks in distributed
training, particularly in clusters with limited network bandwidths. To mitigate
the drawbacks of distributed communications, researchers have proposed various
optimization strategies. In this paper, we provide a comprehensive survey of
communication strategies from both an algorithm viewpoint and a computer
network perspective. Algorithm optimizations focus on reducing the
communication volumes used in distributed training, while network optimizations
focus on accelerating the communications between distributed devices. At the
algorithm level, we describe how to reduce the number of communication rounds
and transmitted bits per round. In addition, we elucidate how to overlap
computation and communication. At the network level, we discuss the effects
caused by network infrastructures, including logical communication schemes and
network protocols. Finally, we extrapolate the potential future challenges and
new research directions to accelerate communications for distributed deep
neural network training.
