Trade-offs of Local SGD at Scale: An Empirical Study
  As datasets and models become increasingly large, distributed training has
become a necessary component to allow deep neural networks to train in
reasonable amounts of time. However, distributed training can have substantial
communication overhead that hinders its scalability. One strategy for reducing
this overhead is to perform multiple unsynchronized SGD steps independently on
each worker between synchronization steps, a technique known as local SGD. We
conduct a comprehensive empirical study of local SGD and related methods on a
large-scale image classification task. We find that performing local SGD comes
at a price: lower communication costs (and thereby faster training) are
accompanied by lower accuracy. This finding is in contrast from the
smaller-scale experiments in prior work, suggesting that local SGD encounters
challenges at scale. We further show that incorporating the slow momentum
framework of Wang et al. (2020) consistently improves accuracy without
requiring additional communication, hinting at future directions for
potentially escaping this trade-off.
