Adaptive Worker Grouping For Communication-Efficient and
  Straggler-Tolerant Distributed SGD
  Wall-clock convergence time and communication load are key performance
metrics for the distributed implementation of stochastic gradient descent (SGD)
in parameter server settings. Communication-adaptive distributed Adam (CADA)
has been recently proposed as a way to reduce communication load via the
adaptive selection of workers. CADA is subject to performance degradation in
terms of wall-clock convergence time in the presence of stragglers. This paper
proposes a novel scheme named grouping-based CADA (G-CADA) that retains the
advantages of CADA in reducing the communication load, while increasing the
robustness to stragglers at the cost of additional storage at the workers.
G-CADA partitions the workers into groups of workers that are assigned the same
data shards. Groups are scheduled adaptively at each iteration, and the server
only waits for the fastest worker in each selected group. We provide analysis
and experimental results to elaborate the significant gains on the wall-clock
time, as well as communication load and computation load, of G-CADA over other
benchmark schemes.
