Hop: Heterogeneity-Aware Decentralized Training
  Recent work has shown that decentralized algorithms can deliver superior
performance over centralized ones in the context of machine learning. The two
approaches, with the main difference residing in their distinct communication
patterns, are both susceptible to performance degradation in heterogeneous
environments. Although vigorous efforts have been devoted to supporting
centralized algorithms against heterogeneity, little has been explored in
decentralized algorithms regarding this problem.
  This paper proposes Hop, the first heterogeneity-aware decentralized training
protocol. Based on a unique characteristic of decentralized training that we
have identified, the iteration gap, we propose a queue-based synchronization
mechanism that can efficiently implement backup workers and bounded staleness
in the decentralized setting. To cope with deterministic slowdown, we propose
skipping iterations so that the effect of slower workers is further mitigated.
We build a prototype implementation of Hop on TensorFlow. The experiment
results on CNN and SVM show significant speedup over standard decentralized
training in heterogeneous settings.
