Sparse-Push: Communication- & Energy-Efficient Decentralized Distributed
  Learning over Directed & Time-Varying Graphs with non-IID Datasets
  Current deep learning (DL) systems rely on a centralized computing paradigm
which limits the amount of available training data, increases system latency,
and adds privacy and security constraints. On-device learning, enabled by
decentralized and distributed training of DL models over peer-to-peer
wirelessly connected edge devices, not only alleviate the above limitations but
also enable next-gen applications that need DL models to continuously interact
and learn from their environment. However, this necessitates the development of
novel training algorithms that train DL models over time-varying and directed
peer-to-peer graph structures while minimizing the amount of communication
between the devices and also being resilient to non-IID data distributions. In
this work we propose, Sparse-Push, a communication efficient decentralized
distributed training algorithm that supports training over peer-to-peer,
directed, and time-varying graph topologies. The proposed algorithm enables
466x reduction in communication with only 1% degradation in performance when
training various DL models such as ResNet-20 and VGG11 over the CIFAR-10
dataset. Further, we demonstrate how communication compression can lead to
significant performance degradation in-case of non-IID datasets, and propose
Skew-Compensated Sparse Push algorithm that recovers this performance drop
while maintaining similar levels of communication compression.
