Byzantine-resilient Decentralized Stochastic Gradient Descent
  Decentralized learning has gained great popularity to improve learning
efficiency and preserve data privacy. Each computing node makes equal
contribution to collaboratively learn a Deep Learning model. The elimination of
centralized Parameter Servers (PS) can effectively address many issues such as
privacy, performance bottleneck and single-point-failure. However, how to
achieve Byzantine Fault Tolerance in decentralized learning systems is rarely
explored, although this problem has been extensively studied in centralized
systems.
  In this paper, we present an in-depth study towards the Byzantine resilience
of decentralized learning systems with two contributions. First, from the
adversarial perspective, we theoretically illustrate that Byzantine attacks are
more dangerous and feasible in decentralized learning systems: even one
malicious participant can arbitrarily alter the models of other participants by
sending carefully crafted updates to its neighbors. Second, from the defense
perspective, we propose UBAR, a novel algorithm to enhance decentralized
learning with Byzantine Fault Tolerance. Specifically, UBAR provides a Uniform
Byzantine-resilient Aggregation Rule for benign nodes to select the useful
parameter updates and filter out the malicious ones in each training iteration.
It guarantees that each benign node in a decentralized system can train a
correct model under very strong Byzantine attacks with an arbitrary number of
faulty nodes. We conduct extensive experiments on standard image classification
tasks and the results indicate that UBAR can effectively defeat both simple and
sophisticated Byzantine attacks with higher performance efficiency than
existing solutions.
