Stochastic Alternating Direction Method of Multipliers for
  Byzantine-Robust Distributed Learning
  This paper aims to solve a distributed learning problem under Byzantine
attacks. In the underlying distributed system, a number of unknown but
malicious workers (termed as Byzantine workers) can send arbitrary messages to
the master and bias the learning process, due to data corruptions, computation
errors or malicious attacks. Prior work has considered a total variation (TV)
norm-penalized approximation formulation to handle the Byzantine attacks, where
the TV norm penalty forces the regular workers' local variables to be close,
and meanwhile, tolerates the outliers sent by the Byzantine workers. To solve
the TV norm-penalized approximation formulation, we propose a Byzantine-robust
stochastic alternating direction method of multipliers (ADMM) that fully
utilizes the separable problem structure. Theoretically, we prove that the
proposed method converges to a bounded neighborhood of the optimal solution at
a rate of O(1/k) under mild assumptions, where k is the number of iterations
and the size of neighborhood is determined by the number of Byzantine workers.
Numerical experiments on the MNIST and COVERTYPE datasets demonstrate the
effectiveness of the proposed method to various Byzantine attacks.
