BROADCAST: Reducing Both Stochastic and Compression Noise to Robustify
  Communication-Efficient Federated Learning
  Communication between workers and the master node to collect local stochastic
gradients is a key bottleneck in a large-scale federated learning system.
Various recent works have proposed to compress the local stochastic gradients
to mitigate the communication overhead. However, robustness to malicious
attacks is rarely considered in such a setting. In this work, we investigate
the problem of Byzantine-robust compressed federated learning, where the
attacks from Byzantine workers can be arbitrarily malicious. We theoretically
point out that different to the attacks-free compressed stochastic gradient
descent (SGD), its vanilla combination with geometric median-based robust
aggregation seriously suffers from the compression noise in the presence of
Byzantine attacks. In light of this observation, we propose to reduce the
compression noise with gradient difference compression so as to improve the
Byzantine-robustness. We also observe the impact of the intrinsic stochastic
noise caused by selecting random samples, and adopt the stochastic average
gradient algorithm (SAGA) to gradually eliminate the inner variations of
regular workers. We theoretically prove that the proposed algorithm reaches a
neighborhood of the optimal solution at a linear convergence rate, and the
asymptotic learning error is in the same order as that of the state-of-the-art
uncompressed method. Finally, numerical experiments demonstrate the
effectiveness of the proposed method.
