Gradient Masked Averaging for Federated Learning
  Federated learning is an emerging paradigm that permits a large number of
clients with heterogeneous data to coordinate learning of a unified global
model without the need to share data amongst each other. Standard federated
learning algorithms involve averaging of model parameters or gradient updates
to approximate the global model at the server. However, in heterogeneous
settings averaging can result in information loss and lead to poor
generalization due to the bias induced by dominant clients. We hypothesize that
to generalize better across non-i.i.d datasets as in FL settings, the
algorithms should focus on learning the invariant mechanism that is constant
while ignoring spurious mechanisms that differ across clients. Inspired from
recent work in the Out-of-Distribution literature, we propose a gradient masked
averaging approach for federated learning as an alternative to the standard
averaging of client updates. This client update aggregation technique can be
adapted as a drop-in replacement in most existing federated algorithms. We
perform extensive experiments with gradient masked approach on multiple FL
algorithms with in-distribution, real-world, and out-of-distribution (as the
worst case scenario) test dataset and show that it provides consistent
improvements, particularly in the case of heterogeneous clients.
