Improving Semi-supervised Federated Learning by Reducing the Gradient
  Diversity of Models
  Federated learning (FL) is a promising way to use the computing power of
mobile devices while maintaining the privacy of users. Current work in FL,
however, makes the unrealistic assumption that the users have ground-truth
labels on their devices, while also assuming that the server has neither data
nor labels. In this work, we consider the more realistic scenario where the
users have only unlabeled data, while the server has some labeled data, and
where the amount of labeled data is smaller than the amount of unlabeled data.
We call this learning problem semi-supervised federated learning (SSFL). For
SSFL, we demonstrate that a critical issue that affects the test accuracy is
the large gradient diversity of the models from different users. Based on this,
we investigate several design choices. First, we find that the so-called
consistency regularization loss (CRL), which is widely used in semi-supervised
learning, performs reasonably well but has large gradient diversity. Second, we
find that Batch Normalization (BN) increases gradient diversity. Replacing BN
with the recently-proposed Group Normalization (GN) can reduce gradient
diversity and improve test accuracy. Third, we show that CRL combined with GN
still has a large gradient diversity when the number of users is large. Based
on these results, we propose a novel grouping-based model averaging method to
replace the FedAvg averaging method. Overall, our grouping-based averaging,
combined with GN and CRL, achieves better test accuracy than not just a
contemporary paper on SSFL in the same settings (>10\%), but also four
supervised FL algorithms.
