FedCorr: Multi-Stage Federated Learning for Label Noise Correction
  Federated learning (FL) is a privacy-preserving distributed learning paradigm
that enables clients to jointly train a global model. In real-world FL
implementations, client data could have label noise, and different clients
could have vastly different label noise levels. Although there exist methods in
centralized learning for tackling label noise, such methods do not perform well
on heterogeneous label noise in FL settings, due to the typically smaller sizes
of client datasets and data privacy requirements in FL. In this paper, we
propose $\texttt{FedCorr}$, a general multi-stage framework to tackle
heterogeneous label noise in FL, without making any assumptions on the noise
models of local clients, while still maintaining client data privacy. In
particular, (1) $\texttt{FedCorr}$ dynamically identifies noisy clients by
exploiting the dimensionalities of the model prediction subspaces independently
measured on all clients, and then identifies incorrect labels on noisy clients
based on per-sample losses. To deal with data heterogeneity and to increase
training stability, we propose an adaptive local proximal regularization term
that is based on estimated local noise levels. (2) We further finetune the
global model on identified clean clients and correct the noisy labels for the
remaining noisy clients after finetuning. (3) Finally, we apply the usual
training on all clients to make full use of all local data. Experiments
conducted on CIFAR-10/100 with federated synthetic label noise, and on a
real-world noisy dataset, Clothing1M, demonstrate that $\texttt{FedCorr}$ is
robust to label noise and substantially outperforms the state-of-the-art
methods at multiple noise levels.
