CD$^2$-pFed: Cyclic Distillation-guided Channel Decoupling for Model
  Personalization in Federated Learning
  Federated learning (FL) is a distributed learning paradigm that enables
multiple clients to collaboratively learn a shared global model. Despite the
recent progress, it remains challenging to deal with heterogeneous data
clients, as the discrepant data distributions usually prevent the global model
from delivering good generalization ability on each participating client. In
this paper, we propose CD^2-pFed, a novel Cyclic Distillation-guided Channel
Decoupling framework, to personalize the global model in FL, under various
settings of data heterogeneity. Different from previous works which establish
layer-wise personalization to overcome the non-IID data across different
clients, we make the first attempt at channel-wise assignment for model
personalization, referred to as channel decoupling. To further facilitate the
collaboration between private and shared weights, we propose a novel cyclic
distillation scheme to impose a consistent regularization between the local and
global model representations during the federation. Guided by the cyclical
distillation, our channel decoupling framework can deliver more accurate and
generalized results for different kinds of heterogeneity, such as feature skew,
label distribution skew, and concept shift. Comprehensive experiments on four
benchmarks, including natural image and medical image analysis tasks,
demonstrate the consistent effectiveness of our method on both local and
external validations.
