PFL-MoE: Personalized Federated Learning Based on Mixture of Experts
  Federated learning (FL) is an emerging distributed machine learning paradigm
that avoids data sharing among training nodes so as to protect data privacy.
Under coordination of the FL server, each client conducts model training using
its own computing resource and private data set. The global model can be
created by aggregating the training results of clients. To cope with highly
non-IID data distributions, personalized federated learning (PFL) has been
proposed to improve overall performance by allowing each client to learn a
personalized model. However, one major drawback of a personalized model is the
loss of generalization. To achieve model personalization while maintaining
generalization, in this paper, we propose a new approach, named PFL-MoE, which
mixes outputs of the personalized model and global model via the MoE
architecture. PFL-MoE is a generic approach and can be instantiated by
integrating existing PFL algorithms. Particularly, we propose the PFL-MF
algorithm which is an instance of PFL-MoE based on the freeze-base PFL
algorithm. We further improve PFL-MF by enhancing the decision-making ability
of MoE gating network and propose a variant algorithm PFL-MFE. We demonstrate
the effectiveness of PFL-MoE by training the LeNet-5 and VGG-16 models on the
Fashion-MNIST and CIFAR-10 datasets with non-IID partitions.
