Personalized Federated Learning for Heterogeneous Clients with Clustered
  Knowledge Transfer
  Personalized federated learning (FL) aims to train model(s) that can perform
well for individual clients that are highly data and system heterogeneous. Most
work in personalized FL, however, assumes using the same model architecture at
all clients and increases the communication cost by sending/receiving models.
This may not be feasible for realistic scenarios of FL. In practice, clients
have highly heterogeneous system-capabilities and limited communication
resources. In our work, we propose a personalized FL framework, PerFed-CKT,
where clients can use heterogeneous model architectures and do not directly
communicate their model parameters. PerFed-CKT uses clustered co-distillation,
where clients use logits to transfer their knowledge to other clients that have
similar data-distributions. We theoretically show the convergence and
generalization properties of PerFed-CKT and empirically show that PerFed-CKT
achieves high test accuracy with several orders of magnitude lower
communication cost compared to the state-of-the-art personalized FL schemes.
