Personalised Federated Learning: A Combinational Approach
  Federated learning (FL) is a distributed machine learning approach involving
multiple clients collaboratively training a shared model. Such a system has the
advantage of more training data from multiple clients, but data can be
non-identically and independently distributed (non-i.i.d.). Privacy and
integrity preserving features such as differential privacy (DP) and robust
aggregation (RA) are commonly used in FL. In this work, we show that on common
deep learning tasks, the performance of FL models differs amongst clients and
situations, and FL models can sometimes perform worse than local models due to
non-i.i.d. data. Secondly, we show that incorporating DP and RA degrades
performance further. Then, we conduct an ablation study on the performance
impact of different combinations of common personalization approaches for FL,
such as finetuning, mixture-of-experts ensemble, multi-task learning, and
knowledge distillation. It is observed that certain combinations of
personalization approaches are more impactful in certain scenarios while others
always improve performance, and combination approaches are better than
individual ones. Most clients obtained better performance with combined
personalized FL and recover from performance degradation caused by non-i.i.d.
data, DP, and RA.
