Multi-Model Federated Learning
  Federated learning is a form of distributed learning with the key challenge
being the non-identically distributed nature of the data in the participating
clients. In this paper, we extend federated learning to the setting where
multiple unrelated models are trained simultaneously. Specifically, every
client is able to train any one of M models at a time and the server maintains
a model for each of the M models which is typically a suitably averaged version
of the model computed by the clients. We propose multiple policies for
assigning learning tasks to clients over time. In the first policy, we extend
the widely studied FedAvg to multi-model learning by allotting models to
clients in an i.i.d. stochastic manner. In addition, we propose two new
policies for client selection in a multi-model federated setting which make
decisions based on current local losses for each client-model pair. We compare
the performance of the policies on tasks involving synthetic and real-world
data and characterize the performance of the proposed policies. The key
take-away from our work is that the proposed multi-model policies perform
better or at least as good as single model training using FedAvg.
