Federated Asymptotics: a model to compare federated learning algorithms
  We propose an asymptotic framework to analyze the performance of
(personalized) federated learning algorithms. In this new framework, we
formulate federated learning as a multi-criterion objective, where the goal is
to minimize each client's loss using information from all of the clients. We
analyze a linear regression model where, for a given client, we may
theoretically compare the performance of various algorithms in the
high-dimensional asymptotic limit. This asymptotic multi-criterion approach
naturally models the high-dimensional, many-device nature of federated
learning. These tools make fairly precise predictions about the benefits of
personalization and information sharing in federated scenarios -- at least in
our (stylized) model -- including that Federated Averaging with simple client
fine-tuning achieves the same asymptotic risk as the more intricate
meta-learning and proximal-regularized approaches and outperforming Federated
Averaging without personalization. We evaluate these predictions on federated
versions of the EMNIST, CIFAR-100, Shakespeare, and Stack Overflow datasets,
where the experiments corroborate the theoretical predictions, suggesting such
frameworks may provide a useful guide to practical algorithmic development.
