Scaling Language Model Size in Cross-Device Federated Learning
  Most studies in cross-device federated learning focus on small models, due to
the server-client communication and on-device computation bottlenecks. In this
work, we leverage various techniques for mitigating these bottlenecks to train
larger language models in cross-device federated learning. With systematic
applications of partial model training, quantization, efficient transfer
learning, and communication-efficient optimizers, we are able to train a $21$M
parameter Transformer that achieves the same perplexity as that of a similarly
sized LSTM with $\sim10\times$ smaller client-to-server communication cost and
$11\%$ lower perplexity than smaller LSTMs commonly studied in literature.
