FedLite: A Scalable Approach for Federated Learning on
  Resource-constrained Clients
  In classical federated learning, the clients contribute to the overall
training by communicating local updates for the underlying model on their
private data to a coordinating server. However, updating and communicating the
entire model becomes prohibitively expensive when resource-constrained clients
collectively aim to train a large machine learning model. Split learning
provides a natural solution in such a setting, where only a small part of the
model is stored and trained on clients while the remaining large part of the
model only stays at the servers. However, the model partitioning employed in
split learning introduces a significant amount of communication cost. This
paper addresses this issue by compressing the additional communication using a
novel clustering scheme accompanied by a gradient correction method. Extensive
empirical evaluations on image and text benchmarks show that the proposed
method can achieve up to $490\times$ communication cost reduction with minimal
drop in accuracy, and enables a desirable performance vs. communication
trade-off.
