FedKD: Communication Efficient Federated Learning via Knowledge
  Distillation
  Federated learning is widely used to learn intelligent models from
decentralized data. In federated learning, clients need to communicate their
local model updates in each iteration of model learning. However, model updates
are large in size if the model contains numerous parameters, and there usually
needs many rounds of communication until model converges. Thus, the
communication cost in federated learning can be quite heavy. In this paper, we
propose a communication efficient federated learning method based on knowledge
distillation. Instead of directly communicating the large models between
clients and server, we propose an adaptive mutual distillation framework to
reciprocally learn a student and a teacher model on each client, where only the
student model is shared by different clients and updated collaboratively to
reduce the communication cost. Both the teacher and student on each client are
learned on its local data and the knowledge distilled from each other, where
their distillation intensities are controlled by their prediction quality. To
further reduce the communication cost, we propose a dynamic gradient
approximation method based on singular value decomposition to approximate the
exchanged gradients with dynamic precision. Extensive experiments on benchmark
datasets in different tasks show that our approach can effectively reduce the
communication cost and achieve competitive results.
