Federated Knowledge Distillation
  Distributed learning frameworks often rely on exchanging model parameters
across workers, instead of revealing their raw data. A prime example is
federated learning that exchanges the gradients or weights of each neural
network model. Under limited communication resources, however, such a method
becomes extremely costly particularly for modern deep neural networks having a
huge number of model parameters. In this regard, federated distillation (FD) is
a compelling distributed learning solution that only exchanges the model
outputs whose dimensions are commonly much smaller than the model sizes (e.g.,
10 labels in the MNIST dataset). The goal of this chapter is to provide a deep
understanding of FD while demonstrating its communication efficiency and
applicability to a variety of tasks. To this end, towards demystifying the
operational principle of FD, the first part of this chapter provides a novel
asymptotic analysis for two foundational algorithms of FD, namely knowledge
distillation (KD) and co-distillation (CD), by exploiting the theory of neural
tangent kernel (NTK). Next, the second part elaborates on a baseline
implementation of FD for a classification task, and illustrates its performance
in terms of accuracy and communication efficiency compared to FL. Lastly, to
demonstrate the applicability of FD to various distributed learning tasks and
environments, the third part presents two selected applications, namely FD over
asymmetric uplink-and-downlink wireless channels and FD for reinforcement
learning.
