Edge Bias in Federated Learning and its Solution by Buffered Knowledge
  Distillation
  Federated learning (FL), which utilizes communication between the server
(core) and local devices (edges) to indirectly learn from more data, is an
emerging field in deep learning research. Recently, Knowledge
Distillation-based FL methods with notable performance and high applicability
have been suggested. In this paper, we choose knowledge distillation-based FL
method as our baseline and tackle a challenging problem that ensues from using
these methods. Especially, we focus on the problem incurred in the server model
that tries to mimic different datasets, each of which is unique to an
individual edge device. We dub the problem 'edge bias', which occurs when
multiple teacher models trained on different datasets are used individually to
distill knowledge. We introduce this nuisance that occurs in certain scenarios
of FL, and to alleviate it, we propose a simple yet effective distillation
scheme named 'buffered distillation'. In addition, we also experimentally show
that this scheme is effective in mitigating the straggler problem caused by
delayed edges.
