Efficient Federated Learning for AIoT Applications Using Knowledge
  Distillation
  As a promising distributed machine learning paradigm, Federated Learning (FL)
trains a central model with decentralized data without compromising user
privacy, which has made it widely used by Artificial Intelligence Internet of
Things (AIoT) applications. However, the traditional FL suffers from model
inaccuracy since it trains local models using hard labels of data and ignores
useful information of incorrect predictions with small probabilities. Although
various solutions try to tackle the bottleneck of the traditional FL, most of
them introduce significant communication and memory overhead, making the
deployment of large-scale AIoT devices a great challenge. To address the above
problem, this paper presents a novel Distillation-based Federated Learning
(DFL) architecture that enables efficient and accurate FL for AIoT
applications. Inspired by Knowledge Distillation (KD) that can increase the
model accuracy, our approach adds the soft targets used by KD to the FL model
training, which occupies negligible network resources. The soft targets are
generated by local sample predictions of each AIoT device after each round of
local training and used for the next round of model training. During the local
training of DFL, both soft targets and hard labels are used as approximation
objectives of model predictions to improve model accuracy by supplementing the
knowledge of soft targets. To further improve the performance of our DFL model,
we design a dynamic adjustment strategy for tuning the ratio of two loss
functions used in KD, which can maximize the use of both soft targets and hard
labels. Comprehensive experimental results on well-known benchmarks show that
our approach can significantly improve the model accuracy of FL with both
Independent and Identically Distributed (IID) and non-IID data.
