Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge
  Scaling up the convolutional neural network (CNN) size (e.g., width, depth,
etc.) is known to effectively improve model accuracy. However, the large model
size impedes training on resource-constrained edge devices. For instance,
federated learning (FL) may place undue burden on the compute capability of
edge nodes, even though there is a strong practical need for FL due to its
privacy and confidentiality properties. To address the resource-constrained
reality of edge devices, we reformulate FL as a group knowledge transfer
training algorithm, called FedGKT. FedGKT designs a variant of the alternating
minimization approach to train small CNNs on edge nodes and periodically
transfer their knowledge by knowledge distillation to a large server-side CNN.
FedGKT consolidates several advantages into a single framework: reduced demand
for edge computation, lower communication bandwidth for large CNNs, and
asynchronous training, all while maintaining model accuracy comparable to
FedAvg. We train CNNs designed based on ResNet-56 and ResNet-110 using three
distinct datasets (CIFAR-10, CIFAR-100, and CINIC-10) and their non-I.I.D.
variants. Our results show that FedGKT can obtain comparable or even slightly
higher accuracy than FedAvg. More importantly, FedGKT makes edge training
affordable. Compared to the edge training using FedAvg, FedGKT demands 9 to 17
times less computational power (FLOPs) on edge devices and requires 54 to 105
times fewer parameters in the edge CNN. Our source code is released at FedML
(https://fedml.ai).
