Federated Dropout -- A Simple Approach for Enabling Federated Learning
  on Resource Constrained Devices
  Federated learning (FL) is a popular framework for training an AI model using
distributed mobile data in a wireless network. It features data parallelism by
distributing the learning task to multiple edge devices while attempting to
preserve their local-data privacy. One main challenge confronting practical FL
is that resource constrained devices struggle with the computation intensive
task of updating of a deep-neural network model. To tackle the challenge, in
this paper, a federated dropout (FedDrop) scheme is proposed building on the
classic dropout scheme for random model pruning. Specifically, in each
iteration of the FL algorithm, several subnets are independently generated from
the global model at the server using dropout but with heterogeneous dropout
rates (i.e., parameter-pruning probabilities),each of which is adapted to the
state of an assigned channel. The subnets are downloaded to associated devices
for updating. Thereby, FedDrop reduces both the communication overhead and
devices' computation loads compared with the conventional FL while
outperforming the latter in the case of overfitting and also the FL scheme with
uniform dropout (i.e., identical subnets).
