Communication-Efficient Federated Learning with Compensated
  Overlap-FedAvg
  Petabytes of data are generated each day by emerging Internet of Things
(IoT), but only few of them can be finally collected and used for Machine
Learning (ML) purposes due to the apprehension of data & privacy leakage, which
seriously retarding ML's growth. To alleviate this problem, Federated learning
is proposed to perform model training by multiple clients' combined data
without the dataset sharing within the cluster. Nevertheless, federated
learning introduces massive communication overhead as the synchronized data in
each epoch is of the same size as the model, and thereby leading to a low
communication efficiency. Consequently, variant methods mainly focusing on the
communication rounds reduction and data compression are proposed to reduce the
communication overhead of federated learning. In this paper, we propose
Overlap-FedAvg, a framework that parallels the model training phase with model
uploading & downloading phase, so that the latter phase can be totally covered
by the former phase. Compared to vanilla FedAvg, Overlap-FedAvg is further
developed with a hierarchical computing strategy, a data compensation mechanism
and a nesterov accelerated gradients~(NAG) algorithm. Besides, Overlap-FedAvg
is orthogonal to many other compression methods so that they can be applied
together to maximize the utilization of the cluster. Furthermore, the
theoretical analysis is provided to prove the convergence of the proposed
Overlap-FedAvg framework. Extensive experiments on both conventional and
recurrent tasks with multiple models and datasets also demonstrate that the
proposed Overlap-FedAvg framework substantially boosts the federated learning
process.
