FedPAQ: A Communication-Efficient Federated Learning Method with
  Periodic Averaging and Quantization
  Federated learning is a distributed framework according to which a model is
trained over a set of devices, while keeping data localized. This framework
faces several systems-oriented challenges which include (i) communication
bottleneck since a large number of devices upload their local updates to a
parameter server, and (ii) scalability as the federated network consists of
millions of devices. Due to these systems challenges as well as issues related
to statistical heterogeneity of data and privacy concerns, designing a provably
efficient federated learning method is of significant importance yet it remains
challenging. In this paper, we present FedPAQ, a communication-efficient
Federated Learning method with Periodic Averaging and Quantization. FedPAQ
relies on three key features: (1) periodic averaging where models are updated
locally at devices and only periodically averaged at the server; (2) partial
device participation where only a fraction of devices participate in each round
of the training; and (3) quantized message-passing where the edge nodes
quantize their updates before uploading to the parameter server. These features
address the communications and scalability challenges in federated learning. We
also show that FedPAQ achieves near-optimal theoretical guarantees for strongly
convex and non-convex loss functions and empirically demonstrate the
communication-computation tradeoff provided by our method.
