A Unified Linear Speedup Analysis of Stochastic FedAvg and Nesterov
  Accelerated FedAvg
  Federated learning (FL) learns a model jointly from a set of participating
devices without sharing each other's privately held data. The characteristics
of non-i.i.d. data across the network, low device participation, high
communication costs, and the mandate that data remain private bring challenges
in understanding the convergence of FL algorithms, particularly with regards to
how convergence scales with the number of participating devices. In this paper,
we focus on Federated Averaging (FedAvg)--arguably the most popular and
effective FL algorithm class in use today--and provide a unified and
comprehensive study of its convergence rate. Although FedAvg has recently been
studied by an emerging line of literature, a systematic study of how FedAvg's
convergence scales with the number of participating devices in the fully
heterogeneous FL setting is lacking--a crucial issue whose answer would shed
light on the performance of FedAvg in large FL systems in practice. We fill
this gap by providing a unified analysis that establishes convergence
guarantees for FedAvg under strongly convex smooth, convex smooth problems, and
overparameterized strongly convex smooth problems. We show that FedAvg enjoys
linear speedup in each case, although with different convergence rates and
communication efficiencies. While there have been linear speedup results from
distributed optimization that assumes full participation, ours are the first to
establish linear speedup for FedAvg under both statistical and system
heterogeneity. For strongly convex and convex problems, we also characterize
the corresponding convergence rates for the Nesterov accelerated FedAvg
algorithm, which are the first linear speedup guarantees for momentum variants
of FedAvg in convex settings. Empirical studies of the algorithms in various
settings have supported our theoretical results.
