Faster Non-Convex Federated Learning via Global and Local Momentum
  We propose \texttt{FedGLOMO}, a novel federated learning (FL) algorithm with
an iteration complexity of $\mathcal{O}(\epsilon^{-1.5})$ to converge to an
$\epsilon$-stationary point (i.e., $\mathbb{E}[\|\nabla f(\bm{x})\|^2] \leq
\epsilon$) for smooth non-convex functions -- under arbitrary client
heterogeneity and compressed communication -- compared to the
$\mathcal{O}(\epsilon^{-2})$ complexity of most prior works. Our key
algorithmic idea that enables achieving this improved complexity is based on
the observation that the convergence in FL is hampered by two sources of high
variance: (i) the global server aggregation step with multiple local updates,
exacerbated by client heterogeneity, and (ii) the noise of the local
client-level stochastic gradients. By modeling the server aggregation step as a
generalized gradient-type update, we propose a variance-reducing momentum-based
global update at the server, which when applied in conjunction with
variance-reduced local updates at the clients, enables \texttt{FedGLOMO} to
enjoy an improved convergence rate. Moreover, we derive our results under a
novel and more realistic client-heterogeneity assumption which we verify
empirically -- unlike prior assumptions that are hard to verify. Our
experiments illustrate the intrinsic variance reduction effect of
\texttt{FedGLOMO}, which implicitly suppresses client-drift in heterogeneous
data distribution settings and promotes communication efficiency.
