Local Adaptivity in Federated Learning: Convergence and Consistency
  The federated learning (FL) framework trains a machine learning model using
decentralized data stored at edge client devices by periodically aggregating
locally trained models. Popular optimization algorithms of FL use vanilla
(stochastic) gradient descent for both local updates at clients and global
updates at the aggregating server. Recently, adaptive optimization methods such
as AdaGrad have been studied for server updates. However, the effect of using
adaptive optimization methods for local updates at clients is not yet
understood. We show in both theory and practice that while local adaptive
methods can accelerate convergence, they can cause a non-vanishing solution
bias, where the final converged solution may be different from the stationary
point of the global objective function. We propose correction techniques to
overcome this inconsistency and complement the local adaptive methods for FL.
Extensive experiments on realistic federated training tasks show that the
proposed algorithms can achieve faster convergence and higher test accuracy
than the baselines without local adaptivity.
