Local SGD: Unified Theory and New Efficient Methods
  We present a unified framework for analyzing local SGD methods in the convex
and strongly convex regimes for distributed/federated training of supervised
machine learning models. We recover several known methods as a special case of
our general framework, including Local-SGD/FedAvg, SCAFFOLD, and several
variants of SGD not originally designed for federated learning. Our framework
covers both the identical and heterogeneous data settings, supports both random
and deterministic number of local steps, and can work with a wide array of
local stochastic gradient estimators, including shifted estimators which are
able to adjust the fixed points of local iterations for faster convergence. As
an application of our framework, we develop multiple novel FL optimizers which
are superior to existing methods. In particular, we develop the first linearly
converging local SGD method which does not require any data homogeneity or
other strong assumptions.
