DESTRESS: Computation-Optimal and Communication-Efficient Decentralized
  Nonconvex Finite-Sum Optimization
  Emerging applications in multi-agent environments such as internet-of-things,
networked sensing, autonomous systems and federated learning, call for
decentralized algorithms for finite-sum optimizations that are
resource-efficient in terms of both computation and communication. In this
paper, we consider the prototypical setting where the agents work
collaboratively to minimize the sum of local loss functions by only
communicating with their neighbors over a predetermined network topology. We
develop a new algorithm, called DEcentralized STochastic REcurSive gradient
methodS (DESTRESS) for nonconvex finite-sum optimization, which matches the
optimal incremental first-order oracle (IFO) complexity of centralized
algorithms for finding first-order stationary points, while maintaining
communication efficiency. Detailed theoretical and numerical comparisons
corroborate that the resource efficiencies of DESTRESS improve upon prior
decentralized algorithms over a wide range of parameter regimes. DESTRESS
leverages several key algorithm design ideas including randomly activated
stochastic recursive gradient updates with mini-batches for local computation,
gradient tracking with extra mixing (i.e., multiple gossiping rounds) for
per-iteration communication, together with careful choices of hyper-parameters
and new analysis frameworks to provably achieve a desirable
computation-communication trade-off.
