GT-STORM: Taming Sample, Communication, and Memory Complexities in
  Decentralized Non-Convex Learning
  Decentralized nonconvex optimization has received increasing attention in
recent years in machine learning due to its advantages in system robustness,
data privacy, and implementation simplicity. However, three fundamental
challenges in designing decentralized optimization algorithms are how to reduce
their sample, communication, and memory complexities. In this paper, we propose
a \underline{g}radient-\underline{t}racking-based \underline{sto}chastic
\underline{r}ecursive \underline{m}omentum (GT-STORM) algorithm for efficiently
solving nonconvex optimization problems. We show that to reach an
$\epsilon^2$-stationary solution, the total number of sample evaluations of our
algorithm is $\tilde{O}(m^{1/2}\epsilon^{-3})$ and the number of communication
rounds is $\tilde{O}(m^{-1/2}\epsilon^{-3})$, which improve the
$O(\epsilon^{-4})$ costs of sample evaluations and communications for the
existing decentralized stochastic gradient algorithms. We conduct extensive
experiments with a variety of learning models, including non-convex logistical
regression and convolutional neural networks, to verify our theoretical
findings. Collectively, our results contribute to the state of the art of
theories and algorithms for decentralized network optimization.
