An introduction to decentralized stochastic optimization with gradient
  tracking
  Decentralized solutions to finite-sum minimization are of significant
importance in many signal processing, control, and machine learning
applications. In such settings, the data is distributed over a network of
arbitrarily-connected nodes and raw data sharing is prohibitive often due to
communication or privacy constraints. In this article, we review decentralized
stochastic first-order optimization methods and illustrate some recent
improvements based on gradient tracking and variance reduction, focusing
particularly on smooth and strongly-convex objective functions. We provide
intuitive illustrations of the main technical ideas as well as applications of
the algorithms in the context of decentralized training of machine learning
models.
