Advances in Asynchronous Parallel and Distributed Optimization
  Motivated by large-scale optimization problems arising in the context of
machine learning, there have been several advances in the study of asynchronous
parallel and distributed optimization methods during the past decade.
Asynchronous methods do not require all processors to maintain a consistent
view of the optimization variables. Consequently, they generally can make more
efficient use of computational resources than synchronous methods, and they are
not sensitive to issues like stragglers (i.e., slow nodes) and unreliable
communication links. Mathematical modeling of asynchronous methods involves
proper accounting of information delays, which makes their analysis
challenging. This article reviews recent developments in the design and
analysis of asynchronous optimization methods, covering both centralized
methods, where all processors update a master copy of the optimization
variables, and decentralized methods, where each processor maintains a local
copy of the variables. The analysis provides insights as to how the degree of
asynchrony impacts convergence rates, especially in stochastic optimization
methods.
