The Convergence of Stochastic Gradient Descent in Asynchronous Shared
  Memory
  Stochastic Gradient Descent (SGD) is a fundamental algorithm in machine
learning, representing the optimization backbone for training several classic
models, from regression to neural networks. Given the recent practical focus on
distributed machine learning, significant work has been dedicated to the
convergence properties of this algorithm under the inconsistent and noisy
updates arising from execution in a distributed environment. However,
surprisingly, the convergence properties of this classic algorithm in the
standard shared-memory model are still not well-understood.
  In this work, we address this gap, and provide new convergence bounds for
lock-free concurrent stochastic gradient descent, executing in the classic
asynchronous shared memory model, against a strong adaptive adversary. Our
results give improved upper and lower bounds on the "price of asynchrony" when
executing the fundamental SGD algorithm in a concurrent setting. They show that
this classic optimization tool can converge faster and with a wider range of
parameters than previously known under asynchronous iterations. At the same
time, we exhibit a fundamental trade-off between the maximum delay in the
system and the rate at which SGD can converge, which governs the set of
parameters under which this algorithm can still work efficiently.
