Stochastic modified equations for the asynchronous stochastic gradient
  descent
  We propose a stochastic modified equations (SME) for modeling the
asynchronous stochastic gradient descent (ASGD) algorithms. The resulting SME
of Langevin type extracts more information about the ASGD dynamics and
elucidates the relationship between different types of stochastic gradient
algorithms. We show the convergence of ASGD to the SME in the continuous time
limit, as well as the SME's precise prediction to the trajectories of ASGD with
various forcing terms. As an application of the SME, we propose an optimal
mini-batching strategy for ASGD via solving the optimal control problem of the
associated SME.
