A continuous-time analysis of distributed stochastic gradient
  We analyze the effect of synchronization on distributed stochastic gradient
algorithms. By exploiting an analogy with dynamical models of biological quorum
sensing - where synchronization between agents is induced through communication
with a common signal - we quantify how synchronization can significantly reduce
the magnitude of the noise felt by the individual distributed agents and by
their spatial mean. This noise reduction is in turn associated with a reduction
in the smoothing of the loss function imposed by the stochastic gradient
approximation. Through simulations on model non-convex objectives, we
demonstrate that coupling can stabilize higher noise levels and improve
convergence. We provide a convergence analysis for strongly convex functions by
deriving a bound on the expected deviation of the spatial mean of the agents
from the global minimizer for an algorithm based on quorum sensing, the same
algorithm with momentum, and the Elastic Averaging SGD (EASGD) algorithm. We
discuss extensions to new algorithms that allow each agent to broadcast its
current measure of success and shape the collective computation accordingly. We
supplement our theoretical analysis with numerical experiments on convolutional
neural networks trained on the CIFAR-10 dataset, where we note a surprising
regularizing property of EASGD even when applied to the non-distributed case.
This observation suggests alternative second-order in-time algorithms for
non-distributed optimization that are competitive with momentum methods.
