Deep learning with Elastic Averaging SGD
  We study the problem of stochastic optimization for deep learning in the
parallel computing environment under communication constraints. A new algorithm
is proposed in this setting where the communication and coordination of work
among concurrent processes (local workers), is based on an elastic force which
links the parameters they compute with a center variable stored by the
parameter server (master). The algorithm enables the local workers to perform
more exploration, i.e. the algorithm allows the local variables to fluctuate
further from the center variable by reducing the amount of communication
between local workers and the master. We empirically demonstrate that in the
deep learning setting, due to the existence of many local optima, allowing more
exploration can lead to the improved performance. We propose synchronous and
asynchronous variants of the new algorithm. We provide the stability analysis
of the asynchronous variant in the round-robin scheme and compare it with the
more common parallelized method ADMM. We show that the stability of EASGD is
guaranteed when a simple stability condition is satisfied, which is not the
case for ADMM. We additionally propose the momentum-based version of our
algorithm that can be applied in both synchronous and asynchronous settings.
Asynchronous variant of the algorithm is applied to train convolutional neural
networks for image classification on the CIFAR and ImageNet datasets.
Experiments demonstrate that the new algorithm accelerates the training of deep
architectures compared to DOWNPOUR and other common baseline approaches and
furthermore is very communication efficient.
