Leader Stochastic Gradient Descent for Distributed Training of Deep
  Learning Models: Extension
  We consider distributed optimization under communication constraints for
training deep learning models. We propose a new algorithm, whose parameter
updates rely on two forces: a regular gradient step, and a corrective direction
dictated by the currently best-performing worker (leader). Our method differs
from the parameter-averaging scheme EASGD in a number of ways: (i) our
objective formulation does not change the location of stationary points
compared to the original optimization problem; (ii) we avoid convergence
decelerations caused by pulling local workers descending to different local
minima to each other (i.e. to the average of their parameters); (iii) our
update by design breaks the curse of symmetry (the phenomenon of being trapped
in poorly generalizing sub-optimal solutions in symmetric non-convex
landscapes); and (iv) our approach is more communication efficient since it
broadcasts only parameters of the leader rather than all workers. We provide
theoretical analysis of the batch version of the proposed algorithm, which we
call Leader Gradient Descent (LGD), and its stochastic variant (LSGD). Finally,
we implement an asynchronous version of our algorithm and extend it to the
multi-leader setting, where we form groups of workers, each represented by its
own local leader (the best performer in a group), and update each worker with a
corrective direction comprised of two attractive forces: one to the local, and
one to the global leader (the best performer among all workers). The
multi-leader setting is well-aligned with current hardware architecture, where
local workers forming a group lie within a single computational node and
different groups correspond to different nodes. For training convolutional
neural networks, we empirically demonstrate that our approach compares
favorably to state-of-the-art baselines. This work is a gentle extension of
[2].
