Distributed SGD Generalizes Well Under Asynchrony
  The performance of fully synchronized distributed systems has faced a
bottleneck due to the big data trend, under which asynchronous distributed
systems are becoming a major popularity due to their powerful scalability. In
this paper, we study the generalization performance of stochastic gradient
descent (SGD) on a distributed asynchronous system. The system consists of
multiple worker machines that compute stochastic gradients which are further
sent to and aggregated on a common parameter server to update the variables,
and the communication in the system suffers from possible delays. Under the
algorithm stability framework, we prove that distributed asynchronous SGD
generalizes well given enough data samples in the training optimization. In
particular, our results suggest to reduce the learning rate as we allow more
asynchrony in the distributed system. Such adaptive learning rate strategy
improves the stability of the distributed algorithm and reduces the
corresponding generalization error. Then, we confirm our theoretical findings
via numerical experiments.
