Accelerate Distributed Stochastic Descent for Nonconvex Optimization
  with Momentum
  Momentum method has been used extensively in optimizers for deep learning.
Recent studies show that distributed training through K-step averaging has many
nice properties. We propose a momentum method for such model averaging
approaches. At each individual learner level traditional stochastic gradient is
applied. At the meta-level (global learner level), one momentum term is applied
and we call it block momentum. We analyze the convergence and scaling
properties of such momentum methods. Our experimental results show that block
momentum not only accelerates training, but also achieves better results.
