Adaptive Braking for Mitigating Gradient Delay
  Neural network training is commonly accelerated by using multiple
synchronized workers to compute gradient updates in parallel. Asynchronous
methods remove synchronization overheads and improve hardware utilization at
the cost of introducing gradient delay, which impedes optimization and can lead
to lower final model performance. We introduce Adaptive Braking (AB), a
modification for momentum-based optimizers that mitigates the effects of
gradient delay. AB dynamically scales the gradient based on the alignment of
the gradient and the velocity. This can dampen oscillations along high
curvature directions of the loss surface, stabilizing and accelerating
asynchronous training. We show that applying AB on top of SGD with momentum
enables training ResNets on CIFAR-10 and ImageNet-1k with delays $D \geq$ 32
update steps with minimal drop in final test accuracy.
