Concurrent Adversarial Learning for Large-Batch Training
  Large-batch training has become a commonly used technique when training
neural networks with a large number of GPU/TPU processors. As batch size
increases, stochastic optimizers tend to converge to sharp local minima,
leading to degraded test performance. Current methods usually use extensive
data augmentation to increase the batch size, but we found the performance gain
with data augmentation decreases as batch size increases, and data augmentation
will become insufficient after certain point. In this paper, we propose to use
adversarial learning to increase the batch size in large-batch training.
Despite being a natural choice for smoothing the decision surface and biasing
towards a flat region, adversarial learning has not been successfully applied
in large-batch training since it requires at least two sequential gradient
computations at each step, which will at least double the running time compared
with vanilla training even with a large number of processors. To overcome this
issue, we propose a novel Concurrent Adversarial Learning (ConAdv) method that
decouple the sequential gradient computations in adversarial learning by
utilizing staled parameters. Experimental results demonstrate that ConAdv can
successfully increase the batch size on ResNet-50 training on ImageNet while
maintaining high accuracy. In particular, we show ConAdv along can achieve
75.3\% top-1 accuracy on ImageNet ResNet-50 training with 96K batch size, and
the accuracy can be further improved to 76.2\% when combining ConAdv with data
augmentation. This is the first work successfully scales ResNet-50 training
batch size to 96K.
