Accelerated Large Batch Optimization of BERT Pretraining in 54 minutes
  BERT has recently attracted a lot of attention in natural language
understanding (NLU) and achieved state-of-the-art results in various NLU tasks.
However, its success requires large deep neural networks and huge amount of
data, which result in long training time and impede development progress. Using
stochastic gradient methods with large mini-batch has been advocated as an
efficient tool to reduce the training time. Along this line of research, LAMB
is a prominent example that reduces the training time of BERT from 3 days to 76
minutes on a TPUv3 Pod. In this paper, we propose an accelerated gradient
method called LANS to improve the efficiency of using large mini-batches for
training. As the learning rate is theoretically upper bounded by the inverse of
the Lipschitz constant of the function, one cannot always reduce the number of
optimization iterations by selecting a larger learning rate. In order to use
larger mini-batch size without accuracy loss, we develop a new learning rate
scheduler that overcomes the difficulty of using large learning rate. Using the
proposed LANS method and the learning rate scheme, we scaled up the mini-batch
sizes to 96K and 33K in phases 1 and 2 of BERT pretraining, respectively. It
takes 54 minutes on 192 AWS EC2 P3dn.24xlarge instances to achieve a target F1
score of 90.5 or higher on SQuAD v1.1, achieving the fastest BERT training time
in the cloud.
