Curriculum Learning: A Regularization Method for Efficient and Stable
  Billion-Scale GPT Model Pre-Training
  Recent works have demonstrated great success in training large autoregressive
language models (e.g., GPT-3) on unlabeled text corpus for text generation. To
reduce their expensive training cost, practitioners attempt to increase the
batch sizes and learning rates. However, increasing them often cause training
instabilities and poor generalization. On the other side, using smaller batch
sizes or learning rates would reduce the training efficiency, significantly
increasing training time and cost. We investigate this stability-efficiency
dilemma and identify that long sequence length is one of the main causes of
training instability in large-scale GPT model pre-training.
  Based on our analysis, we present a novel sequence length warmup method that
simultaneously improves training stability and efficiency. As a kind of
curriculum learning approach, our method improves the training convergence
speed of autoregressive models. More importantly, our in-depth analysis shows
that our method exerts a gradient variance reduction effect and regularizes
early stages of training where the amount of training data is much smaller than
the model capacity. This enables stable training with much larger batch sizes
and learning rates, further improving the training speed. Evaluations show that
our approach enables stable GPT-2 (117M and 1.5B) pre-training with 8x larger
batch size and 4x larger learning rate, whereas the baseline approach struggles
with training instability. To achieve the same or better zero-shot
WikiText-103/LAMBADA evaluation results, our approach reduces the required
number of pre-training tokens and wall clock time by up to 55\% and 73\%,
respectively.
