Packing: Towards 2x NLP BERT Acceleration
  We find that at sequence length 512 padding tokens represent in excess of 50%
of the Wikipedia dataset used for pretraining BERT (Bidirectional Encoder
Representations from Transformers). Therefore by removing all padding we
achieve a 2x speed-up in terms of sequences/sec. To exploit this characteristic
of the dataset, we develop and contrast two deterministic packing algorithms.
Both algorithms rely on the assumption that sequences are interchangeable and
therefore packing can be performed on the histogram of sequence lengths, rather
than per sample. This transformation of the problem leads to algorithms which
are fast and have linear complexity in dataset size. The shortest-pack-first
histogram-packing (SPFHP) algorithm determines the packing order for the
Wikipedia dataset of over 16M sequences in 0.02 seconds. The non-negative
least-squares histogram-packing (NNLSHP) algorithm converges in 28.4 seconds
but produces solutions which are more depth efficient, managing to get near
optimal packing by combining a maximum of 3 sequences in one sample. Using the
dataset with multiple sequences per sample requires additional masking in the
attention layer and a modification of the MLM loss function. We demonstrate
that both of these changes are straightforward to implement and have relatively
little impact on the achievable performance gain on modern hardware. Finally,
we pretrain BERT-Large using the packed dataset, demonstrating no loss of
convergence and the desired 2x speed-up.
