Benchmarking down-scaled (not so large) pre-trained language models
  Large Transformer-based language models are pre-trained on corpora of varying
sizes, for a different number of steps and with different batch sizes. At the
same time, more fundamental components, such as the pre-training objective or
architectural hyperparameters, are modified. In total, it is therefore
difficult to ascribe changes in performance to specific factors. Since
searching the hyperparameter space over the full systems is too costly, we
pre-train down-scaled versions of several popular Transformer-based
architectures on a common pre-training corpus and benchmark them on a subset of
the GLUE tasks (Wang et al., 2018). Specifically, we systematically compare
three pre-training objectives for different shape parameters and model sizes,
while also varying the number of pre-training steps and the batch size. In our
experiments MLM + NSP (BERT-style) consistently outperforms MLM (RoBERTa-style)
as well as the standard LM objective. Furthermore, we find that additional
compute should be mainly allocated to an increased model size, while training
for more steps is inefficient. Based on these observations, as a final step we
attempt to scale up several systems using compound scaling (Tan and Le, 2019)
adapted to Transformer-based language models.
