Scale Efficiently: Insights from Pre-training and Fine-tuning
  Transformers
  There remain many open questions pertaining to the scaling behaviour of
Transformer architectures. These scaling decisions and findings can be
critical, as training runs often come with an associated computational cost
which have both financial and/or environmental impact. The goal of this paper
is to present scaling insights from pretraining and finetuning Transformers.
While Kaplan et al. presents a comprehensive study of the scaling behaviour of
Transformer language models, the scope is only on the upstream (pretraining)
loss. Therefore, it is still unclear if these set of findings transfer to
downstream task within the context of the pretrain-finetune paradigm. The key
findings of this paper are as follows: (1) we show that aside from only the
model size, model shape matters for downstream fine-tuning, (2) scaling
protocols operate differently at different compute regions, (3) widely adopted
T5-base and T5-large sizes are Pareto-inefficient. To this end, we present
improved scaling protocols whereby our redesigned models achieve similar
downstream fine-tuning quality while having 50\% fewer parameters and training
40\% faster compared to the widely adopted T5-base model. We publicly release
over 100 pretrained checkpoints of different T5 configurations to facilitate
future research and analysis.
