ScholarBERT: Bigger is Not Always Better
  Transformer-based masked language models trained on general corpora, such as
BERT and RoBERTa, have shown impressive performance on various downstream
tasks. Increasingly, researchers are "finetuning" these models to improve
performance on domain-specific tasks. Here, we report a broad study in which we
applied 14 transformer-based models to 11 scientific tasks in order to evaluate
how downstream performance is affected by changes along various dimensions
(e.g., training data, model size, pretraining time, finetuning length). In this
process, we created the largest and most diverse scientific language model to
date, ScholarBERT, by training a 770M-parameter BERT model on an 221B token
scientific literature dataset spanning many disciplines. Counterintuitively,
our evaluation of the 14 BERT-based models (seven versions of ScholarBERT, five
science-specific large language models from the literature, BERT-Base, and
BERT-Large) reveals little difference in performance across the 11
science-focused tasks, despite major differences in model size and training
data. We argue that our results establish an upper bound for the performance
achievable with BERT-based architectures on tasks from the scientific domain.
