Aligning the Pretraining and Finetuning Objectives of Language Models
  We demonstrate that explicitly aligning the pretraining objectives to the
finetuning objectives in language model training significantly improves the
finetuning task performance and reduces the minimum amount of finetuning
examples required. The performance margin gained from objective alignment
allows us to build language models with smaller sizes for tasks with less
available training data. We provide empirical evidence of these claims by
applying objective alignment to concept-of-interest tagging and acronym
detection tasks. We found that, with objective alignment, our 768 by 3 and 512
by 3 transformer language models can reach accuracy of 83.9%/82.5% for
concept-of-interest tagging and 73.8%/70.2% for acronym detection using only
200 finetuning examples per task, outperforming the 768 by 3 model pretrained
without objective alignment by +4.8%/+3.4% and +9.9%/+6.3%. We name finetuning
small language models in the presence of hundreds of training examples or less
"Few Example learning". In practice, Few Example Learning enabled by objective
alignment not only saves human labeling costs, but also makes it possible to
leverage language models in more real-time applications.
