To Tune or Not to Tune? Adapting Pretrained Representations to Diverse
  Tasks
  While most previous work has focused on different pretraining objectives and
architectures for transfer learning, we ask how to best adapt the pretrained
model to a given target task. We focus on the two most common forms of
adaptation, feature extraction (where the pretrained weights are frozen), and
directly fine-tuning the pretrained model. Our empirical results across diverse
NLP tasks with two state-of-the-art models show that the relative performance
of fine-tuning vs. feature extraction depends on the similarity of the
pretraining and target tasks. We explore possible explanations for this finding
and provide a set of adaptation guidelines for the NLP practitioner.
