Investigating Transferability in Pretrained Language Models
  How does language model pretraining help transfer learning? We consider a
simple ablation technique for determining the impact of each pretrained layer
on transfer task performance. This method, partial reinitialization, involves
replacing different layers of a pretrained model with random weights, then
finetuning the entire model on the transfer task and observing the change in
performance. This technique reveals that in BERT, layers with high probing
performance on downstream GLUE tasks are neither necessary nor sufficient for
high accuracy on those tasks. Furthermore, the benefit of using pretrained
parameters for a layer varies dramatically with finetuning dataset size:
parameters that provide tremendous performance improvement when data is
plentiful may provide negligible benefits in data-scarce settings. These
results reveal the complexity of the transfer learning process, highlighting
the limitations of methods that operate on frozen models or single data
samples.
