Unsupervised Transfer Learning via BERT Neuron Selection
  Recent advancements in language representation models such as BERT have led
to a rapid improvement in numerous natural language processing tasks. However,
language models usually consist of a few hundred million trainable parameters
with embedding space distributed across multiple layers, thus making them
challenging to be fine-tuned for a specific task or to be transferred to a new
domain. To determine whether there are task-specific neurons that can be
exploited for unsupervised transfer learning, we introduce a method for
selecting the most important neurons to solve a specific classification task.
This algorithm is further extended to multi-source transfer learning by
computing the importance of neurons for several single-source transfer learning
scenarios between different subsets of data sources. Besides, a task-specific
fingerprint for each data source is obtained based on the percentage of the
selected neurons in each layer. We perform extensive experiments in
unsupervised transfer learning for sentiment analysis, natural language
inference and sentence similarity, and compare our results with the existing
literature and baselines. Significantly, we found that the source and target
data sources with higher degrees of similarity between their task-specific
fingerprints demonstrate a better transferability property. We conclude that
our method can lead to better performance using just a few hundred
task-specific and interpretable neurons.
