On the Complementarity of Data Selection and Fine Tuning for Domain
  Adaptation
  Domain adaptation of neural networks commonly relies on three training
phases: pretraining, selected data training and then fine tuning. Data
selection improves target domain generalization by training further on
pretraining data identified by relying on a small sample of target domain data.
This work examines the benefit of data selection for language modeling and
machine translation. Our experiments assess the complementarity of selection
with fine tuning and result in practical recommendations: (i) selected data
must be similar to the fine-tuning domain but not so much as to erode the
complementary effect of fine-tuning; (ii) there is a trade-off between
selecting little data for fast but limited progress or much data for slow but
long lasting progress; (iii) data selection can be applied early during
pretraining, with performance gains comparable to long pretraining session;
(iv) data selection from domain classifiers is often more effective than the
popular contrastive data selection method.
