On the Complementarity between Pre-Training and Back-Translation for
  Neural Machine Translation
  Pre-training (PT) and back-translation (BT) are two simple and powerful
methods to utilize monolingual data for improving the model performance of
neural machine translation (NMT). This paper takes the first step to
investigate the complementarity between PT and BT. We introduce two probing
tasks for PT and BT respectively and find that PT mainly contributes to the
encoder module while BT brings more benefits to the decoder. Experimental
results show that PT and BT are nicely complementary to each other,
establishing state-of-the-art performances on the WMT16 English-Romanian and
English-Russian benchmarks. Through extensive analyses on sentence originality
and word frequency, we also demonstrate that combining Tagged BT with PT is
more helpful to their complementarity, leading to better translation quality.
Source code is freely available at https://github.com/SunbowLiu/PTvsBT.
