PARADISE: Exploiting Parallel Data for Multilingual Sequence-to-Sequence
  Pretraining
  Despite the success of multilingual sequence-to-sequence pretraining, most
existing approaches rely on monolingual corpora, and do not make use of the
strong cross-lingual signal contained in parallel data. In this paper, we
present PARADISE (PARAllel & Denoising Integration in SEquence-to-sequence
models), which extends the conventional denoising objective used to train these
models by (i) replacing words in the noised sequence according to a
multilingual dictionary, and (ii) predicting the reference translation
according to a parallel corpus instead of recovering the original sequence. Our
experiments on machine translation and cross-lingual natural language inference
show an average improvement of 2.0 BLEU points and 6.7 accuracy points from
integrating parallel data into pretraining, respectively, obtaining results
that are competitive with several popular models at a fraction of their
computational cost.
