On the use of BERT for Neural Machine Translation
  Exploiting large pretrained models for various NMT tasks have gained a lot of
visibility recently. In this work we study how BERT pretrained models could be
exploited for supervised Neural Machine Translation. We compare various ways to
integrate pretrained BERT model with NMT model and study the impact of the
monolingual data used for BERT training on the final translation quality. We
use WMT-14 English-German, IWSLT15 English-German and IWSLT14 English-Russian
datasets for these experiments. In addition to standard task test set
evaluation, we perform evaluation on out-of-domain test sets and noise injected
test sets, in order to assess how BERT pretrained representations affect model
robustness.
