End-to-End Training of Both Translation Models in the Back-Translation
  Framework
  Semi-supervised learning algorithms in neural machine translation (NMT) have
significantly improved translation quality compared to the supervised learning
algorithms by using additional monolingual corpora. Among them,
back-translation is a theoretically well-structured and cutting-edge method.
Given two pre-trained NMT models between source and target languages, one
translates a monolingual sentence as a latent sentence, and the other
reconstructs the monolingual input sentence given the latent sentence.
Therefore, previous works tried to apply the variational auto-encoder's (VAE)
training framework to the back-translation framework. However, the discrete
property of the latent sentence made it impossible to use backpropagation in
the framework. This paper proposes a categorical reparameterization trick that
generates a differentiable sentence, with which we practically implement the
VAE's training framework for the back-translation and train it by end-to-end
backpropagation. In addition, we propose several regularization techniques that
are especially advantageous to this framework. In our experiments, we
demonstrate that our method makes backpropagation available through the latent
sentences and improves the BLEU scores on the datasets of the WMT18 translation
task.
