Scrambled Translation Problem: A Problem of Denoising UNMT
  In this paper, we identify an interesting kind of error in the output of
Unsupervised Neural Machine Translation (UNMT) systems like
\textit{Undreamt}(footnote). We refer to this error type as \textit{Scrambled
Translation problem}. We observe that UNMT models which use \textit{word
shuffle} noise (as in case of Undreamt) can generate correct words, but fail to
stitch them together to form phrases. As a result, words of the translated
sentence look \textit{scrambled}, resulting in decreased BLEU. We hypothesise
that the reason behind \textit{scrambled translation problem} is 'shuffling
noise' which is introduced in every input sentence as a denoising strategy. To
test our hypothesis, we experiment by retraining UNMT models with a simple
\textit{retraining} strategy. We stop the training of the Denoising UNMT model
after a pre-decided number of iterations and resume the training for the
remaining iterations -- which number is also pre-decided -- using original
sentence as input without adding any noise. Our proposed solution achieves
significant performance improvement UNMT models that train conventionally. We
demonstrate these performance gains on four language pairs, \textit{viz.},
English-French, English-German, English-Spanish, Hindi-Punjabi. Our qualitative
and quantitative analysis shows that the retraining strategy helps achieve
better alignment as observed by attention heatmap and better phrasal
translation, leading to statistically significant improvement in BLEU scores.
