Original or Translated? A Causal Analysis of the Impact of Translationese on Machine Translation Performance
Human-translated text displays distinct features from naturally written text
in the same language. This phenomena, known as translationese, has been argued
to confound the machine translation (MT) evaluation. Yet, we find that existing
work on translationese neglects some important factors and the conclusions are
mostly correlational but not causal. In this work, we collect CausalMT, a
dataset where the MT training data are also labeled with the human translation
directions. We inspect two critical factors, the train-test direction match
(whether the human translation directions in the training and test sets are
aligned), and data-model direction match (whether the model learns in the same
direction as the human translation direction in the dataset). We show that
these two factors have a large causal effect on the MT performance, in addition
to the test-model direction mismatch highlighted by existing work on the impact
of translationese. In light of our findings, we provide a set of suggestions
for MT training and evaluation. Our code and data are at
https://github.com/EdisonNi-hku/CausalMT