When Does Translation Require Context? A Data-driven, Multilingual
  Exploration
  Although proper handling of discourse phenomena significantly contributes to
the quality of machine translation (MT), common translation quality metrics do
not adequately capture them. Recent works in context-aware MT attempt to target
a small set of these phenomena during evaluation. In this paper, we propose a
new metric, P-CXMI, which allows us to identify translations that require
context systematically and confirm the difficulty of previously studied
phenomena as well as uncover new ones that have not been addressed in previous
work. We then develop the Multilingual Discourse-Aware (MuDA) benchmark, a
series of taggers for these phenomena in 14 different language pairs, which we
use to evaluate context-aware MT. We find that state-of-the-art context-aware
MT models find marginal improvements over context-agnostic models on our
benchmark, which suggests current models do not handle these ambiguities
effectively. We release code and data to invite the MT research community to
increase efforts on context-aware translation on discourse phenomena and
languages that are currently overlooked.
