Do Context-Aware Translation Models Pay the Right Attention?
  Context-aware machine translation models are designed to leverage contextual
information, but often fail to do so. As a result, they inaccurately
disambiguate pronouns and polysemous words that require context for resolution.
In this paper, we ask several questions: What contexts do human translators use
to resolve ambiguous words? Are models paying large amounts of attention to the
same context? What if we explicitly train them to do so? To answer these
questions, we introduce SCAT (Supporting Context for Ambiguous Translations), a
new English-French dataset comprising supporting context words for 14K
translations that professional translators found useful for pronoun
disambiguation. Using SCAT, we perform an in-depth analysis of the context used
to disambiguate, examining positional and lexical characteristics of the
supporting words. Furthermore, we measure the degree of alignment between the
model's attention scores and the supporting context from SCAT, and apply a
guided attention strategy to encourage agreement between the two.
