Do Multilingual Neural Machine Translation Models Contain Language Pair
  Specific Attention Heads?
  Recent studies on the analysis of the multilingual representations focus on
identifying whether there is an emergence of language-independent
representations, or whether a multilingual model partitions its weights among
different languages. While most of such work has been conducted in a
"black-box" manner, this paper aims to analyze individual components of a
multilingual neural translation (NMT) model. In particular, we look at the
encoder self-attention and encoder-decoder attention heads (in a many-to-one
NMT model) that are more specific to the translation of a certain language pair
than others by (1) employing metrics that quantify some aspects of the
attention weights such as "variance" or "confidence", and (2) systematically
ranking the importance of attention heads with respect to translation quality.
Experimental results show that surprisingly, the set of most important
attention heads are very similar across the language pairs and that it is
possible to remove nearly one-third of the less important heads without hurting
the translation quality greatly.
