Interpreting Language Models with Contrastive Explanations
  Model interpretability methods are often used to explain NLP model decisions
on tasks such as text classification, where the output space is relatively
small. However, when applied to language generation, where the output space
often consists of tens of thousands of tokens, these methods are unable to
provide informative explanations. Language models must consider various
features to predict a token, such as its part of speech, number, tense, or
semantics. Existing explanation methods conflate evidence for all these
features into a single explanation, which is less interpretable for human
understanding.
  To disentangle the different decisions in language modeling, we focus on
explaining language models contrastively: we look for salient input tokens that
explain why the model predicted one token instead of another. We demonstrate
that contrastive explanations are quantifiably better than non-contrastive
explanations in verifying major grammatical phenomena, and that they
significantly improve contrastive model simulatability for human observers. We
also identify groups of contrastive decisions where the model uses similar
evidence, and we are able to characterize what input tokens models use during
various language generation decisions.
