Sparse Interventions in Language Models with Differentiable Masking
  There has been a lot of interest in understanding what information is
captured by hidden representations of language models (LMs). Typically,
interpretation methods i) do not guarantee that the model actually uses the
encoded information, and ii) do not discover small subsets of neurons
responsible for a considered phenomenon. Inspired by causal mediation analysis,
we propose a method that discovers within a neural LM a small subset of neurons
responsible for a particular linguistic phenomenon, i.e., subsets causing a
change in the corresponding token emission probabilities. We use a
differentiable relaxation to approximately search through the combinatorial
space. An $L_0$ regularization term ensures that the search converges to
discrete and sparse solutions. We apply our method to analyze subject-verb
number agreement and gender bias detection in LSTMs. We observe that it is fast
and finds better solutions than the alternative (REINFORCE). Our experiments
confirm that each of these phenomenons is mediated through a small subset of
neurons that do not play any other discernible role.
