An Empirical Comparison of Instance Attribution Methods for NLP
  Widespread adoption of deep models has motivated a pressing need for
approaches to interpret network outputs and to facilitate model debugging.
Instance attribution methods constitute one means of accomplishing these goals
by retrieving training instances that (may have) led to a particular
prediction. Influence functions (IF; Koh and Liang 2017) provide machinery for
doing this by quantifying the effect that perturbing individual train instances
would have on a specific test prediction. However, even approximating the IF is
computationally expensive, to the degree that may be prohibitive in many cases.
Might simpler approaches (e.g., retrieving train examples most similar to a
given test point) perform comparably? In this work, we evaluate the degree to
which different potential instance attribution agree with respect to the
importance of training samples. We find that simple retrieval methods yield
training instances that differ from those identified via gradient-based methods
(such as IFs), but that nonetheless exhibit desirable characteristics similar
to more complex attribution methods. Code for all methods and experiments in
this paper is available at:
https://github.com/successar/instance_attributions_NLP.
