FastIF: Scalable Influence Functions for Efficient Model Interpretation
  and Debugging
  Influence functions approximate the "influences" of training data-points for
test predictions and have a wide variety of applications. Despite the
popularity, their computational cost does not scale well with model and
training data size. We present FastIF, a set of simple modifications to
influence functions that significantly improves their run-time. We use
k-Nearest Neighbors (kNN) to narrow the search space down to a subset of good
candidate data points, identify the configurations that best balance the
speed-quality trade-off in estimating the inverse Hessian-vector product, and
introduce a fast parallel variant. Our proposed method achieves about 80X
speedup while being highly correlated with the original influence values. With
the availability of the fast influence functions, we demonstrate their
usefulness in four applications. First, we examine whether influential
data-points can "explain" test time behavior using the framework of
simulatability. Second, we visualize the influence interactions between
training and test data-points. Third, we show that we can correct model errors
by additional fine-tuning on certain influential data-points, improving the
accuracy of a trained MultiNLI model by 2.5% on the HANS dataset. Finally, we
experiment with a similar setup but fine-tuning on datapoints not seen during
training, improving the model accuracy by 2.8% and 1.7% on HANS and ANLI
datasets respectively. Overall, our fast influence functions can be efficiently
applied to large models and datasets, and our experiments demonstrate the
potential of influence functions in model interpretation and correcting model
errors. Code is available at
https://github.com/salesforce/fast-influence-functions
