Estimating Training Data Influence by Tracing Gradient Descent
  We introduce a method called TracIn that computes the influence of a training
example on a prediction made by the model. The idea is to trace how the loss on
the test point changes during the training process whenever the training
example of interest was utilized. We provide a scalable implementation of
TracIn via: (a) a first-order gradient approximation to the exact computation,
(b) saved checkpoints of standard training procedures, and (c) cherry-picking
layers of a deep neural network. In contrast with previously proposed methods,
TracIn is simple to implement; all it needs is the ability to work with
gradients, checkpoints, and loss functions. The method is general. It applies
to any machine learning model trained using stochastic gradient descent or a
variant of it, agnostic of architecture, domain and task. We expect the method
to be widely useful within processes that study and improve training data.
