DIVA: Dataset Derivative of a Learning Task
  We present a method to compute the derivative of a learning task with respect
to a dataset. A learning task is a function from a training set to the
validation error, which can be represented by a trained deep neural network
(DNN). The "dataset derivative" is a linear operator, computed around the
trained model, that informs how perturbations of the weight of each training
sample affect the validation error, usually computed on a separate validation
dataset. Our method, DIVA (Differentiable Validation) hinges on a closed-form
differentiable expression of the leave-one-out cross-validation error around a
pre-trained DNN. Such expression constitutes the dataset derivative. DIVA could
be used for dataset auto-curation, for example removing samples with faulty
annotations, augmenting a dataset with additional relevant samples, or
rebalancing. More generally, DIVA can be used to optimize the dataset, along
with the parameters of the model, as part of the training process without the
need for a separate validation dataset, unlike bi-level optimization methods
customary in AutoML. To illustrate the flexibility of DIVA, we report
experiments on sample auto-curation tasks such as outlier rejection, dataset
extension, and automatic aggregation of multi-modal data.
