Dataset Pruning: Reducing Training Data by Examining Generalization Influence
The great success of deep learning heavily relies on increasingly larger
training data, which comes at a price of huge computational and infrastructural
costs. This poses crucial questions that, do all training data contribute to
model's performance? How much does each individual training sample or a
sub-training-set affect the model's generalization, and how to construct a
smallest subset from the entire training data as a proxy training set without
significantly sacrificing the model's performance? To answer these, we propose
dataset pruning, an optimization-based sample selection method that can (1)
examine the influence of removing a particular set of training samples on
model's generalization ability with theoretical guarantee, and (2) construct a
smallest subset of training data that yields strictly constrained
generalization gap. The empirically observed generalization gap of dataset
pruning is substantially consistent with our theoretical expectations.
Furthermore, the proposed method prunes 40% training examples on the CIFAR-10
dataset, halves the convergence time with only 1.3% test accuracy decrease,
which is superior to previous score-based sample selection methods.