Test Selection for Deep Learning Systems
  Testing of deep learning models is challenging due to the excessive number
and complexity of computations involved. As a result, test data selection is
performed manually and in an ad hoc way. This raises the question of how we can
automatically select candidate test data to test deep learning models. Recent
research has focused on adapting test selection metrics from code-based
software testing (such as coverage) to deep learning. However, deep learning
models have different attributes from code such as spread of computations
across the entire network reflecting training data properties, balance of
neuron weights and redundancy (use of many more neurons than needed). Such
differences make code-based metrics inappropriate to select data that can
challenge the models (can trigger misclassification). We thus propose a set of
test selection metrics based on the notion of model uncertainty (model
confidence on specific inputs). Intuitively, the more uncertain we are about a
candidate sample, the more likely it is that this sample triggers a
misclassification. Similarly, the samples for which we are the most uncertain,
are the most informative and should be used to improve the model by retraining.
We evaluate these metrics on two widely-used image classification problems
involving real and artificial (adversarial) data. We show that
uncertainty-based metrics have a strong ability to select data that are
misclassified and lead to major improvement in classification accuracy during
retraining: up to 80% more gain than random selection and other
state-of-the-art metrics on one dataset and up to 29% on the other.
