Testing Deep Neural Networks
  Deep neural networks (DNNs) have a wide range of applications, and software
employing them must be thoroughly tested, especially in safety-critical
domains. However, traditional software test coverage metrics cannot be applied
directly to DNNs. In this paper, inspired by the MC/DC coverage criterion, we
propose a family of four novel test criteria that are tailored to structural
features of DNNs and their semantics. We validate the criteria by demonstrating
that the generated test inputs guided via our proposed coverage criteria are
able to capture undesired behaviours in a DNN. Test cases are generated using a
symbolic approach and a gradient-based heuristic search. By comparing them with
existing methods, we show that our criteria achieve a balance between their
ability to find bugs (proxied using adversarial examples) and the computational
cost of test case generation. Our experiments are conducted on state-of-the-art
DNNs obtained using popular open source datasets, including MNIST, CIFAR-10 and
ImageNet.
