HOMRS: High Order Metamorphic Relations Selector for Deep Neural
  Networks
  Deep Neural Networks (DNN) applications are increasingly becoming a part of
our everyday life, from medical applications to autonomous cars. Traditional
validation of DNN relies on accuracy measures, however, the existence of
adversarial examples has highlighted the limitations of these accuracy
measures, raising concerns especially when DNN are integrated into
safety-critical systems.
  In this paper, we present HOMRS, an approach to boost metamorphic testing by
automatically building a small optimized set of high order metamorphic
relations from an initial set of elementary metamorphic relations. HOMRS'
backbone is a multi-objective search; it exploits ideas drawn from traditional
systems testing such as code coverage, test case, path diversity as well as
input validation.
  We applied HOMRS to MNIST/LeNet and SVHN/VGG and we report evidence that it
builds a small but effective set of high-order transformations that generalize
well to the input data distribution. Moreover, comparing to similar generation
technique such as DeepXplore, we show that our distribution-based approach is
more effective, generating valid transformations from an uncertainty
quantification point of view, while requiring less computation time by
leveraging the generalization ability of the approach.
