Requirements for Developing Robust Neural Networks
  Validation accuracy is a necessary, but not sufficient, measure of a neural
network classifier's quality. High validation accuracy during development does
not guarantee that a model is free of serious flaws, such as vulnerability to
adversarial attacks or a tendency to misclassify (with high confidence) data it
was not trained on. The model may also be incomprehensible to a human or base
its decisions on unreasonable criteria. These problems, which are not unique to
classifiers, have been the focus of a substantial amount of recent research.
However, they are not prioritized during model development, which almost always
optimizes on validation accuracy to the exclusion of everything else. The
product of this approach is likely to fail in unexpected ways outside of the
training environment. We believe that, in addition to validation accuracy, the
model development process must give added weight to other performance metrics
such as explainability, resistance to adversarial attacks, and overconfidence
on out-of-distribution data.
