Outside the Box: Abstraction-Based Monitoring of Neural Networks
  Neural networks have demonstrated unmatched performance in a range of
classification tasks. Despite numerous efforts of the research community,
novelty detection remains one of the significant limitations of neural
networks. The ability to identify previously unseen inputs as novel is crucial
for our understanding of the decisions made by neural networks. At runtime,
inputs not falling into any of the categories learned during training cannot be
classified correctly by the neural network. Existing approaches treat the
neural network as a black box and try to detect novel inputs based on the
confidence of the output predictions. However, neural networks are not trained
to reduce their confidence for novel inputs, which limits the effectiveness of
these approaches. We propose a framework to monitor a neural network by
observing the hidden layers. We employ a common abstraction from program
analysis - boxes - to identify novel behaviors in the monitored layers, i.e.,
inputs that cause behaviors outside the box. For each neuron, the boxes range
over the values seen in training. The framework is efficient and flexible to
achieve a desired trade-off between raising false warnings and detecting novel
inputs. We illustrate the performance and the robustness to variability in the
unknown classes on popular image-classification benchmarks.
