Supporting DNN Safety Analysis and Retraining through Heatmap-based
  Unsupervised Learning
  Deep neural networks (DNNs) are increasingly important in safety-critical
systems, for example in their perception layer to analyze images.
Unfortunately, there is a lack of methods to ensure the functional safety of
DNN-based components. We observe three major challenges with existing practices
regarding DNNs in safety-critical systems: (1) scenarios that are
underrepresented in the test set may lead to serious safety violation risks,
but may, however, remain unnoticed; (2) characterizing such high-risk scenarios
is critical for safety analysis; (3) retraining DNNs to address these risks is
poorly supported when causes of violations are difficult to determine. To
address these problems in the context of DNNs analyzing images, we propose
HUDD, an approach that automatically supports the identification of root causes
for DNN errors. HUDD identifies root causes by applying a clustering algorithm
to heatmaps capturing the relevance of every DNN neuron on the DNN outcome.
Also, HUDD retrains DNNs with images that are automatically selected based on
their relatedness to the identified image clusters. We evaluated HUDD with DNNs
from the automotive domain. HUDD was able to identify all the distinct root
causes of DNN errors, thus supporting safety analysis. Also, our retraining
approach has shown to be more effective at improving DNN accuracy than existing
approaches.
