Evaluating Explanation Methods for Deep Learning in Security
  Deep learning is increasingly used as a building block of security systems.
Unfortunately, neural networks are hard to interpret and typically opaque to
the practitioner. The machine learning community has started to address this
problem by developing methods for explaining the predictions of neural
networks. While several of these approaches have been successfully applied in
the area of computer vision, their application in security has received little
attention so far. It is an open question which explanation methods are
appropriate for computer security and what requirements they need to satisfy.
In this paper, we introduce criteria for comparing and evaluating explanation
methods in the context of computer security. These cover general properties,
such as the accuracy of explanations, as well as security-focused aspects, such
as the completeness, efficiency, and robustness. Based on our criteria, we
investigate six popular explanation methods and assess their utility in
security systems for malware detection and vulnerability discovery. We observe
significant differences between the methods and build on these to derive
general recommendations for selecting and applying explanation methods in
computer security.
