Can the state of relevant neurons in a deep neural networks serve as
  indicators for detecting adversarial attacks?
  We present a method for adversarial attack detection based on the inspection
of a sparse set of neurons. We follow the hypothesis that adversarial attacks
introduce imperceptible perturbations in the input and that these perturbations
change the state of neurons relevant for the concepts modelled by the attacked
model. Therefore, monitoring the status of these neurons would enable the
detection of adversarial attacks. Focusing on the image classification task,
our method identifies neurons that are relevant for the classes predicted by
the model. A deeper qualitative inspection of these sparse set of neurons
indicates that their state changes in the presence of adversarial samples.
Moreover, quantitative results from our empirical evaluation indicate that our
method is capable of recognizing adversarial samples, produced by
state-of-the-art attack methods, with comparable accuracy to that of
state-of-the-art detectors.
