Towards Visual Explanations for Convolutional Neural Networks via Input
  Resampling
  The predictive power of neural networks often costs model interpretability.
Several techniques have been developed for explaining model outputs in terms of
input features; however, it is difficult to translate such interpretations into
actionable insight. Here, we propose a framework to analyze predictions in
terms of the model's internal features by inspecting information flow through
the network. Given a trained network and a test image, we select neurons by two
metrics, both measured over a set of images created by perturbations to the
input image: (1) magnitude of the correlation between the neuron activation and
the network output and (2) precision of the neuron activation. We show that the
former metric selects neurons that exert large influence over the network
output while the latter metric selects neurons that activate on generalizable
features. By comparing the sets of neurons selected by these two metrics, our
framework suggests a way to investigate the internal attention mechanisms of
convolutional neural networks.
