Where do Models go Wrong? Parameter-Space Saliency Maps for
  Explainability
  Conventional saliency maps highlight input features to which neural network
predictions are highly sensitive. We take a different approach to saliency, in
which we identify and analyze the network parameters, rather than inputs, which
are responsible for erroneous decisions. We find that samples which cause
similar parameters to malfunction are semantically similar. We also show that
pruning the most salient parameters for a wrongly classified sample often
improves model behavior. Furthermore, fine-tuning a small number of the most
salient parameters on a single sample results in error correction on other
samples that are misclassified for similar reasons. Based on our parameter
saliency method, we also introduce an input-space saliency technique that
reveals how image features cause specific network components to malfunction.
Further, we rigorously validate the meaningfulness of our saliency maps on both
the dataset and case-study levels.
