Going Beyond Saliency Maps: Training Deep Models to Interpret Deep
  Models
  Interpretability is a critical factor in applying complex deep learning
models to advance the understanding of brain disorders in neuroimaging studies.
To interpret the decision process of a trained classifier, existing techniques
typically rely on saliency maps to quantify the voxel-wise or feature-level
importance for classification through partial derivatives. Despite providing
some level of localization, these maps are not human-understandable from the
neuroscience perspective as they do not inform the specific meaning of the
alteration linked to the brain disorder. Inspired by the image-to-image
translation scheme, we propose to train simulator networks that can warp a
given image to inject or remove patterns of the disease. These networks are
trained such that the classifier produces consistently increased or decreased
prediction logits for the simulated images. Moreover, we propose to couple all
the simulators into a unified model based on conditional convolution. We
applied our approach to interpreting classifiers trained on a synthetic dataset
and two neuroimaging datasets to visualize the effect of the Alzheimer's
disease and alcohol use disorder. Compared to the saliency maps generated by
baseline approaches, our simulations and visualizations based on the Jacobian
determinants of the warping field reveal meaningful and understandable patterns
related to the diseases.
