The Manifold Hypothesis for Gradient-Based Explanations
  When do gradient-based explanation algorithms provide meaningful
explanations? We propose a necessary criterion: their feature attributions need
to be aligned with the tangent space of the data manifold. To provide evidence
for this hypothesis, we introduce a framework based on variational autoencoders
that allows to estimate and generate image manifolds. Through experiments
across a range of different datasets -- MNIST, EMNIST, CIFAR10, X-ray pneumonia
and Diabetic Retinopathy detection -- we demonstrate that the more a feature
attribution is aligned with the tangent space of the data, the more structured
and explanatory it tends to be. In particular, the attributions provided by
popular post-hoc methods such as Integrated Gradients, SmoothGrad and Input
$\times$ Gradient tend to be more strongly aligned with the data manifold than
the raw gradient. As a consequence, we suggest that explanation algorithms
should actively strive to align their explanations with the data manifold. In
part, this can be achieved by adversarial training, which leads to better
alignment across all datasets. Some form of adjustment to the model
architecture or training algorithm is necessary, since we show that
generalization of neural networks alone does not imply the alignment of model
gradients with the data manifold.
