Saliency is a Possible Red Herring When Diagnosing Poor Generalization
  Poor generalization is one symptom of models that learn to predict target
variables using spuriously-correlated image features present only in the
training distribution instead of the true image features that denote a class.
It is often thought that this can be diagnosed visually using attribution (aka
saliency) maps. We study if this assumption is correct. In some prediction
tasks, such as for medical images, one may have some images with masks drawn by
a human expert, indicating a region of the image containing relevant
information to make the prediction. We study multiple methods that take
advantage of such auxiliary labels, by training networks to ignore distracting
features which may be found outside of the region of interest. This mask
information is only used during training and has an impact on generalization
accuracy depending on the severity of the shift between the training and test
distributions. Surprisingly, while these methods improve generalization
performance in the presence of a covariate shift, there is no strong
correspondence between the correction of attribution towards the features a
human expert has labelled as important and generalization performance. These
results suggest that the root cause of poor generalization may not always be
spatially defined, and raise questions about the utility of masks as
"attribution priors" as well as saliency maps for explainable predictions.
