Overinterpretation reveals image classification model pathologies
  Image classifiers are typically scored on their test set accuracy, but high
accuracy can mask a subtle type of model failure. We find that high scoring
convolutional neural networks (CNNs) on popular benchmarks exhibit troubling
pathologies that allow them to display high accuracy even in the absence of
semantically salient features. When a model provides a high-confidence decision
without salient supporting input features, we say the classifier has
overinterpreted its input, finding too much class-evidence in patterns that
appear nonsensical to humans. Here, we demonstrate that neural networks trained
on CIFAR-10 and ImageNet suffer from overinterpretation, and we find models on
CIFAR-10 make confident predictions even when 95% of input images are masked
and humans cannot discern salient features in the remaining pixel-subsets. We
introduce Batched Gradient SIS, a new method for discovering sufficient input
subsets for complex datasets, and use this method to show the sufficiency of
border pixels in ImageNet for training and testing. Although these patterns
portend potential model fragility in real-world deployment, they are in fact
valid statistical patterns of the benchmark that alone suffice to attain high
test accuracy. Unlike adversarial examples, overinterpretation relies upon
unmodified image pixels. We find ensembling and input dropout can each help
mitigate overinterpretation.
