Hold me tight! Influence of discriminative features on deep network
  boundaries
  Important insights towards the explainability of neural networks reside in
the characteristics of their decision boundaries. In this work, we borrow tools
from the field of adversarial robustness, and propose a new perspective that
relates dataset features to the distance of samples to the decision boundary.
This enables us to carefully tweak the position of the training samples and
measure the induced changes on the boundaries of CNNs trained on large-scale
vision datasets. We use this framework to reveal some intriguing properties of
CNNs. Specifically, we rigorously confirm that neural networks exhibit a high
invariance to non-discriminative features, and show that the decision
boundaries of a DNN can only exist as long as the classifier is trained with
some features that hold them together. Finally, we show that the construction
of the decision boundary is extremely sensitive to small perturbations of the
training samples, and that changes in certain directions can lead to sudden
invariances in the orthogonal ones. This is precisely the mechanism that
adversarial training uses to achieve robustness.
