Abstraction Mechanisms Predict Generalization in Deep Neural Networks
  A longstanding problem for Deep Neural Networks (DNNs) is understanding their
puzzling ability to generalize well. We approach this problem through the
unconventional angle of \textit{cognitive abstraction mechanisms}, drawing
inspiration from recent neuroscience work, allowing us to define the Cognitive
Neural Activation metric (CNA) for DNNs, which is the correlation between
information complexity (entropy) of given input and the concentration of higher
activation values in deeper layers of the network. The CNA is highly predictive
of generalization ability, outperforming norm-and-margin-based generalization
metrics on an extensive evaluation of over 100 dataset-and-network-architecture
combinations, especially in cases where additive noise is present and/or
training labels are corrupted. These strong empirical results show the
usefulness of CNA as a generalization metric, and encourage further research on
the connection between information complexity and representations in the deeper
layers of networks in order to better understand the generalization
capabilities of DNNs.
