Understanding Neural Networks and Individual Neuron Importance via
  Information-Ordered Cumulative Ablation
  In this work, we investigate the use of three information-theoretic
quantities -- entropy, mutual information with the class variable, and a class
selectivity measure based on Kullback-Leibler divergence -- to understand and
study the behavior of already trained fully-connected feed-forward neural
networks. We analyze the connection between these information-theoretic
quantities and classification performance on the test set by cumulatively
ablating neurons in networks trained on MNIST, FashionMNIST, and CIFAR-10. Our
results parallel those recently published by Morcos et al., indicating that
class selectivity is not a good indicator for classification performance.
However, looking at individual layers separately, both mutual information and
class selectivity are positively correlated with classification performance, at
least for networks with ReLU activation functions. We provide explanations for
this phenomenon and conclude that it is ill-advised to compare the proposed
information-theoretic quantities across layers. Furthermore, we show that
cumulative ablation of neurons with ascending or descending
information-theoretic quantities can be used to formulate hypotheses regarding
the joint behavior of multiple neurons, such as redundancy and synergy, with
comparably low computational cost. We also draw connections to the information
bottleneck theory for neural networks.
