Neuron ranking -- an informed way to condense convolutional neural
  networks architecture
  Convolutional neural networks (CNNs) in recent years have made a dramatic
impact in science, technology and industry, yet the theoretical mechanism of
CNN architecture design remains surprisingly vague. The CNN neurons, including
its distinctive element, convolutional filters, are known to be learnable
features, yet their individual role in producing the output is rather unclear.
The thesis of this work is that not all neurons are equally important and some
of them contain more useful information to perform a given task . Consequently,
we quantify the significance of each filter and rank its importance in
describing input to produce the desired output. This work presents two
different methods: (1) a game theoretical approach based on Shapley value which
computes the marginal contribution of each filter; and (2) a probabilistic
approach based on what-we-call, the Importance switch using variational
inference. Strikingly, these two vastly different methods produce similar
experimental results, confirming the general theory that some of the filters
are inherently more important that the others. The learned ranks can be readily
useable for network compression and interpretability.
