Information Bottleneck Theory on Convolutional Neural Networks
  Recent years, many researches attempt to open the black box of deep neural
networks and propose a various of theories to understand it. Among them,
Information Bottleneck (IB) theory claims that there are two distinct phases
consisting of fitting phase and compression phase in the course of training.
This statement attracts many attentions since its success in explaining the
inner behavior of feedforward neural networks. In this paper, we employ IB
theory to understand the dynamic behavior of convolutional neural networks
(CNNs) and investigate how the fundamental features such as convolutional layer
width, kernel size, network depth, pooling layers and multi-fully connected
layer have impact on the performance of CNNs. In particular, through a series
of experimental analysis on benchmark of MNIST and Fashion-MNIST, we
demonstrate that the compression phase is not observed in all these cases. This
shows us the CNNs have a rather complicated behavior than feedforward neural
networks.
