Understanding Convolutional Neural Networks with Information Theory: An
  Initial Exploration
  The matrix-based Renyi's \alpha-entropy functional and its multivariate
extension were recently developed in terms of the normalized eigenspectrum of a
Hermitian matrix of the projected data in a reproducing kernel Hilbert space
(RKHS). However, the utility and possible applications of these new estimators
are rather new and mostly unknown to practitioners. In this paper, we first
show that our estimators enable straightforward measurement of information flow
in realistic convolutional neural networks (CNN) without any approximation.
Then, we introduce the partial information decomposition (PID) framework and
develop three quantities to analyze the synergy and redundancy in convolutional
layer representations. Our results validate two fundamental data processing
inequalities and reveal some fundamental properties concerning the training of
CNN.
