Understanding Autoencoders with Information Theoretic Concepts
  Despite their great success in practical applications, there is still a lack
of theoretical and systematic methods to analyze deep neural networks. In this
paper, we illustrate an advanced information theoretic methodology to
understand the dynamics of learning and the design of autoencoders, a special
type of deep learning architectures that resembles a communication channel. By
generalizing the information plane to any cost function, and inspecting the
roles and dynamics of different layers using layer-wise information quantities,
we emphasize the role that mutual information plays in quantifying learning
from data. We further suggest and also experimentally validate, for mean square
error training, three fundamental properties regarding the layer-wise flow of
information and intrinsic dimensionality of the bottleneck layer, using
respectively the data processing inequality and the identification of a
bifurcation point in the information plane that is controlled by the given
data. Our observations have a direct impact on the optimal design of
autoencoders, the design of alternative feedforward training methods, and even
in the problem of generalization.
