On Information Plane Analyses of Neural Network Classifiers -- A Review
  We review the current literature concerned with information plane analyses of
neural network classifiers. While the underlying information bottleneck theory
and the claim that information-theoretic compression is causally linked to
generalization are plausible, empirical evidence was found to be both
supporting and conflicting. We review this evidence together with a detailed
analysis of how the respective information quantities were estimated. Our
survey suggests that compression visualized in information planes is not
necessarily information-theoretic, but is rather often compatible with
geometric compression of the latent representations. This insight gives the
information plane a renewed justification.
  Aside from this, we shed light on the problem of estimating mutual
information in deterministic neural networks and its consequences.
Specifically, we argue that even in feed-forward neural networks the data
processing inequality need not hold for estimates of mutual information.
Similarly, while a fitting phase, in which the mutual information between the
latent representation and the target increases, is necessary (but not
sufficient) for good classification performance, depending on the specifics of
mutual information estimation such a fitting phase need not be visible in the
information plane.
