Entropy and mutual information in models of deep neural networks
  We examine a class of deep learning models with a tractable method to compute
information-theoretic quantities. Our contributions are three-fold: (i) We show
how entropies and mutual informations can be derived from heuristic statistical
physics methods, under the assumption that weight matrices are independent and
orthogonally-invariant. (ii) We extend particular cases in which this result is
known to be rigorously exact by providing a proof for two-layers networks with
Gaussian random weights, using the recently introduced adaptive interpolation
method. (iii) We propose an experiment framework with generative models of
synthetic datasets, on which we train deep neural networks with a weight
constraint designed so that the assumption in (i) is verified during learning.
We study the behavior of entropies and mutual informations throughout learning
and conclude that, in the proposed setting, the relationship between
compression and generalization remains elusive.
