Computing the Information Content of Trained Neural Networks
  How much information does a learning algorithm extract from the training data
and store in a neural network's weights? Too much, and the network would
overfit to the training data. Too little, and the network would not fit to
anything at all. Na\"ively, the amount of information the network stores should
scale in proportion to the number of trainable weights. This raises the
question: how can neural networks with vastly more weights than training data
still generalise? A simple resolution to this conundrum is that the number of
weights is usually a bad proxy for the actual amount of information stored. For
instance, typical weight vectors may be highly compressible. Then another
question occurs: is it possible to compute the actual amount of information
stored? This paper derives both a consistent estimator and a closed-form upper
bound on the information content of infinitely wide neural networks. The
derivation is based on an identification between neural information content and
the negative log probability of a Gaussian orthant. This identification yields
bounds that analytically control the generalisation behaviour of the entire
solution space of infinitely wide networks. The bounds have a simple dependence
on both the network architecture and the training data. Corroborating the
findings of Valle-P\'erez et al. (2019), who conducted a similar analysis using
approximate Gaussian integration techniques, the bounds are found to be both
non-vacuous and correlated with the empirical generalisation behaviour at
finite width.
