PAC-Bayes Information Bottleneck
  Understanding the source of the superior generalization ability of NNs
remains one of the most important problems in ML research. There have been a
series of theoretical works trying to derive non-vacuous bounds for NNs.
Recently, the compression of information stored in weights (IIW) is proved to
play a key role in NNs generalization based on the PAC-Bayes theorem. However,
no solution of IIW has ever been provided, which builds a barrier for further
investigation of the IIW's property and its potential in practical deep
learning. In this paper, we propose an algorithm for the efficient
approximation of IIW. Then, we build an IIW-based information bottleneck on the
trade-off between accuracy and information complexity of NNs, namely PIB. From
PIB, we can empirically identify the fitting to compressing phase transition
during NNs' training and the concrete connection between the IIW compression
and the generalization. Besides, we verify that IIW is able to explain NNs in
broad cases, e.g., varying batch sizes, over-parameterization, and noisy
labels. Moreover, we propose an MCMC-based algorithm to sample from the optimal
weight posterior characterized by PIB, which fulfills the potential of IIW in
enhancing NNs in practice.
