Unpacking Information Bottlenecks: Unifying Information-Theoretic
  Objectives in Deep Learning
  The Information Bottleneck principle offers both a mechanism to explain how
deep neural networks train and generalize, as well as a regularized objective
with which to train models. However, multiple competing objectives are proposed
in the literature, and the information-theoretic quantities used in these
objectives are difficult to compute for large deep neural networks, which in
turn limits their use as a training objective. In this work, we review these
quantities and compare and unify previously proposed objectives, which allows
us to develop surrogate objectives more friendly to optimization without
relying on cumbersome tools such as density estimation. We find that these
surrogate objectives allow us to apply the information bottleneck to modern
neural network architectures. We demonstrate our insights on MNIST, CIFAR-10
and Imagenette with modern DNN architectures (ResNets).
