Training Normalizing Flows with the Information Bottleneck for
  Competitive Generative Classification
  The Information Bottleneck (IB) objective uses information theory to
formulate a task-performance versus robustness trade-off. It has been
successfully applied in the standard discriminative classification setting. We
pose the question whether the IB can also be used to train generative
likelihood models such as normalizing flows. Since normalizing flows use
invertible network architectures (INNs), they are information-preserving by
construction. This seems contradictory to the idea of a bottleneck. In this
work, firstly, we develop the theory and methodology of IB-INNs, a class of
conditional normalizing flows where INNs are trained using the IB objective:
Introducing a small amount of {\em controlled} information loss allows for an
asymptotically exact formulation of the IB, while keeping the INN's generative
capabilities intact. Secondly, we investigate the properties of these models
experimentally, specifically used as generative classifiers. This model class
offers advantages such as improved uncertainty quantification and
out-of-distribution detection, but traditional generative classifier solutions
suffer considerably in classification accuracy. We find the trade-off parameter
in the IB controls a mix of generative capabilities and accuracy close to
standard classifiers. Empirically, our uncertainty estimates in this mixed
regime compare favourably to conventional generative and discriminative
classifiers.
