The Variational InfoMax Learning Objective
  Bayesian Inference and Information Bottleneck are the two most popular
objectives for neural networks, but they can be optimised only via a
variational lower bound: the Variational Information Bottleneck (VIB). In this
manuscript we show that the two objectives are actually equivalent to the
InfoMax: maximise the information between the data and the labels. The InfoMax
representation of the two objectives is not relevant only per se, since it
helps to understand the role of the network capacity, but also because it
allows us to derive a variational objective, the Variational InfoMax (VIM),
that maximises them directly without resorting to any lower bound. The
theoretical improvement of VIM over VIB is highlighted by the computational
experiments, where the model trained by VIM improves the VIB model in three
different tasks: accuracy, robustness to noise and representation quality.
