Variational Information Maximisation for Intrinsically Motivated
  Reinforcement Learning
  The mutual information is a core statistical quantity that has applications
in all areas of machine learning, whether this is in training of density models
over multiple data modalities, in maximising the efficiency of noisy
transmission channels, or when learning behaviour policies for exploration by
artificial agents. Most learning algorithms that involve optimisation of the
mutual information rely on the Blahut-Arimoto algorithm --- an enumerative
algorithm with exponential complexity that is not suitable for modern machine
learning applications. This paper provides a new approach for scalable
optimisation of the mutual information by merging techniques from variational
inference and deep learning. We develop our approach by focusing on the problem
of intrinsically-motivated learning, where the mutual information forms the
definition of a well-known internal drive known as empowerment. Using a
variational lower bound on the mutual information, combined with convolutional
networks for handling visual input streams, we develop a stochastic
optimisation algorithm that allows for scalable information maximisation and
empowerment-based reasoning directly from pixels to actions.
