Learning Efficient Representation for Intrinsic Motivation
  Mutual Information between agent Actions and environment States (MIAS)
quantifies the influence of agent on its environment. Recently, it was found
that the maximization of MIAS can be used as an intrinsic motivation for
artificial agents. In literature, the term empowerment is used to represent the
maximum of MIAS at a certain state. While empowerment has been shown to solve a
broad range of reinforcement learning problems, its calculation in arbitrary
dynamics is a challenging problem because it relies on the estimation of mutual
information. Existing approaches, which rely on sampling, are limited to low
dimensional spaces, because high-confidence distribution-free lower bounds for
mutual information require exponential number of samples. In this work, we
develop a novel approach for the estimation of empowerment in unknown dynamics
from visual observation only, without the need to sample for MIAS. The core
idea is to represent the relation between action sequences and future states
using a stochastic dynamic model in latent space with a specific form. This
allows us to efficiently compute empowerment with the "Water-Filling" algorithm
from information theory. We construct this embedding with deep neural networks
trained on a sophisticated objective function. Our experimental results show
that the designed embedding preserves information-theoretic properties of the
original dynamics.
