A Unified Bellman Optimality Principle Combining Reward Maximization and
  Empowerment
  Empowerment is an information-theoretic method that can be used to
intrinsically motivate learning agents. It attempts to maximize an agent's
control over the environment by encouraging visiting states with a large number
of reachable next states. Empowered learning has been shown to lead to complex
behaviors, without requiring an explicit reward signal. In this paper, we
investigate the use of empowerment in the presence of an extrinsic reward
signal. We hypothesize that empowerment can guide reinforcement learning (RL)
agents to find good early behavioral solutions by encouraging highly empowered
states. We propose a unified Bellman optimality principle for empowered reward
maximization. Our empowered reward maximization approach generalizes both
Bellman's optimality principle as well as recent information-theoretical
extensions to it. We prove uniqueness of the empowered values and show
convergence to the optimal solution. We then apply this idea to develop
off-policy actor-critic RL algorithms which we validate in high-dimensional
continuous robotics domains (MuJoCo). Our methods demonstrate improved initial
and competitive final performance compared to model-free state-of-the-art
techniques.
