Variational Empowerment as Representation Learning for Goal-Based
  Reinforcement Learning
  Learning to reach goal states and learning diverse skills through mutual
information (MI) maximization have been proposed as principled frameworks for
self-supervised reinforcement learning, allowing agents to acquire broadly
applicable multitask policies with minimal reward engineering. Starting from a
simple observation that the standard goal-conditioned RL (GCRL) is encapsulated
by the optimization objective of variational empowerment, we discuss how GCRL
and MI-based RL can be generalized into a single family of methods, which we
name variational GCRL (VGCRL), interpreting variational MI maximization, or
variational empowerment, as representation learning methods that acquire
functionally-aware state representations for goal reaching. This novel
perspective allows us to: (1) derive simple but unexplored variants of GCRL to
study how adding small representation capacity can already expand its
capabilities; (2) investigate how discriminator function capacity and
smoothness determine the quality of discovered skills, or latent goals, through
modifying latent dimensionality and applying spectral normalization; (3) adapt
techniques such as hindsight experience replay (HER) from GCRL to MI-based RL;
and lastly, (4) propose a novel evaluation metric, named latent goal reaching
(LGR), for comparing empowerment algorithms with different choices of latent
dimensionality and discriminator parameterization. Through principled
mathematical derivations and careful experimental studies, our work lays a
novel foundation from which to evaluate, analyze, and develop representation
learning techniques in goal-based RL.
