Wasserstein Unsupervised Reinforcement Learning
  Unsupervised reinforcement learning aims to train agents to learn a handful
of policies or skills in environments without external reward. These
pre-trained policies can accelerate learning when endowed with external reward,
and can also be used as primitive options in hierarchical reinforcement
learning. Conventional approaches of unsupervised skill discovery feed a latent
variable to the agent and shed its empowerment on agent's behavior by mutual
information (MI) maximization. However, the policies learned by MI-based
methods cannot sufficiently explore the state space, despite they can be
successfully identified from each other. Therefore we propose a new framework
Wasserstein unsupervised reinforcement learning (WURL) where we directly
maximize the distance of state distributions induced by different policies.
Additionally, we overcome difficulties in simultaneously training N(N >2)
policies, and amortizing the overall reward to each step. Experiments show
policies learned by our approach outperform MI-based methods on the metric of
Wasserstein distance while keeping high discriminability. Furthermore, the
agents trained by WURL can sufficiently explore the state space in mazes and
MuJoCo tasks and the pre-trained policies can be applied to downstream tasks by
hierarchical learning.
