Explore, Discover and Learn: Unsupervised Discovery of State-Covering
  Skills
  Acquiring abilities in the absence of a task-oriented reward function is at
the frontier of reinforcement learning research. This problem has been studied
through the lens of empowerment, which draws a connection between option
discovery and information theory. Information-theoretic skill discovery methods
have garnered much interest from the community, but little research has been
conducted in understanding their limitations. Through theoretical analysis and
empirical evidence, we show that existing algorithms suffer from a common
limitation -- they discover options that provide a poor coverage of the state
space. In light of this, we propose 'Explore, Discover and Learn' (EDL), an
alternative approach to information-theoretic skill discovery. Crucially, EDL
optimizes the same information-theoretic objective derived from the empowerment
literature, but addresses the optimization problem using different machinery.
We perform an extensive evaluation of skill discovery methods on controlled
environments and show that EDL offers significant advantages, such as
overcoming the coverage problem, reducing the dependence of learned skills on
the initial state, and allowing the user to define a prior over which behaviors
should be learned. Code is publicly available at
https://github.com/victorcampos7/edl.
