Skew-Fit: State-Covering Self-Supervised Reinforcement Learning
  Autonomous agents that must exhibit flexible and broad capabilities will need
to be equipped with large repertoires of skills. Defining each skill with a
manually-designed reward function limits this repertoire and imposes a manual
engineering burden. Self-supervised agents that set their own goals can
automate this process, but designing appropriate goal setting objectives can be
difficult, and often involves heuristic design decisions. In this paper, we
propose a formal exploration objective for goal-reaching policies that
maximizes state coverage. We show that this objective is equivalent to
maximizing goal reaching performance together with the entropy of the goal
distribution, where goals correspond to full state observations. To instantiate
this principle, we present an algorithm called Skew-Fit for learning a
maximum-entropy goal distributions. We prove that, under regularity conditions,
Skew-Fit converges to a uniform distribution over the set of valid states, even
when we do not know this set beforehand. Our experiments show that combining
Skew-Fit for learning goal distributions with existing goal-reaching methods
outperforms a variety of prior methods on open-sourced visual goal-reaching
tasks. Moreover, we demonstrate that Skew-Fit enables a real-world robot to
learn to open a door, entirely from scratch, from pixels, and without any
manually-designed reward function.
