Episodic Curiosity through Reachability
  Rewards are sparse in the real world and most of today's reinforcement
learning algorithms struggle with such sparsity. One solution to this problem
is to allow the agent to create rewards for itself - thus making rewards dense
and more suitable for learning. In particular, inspired by curious behaviour in
animals, observing something novel could be rewarded with a bonus. Such bonus
is summed up with the real task reward - making it possible for RL algorithms
to learn from the combined reward. We propose a new curiosity method which uses
episodic memory to form the novelty bonus. To determine the bonus, the current
observation is compared with the observations in memory. Crucially, the
comparison is done based on how many environment steps it takes to reach the
current observation from those in memory - which incorporates rich information
about environment dynamics. This allows us to overcome the known "couch-potato"
issues of prior work - when the agent finds a way to instantly gratify itself
by exploiting actions which lead to hardly predictable consequences. We test
our approach in visually rich 3D environments in ViZDoom, DMLab and MuJoCo. In
navigational tasks from ViZDoom and DMLab, our agent outperforms the
state-of-the-art curiosity method ICM. In MuJoCo, an ant equipped with our
curiosity module learns locomotion out of the first-person-view curiosity only.
