Rank the Episodes: A Simple Approach for Exploration in
  Procedurally-Generated Environments
  Exploration under sparse reward is a long-standing challenge of model-free
reinforcement learning. The state-of-the-art methods address this challenge by
introducing intrinsic rewards to encourage exploration in novel states or
uncertain environment dynamics. Unfortunately, methods based on intrinsic
rewards often fall short in procedurally-generated environments, where a
different environment is generated in each episode so that the agent is not
likely to visit the same state more than once. Motivated by how humans
distinguish good exploration behaviors by looking into the entire episode, we
introduce RAPID, a simple yet effective episode-level exploration method for
procedurally-generated environments. RAPID regards each episode as a whole and
gives an episodic exploration score from both per-episode and long-term views.
Those highly scored episodes are treated as good exploration behaviors and are
stored in a small ranking buffer. The agent then imitates the episodes in the
buffer to reproduce the past good exploration behaviors. We demonstrate our
method on several procedurally-generated MiniGrid environments, a
first-person-view 3D Maze navigation task from MiniWorld, and several sparse
MuJoCo tasks. The results show that RAPID significantly outperforms the
state-of-the-art intrinsic reward strategies in terms of sample efficiency and
final performance. The code is available at https://github.com/daochenzha/rapid
