Scheduled Intrinsic Drive: A Hierarchical Take on Intrinsically
  Motivated Exploration
  Exploration in sparse reward reinforcement learning remains an open
challenge. Many state-of-the-art methods use intrinsic motivation to complement
the sparse extrinsic reward signal, giving the agent more opportunities to
receive feedback during exploration. Commonly these signals are added as bonus
rewards, which results in a mixture policy that neither conducts exploration
nor task fulfillment resolutely. In this paper, we instead learn separate
intrinsic and extrinsic task policies and schedule between these different
drives to accelerate exploration and stabilize learning. Moreover, we introduce
a new type of intrinsic reward denoted as successor feature control (SFC),
which is general and not task-specific. It takes into account statistics over
complete trajectories and thus differs from previous methods that only use
local information to evaluate intrinsic motivation. We evaluate our proposed
scheduled intrinsic drive (SID) agent using three different environments with
pure visual inputs: VizDoom, DeepMind Lab and DeepMind Control Suite. The
results show a substantially improved exploration efficiency with SFC and the
hierarchical usage of the intrinsic drives. A video of our experimental results
can be found at https://youtu.be/b0MbY3lUlEI.
