Is Curiosity All You Need? On the Utility of Emergent Behaviours from
  Curious Exploration
  Curiosity-based reward schemes can present powerful exploration mechanisms
which facilitate the discovery of solutions for complex, sparse or long-horizon
tasks. However, as the agent learns to reach previously unexplored spaces and
the objective adapts to reward new areas, many behaviours emerge only to
disappear due to being overwritten by the constantly shifting objective. We
argue that merely using curiosity for fast environment exploration or as a
bonus reward for a specific task does not harness the full potential of this
technique and misses useful skills. Instead, we propose to shift the focus
towards retaining the behaviours which emerge during curiosity-based learning.
We posit that these self-discovered behaviours serve as valuable skills in an
agent's repertoire to solve related tasks. Our experiments demonstrate the
continuous shift in behaviour throughout training and the benefits of a simple
policy snapshot method to reuse discovered behaviour for transfer tasks.
