Exploring with Sticky Mittens: Reinforcement Learning with Expert
  Interventions via Option Templates
  Environments with sparse rewards and long horizons pose a significant
challenge for current reinforcement learning algorithms. A key feature enabling
humans to learn challenging control tasks is that they often receive expert
intervention that enables them to understand the high-level structure of the
task before mastering low-level control actions. We propose a framework for
leveraging expert intervention to solve long-horizon reinforcement learning
tasks. We consider option templates, which are specifications encoding a
potential option that can be trained using reinforcement learning. We formulate
expert intervention as allowing the agent to execute option templates before
learning an implementation. This enables them to use an option, before
committing costly resources to learning it. We evaluate our approach on three
challenging reinforcement learning problems, showing that it outperforms state
of-the-art approaches by an order of magnitude. Project website at
https://sites.google.com/view/stickymittens
