Bridging the Imitation Gap by Adaptive Insubordination
  In practice, imitation learning is preferred over pure reinforcement learning
whenever it is possible to design a teaching agent to provide expert
supervision. However, we show that when the teaching agent makes decisions with
access to privileged information that is unavailable to the student, this
information is marginalized during imitation learning, resulting in an
"imitation gap" and, potentially, poor results. Prior work bridges this gap via
a progression from imitation learning to reinforcement learning. While often
successful, gradual progression fails for tasks that require frequent switches
between exploration and memorization. To better address these tasks and
alleviate the imitation gap we propose 'Adaptive Insubordination' (ADVISOR).
ADVISOR dynamically weights imitation and reward-based reinforcement learning
losses during training, enabling on-the-fly switching between imitation and
exploration. On a suite of challenging tasks set within gridworlds, multi-agent
particle environments, and high-fidelity 3D simulators, we show that on-the-fly
switching with ADVISOR outperforms pure imitation, pure reinforcement learning,
as well as their sequential and parallel combinations.
