Human-in-the-Loop Imitation Learning using Remote Teleoperation
  Imitation Learning is a promising paradigm for learning complex robot
manipulation skills by reproducing behavior from human demonstrations. However,
manipulation tasks often contain bottleneck regions that require a sequence of
precise actions to make meaningful progress, such as a robot inserting a pod
into a coffee machine to make coffee. Trained policies can fail in these
regions because small deviations in actions can lead the policy into states not
covered by the demonstrations. Intervention-based policy learning is an
alternative that can address this issue -- it allows human operators to monitor
trained policies and take over control when they encounter failures. In this
paper, we build a data collection system tailored to 6-DoF manipulation
settings, that enables remote human operators to monitor and intervene on
trained policies. We develop a simple and effective algorithm to train the
policy iteratively on new data collected by the system that encourages the
policy to learn how to traverse bottlenecks through the interventions. We
demonstrate that agents trained on data collected by our intervention-based
system and algorithm outperform agents trained on an equivalent number of
samples collected by non-interventional demonstrators, and further show that
our method outperforms multiple state-of-the-art baselines for learning from
the human interventions on a challenging robot threading task and a coffee
making task. Additional results and videos at
https://sites.google.com/stanford.edu/iwr .
