ASHA: Assistive Teleoperation via Human-in-the-Loop Reinforcement
  Learning
  Building assistive interfaces for controlling robots through arbitrary,
high-dimensional, noisy inputs (e.g., webcam images of eye gaze) can be
challenging, especially when it involves inferring the user's desired action in
the absence of a natural 'default' interface. Reinforcement learning from
online user feedback on the system's performance presents a natural solution to
this problem, and enables the interface to adapt to individual users. However,
this approach tends to require a large amount of human-in-the-loop training
data, especially when feedback is sparse. We propose a hierarchical solution
that learns efficiently from sparse user feedback: we use offline pre-training
to acquire a latent embedding space of useful, high-level robot behaviors,
which, in turn, enables the system to focus on using online user feedback to
learn a mapping from user inputs to desired high-level behaviors. The key
insight is that access to a pre-trained policy enables the system to learn more
from sparse rewards than a na\"ive RL algorithm: using the pre-trained policy,
the system can make use of successful task executions to relabel, in hindsight,
what the user actually meant to do during unsuccessful executions. We evaluate
our method primarily through a user study with 12 participants who perform
tasks in three simulated robotic manipulation domains using a webcam and their
eye gaze: flipping light switches, opening a shelf door to reach objects
inside, and rotating a valve. The results show that our method successfully
learns to map 128-dimensional gaze features to 7-dimensional joint torques from
sparse rewards in under 10 minutes of online training, and seamlessly helps
users who employ different gaze strategies, while adapting to distributional
shift in webcam inputs, tasks, and environments.
