Learning from Physical Human Feedback: An Object-Centric One-Shot
  Adaptation Method
  For robots to be effectively deployed in novel environments and tasks, they
must be able to understand the feedback expressed by humans during
intervention. This can either correct undesirable behavior or indicate
additional preferences. Existing methods either require repeated episodes of
interactions or assume prior known reward features, which is data-inefficient
and can hardly transfer to new tasks. We relax these assumptions by describing
human tasks in terms of object-centric sub-tasks and interpreting physical
interventions in relation to specific objects. Our method, Object Preference
Adaptation (OPA), is composed of two key stages: 1) pre-training a base policy
to produce a wide variety of behaviors, and 2) online-updating only certain
weights in the model according to human feedback. The key to our fast, yet
simple adaptation is that general interaction dynamics between agents and
objects are fixed, and only object-specific preferences are updated. Our
adaptation occurs online, requires only one human intervention (one-shot), and
produces new behaviors never seen during training. Trained on cheap synthetic
data instead of expensive human demonstrations, our policy demonstrates
impressive adaptation to human perturbations on challenging, realistic tasks in
our user study. Videos, code, and supplementary material provided.
