Feature Expansive Reward Learning: Rethinking Human Input
  When a person is not satisfied with how a robot performs a task, they can
intervene to correct it. Reward learning methods enable the robot to adapt its
reward function online based on such human input, but they rely on handcrafted
features. When the correction cannot be explained by these features, recent
work in deep Inverse Reinforcement Learning (IRL) suggests that the robot could
ask for task demonstrations and recover a reward defined over the raw state
space. Our insight is that rather than implicitly learning about the missing
feature(s) from demonstrations, the robot should instead ask for data that
explicitly teaches it about what it is missing. We introduce a new type of
human input in which the person guides the robot from states where the feature
being taught is highly expressed to states where it is not. We propose an
algorithm for learning the feature from the raw state space and integrating it
into the reward function. By focusing the human input on the missing feature,
our method decreases sample complexity and improves generalization of the
learned reward over the above deep IRL baseline. We show this in experiments
with a physical 7DOF robot manipulator, as well as in a user study conducted in
a simulated environment.
