Skill Preferences: Learning to Extract and Execute Robotic Skills from
  Human Feedback
  A promising approach to solving challenging long-horizon tasks has been to
extract behavior priors (skills) by fitting generative models to large offline
datasets of demonstrations. However, such generative models inherit the biases
of the underlying data and result in poor and unusable skills when trained on
imperfect demonstration data. To better align skill extraction with human
intent we present Skill Preferences (SkiP), an algorithm that learns a model
over human preferences and uses it to extract human-aligned skills from offline
data. After extracting human-preferred skills, SkiP also utilizes human
feedback to solve down-stream tasks with RL. We show that SkiP enables a
simulated kitchen robot to solve complex multi-step manipulation tasks and
substantially outperforms prior leading RL algorithms with human preferences as
well as leading skill extraction algorithms without human preferences.
