PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via
  Relabeling Experience and Unsupervised Pre-training
  Conveying complex objectives to reinforcement learning (RL) agents can often
be difficult, involving meticulous design of reward functions that are
sufficiently informative yet easy enough to provide. Human-in-the-loop RL
methods allow practitioners to instead interactively teach agents through
tailored feedback; however, such approaches have been challenging to scale
since human feedback is very expensive. In this work, we aim to make this
process more sample- and feedback-efficient. We present an off-policy,
interactive RL algorithm that capitalizes on the strengths of both feedback and
off-policy learning. Specifically, we learn a reward model by actively querying
a teacher's preferences between two clips of behavior and use it to train an
agent. To enable off-policy learning, we relabel all the agent's past
experience when its reward model changes. We additionally show that
pre-training our agents with unsupervised exploration substantially increases
the mileage of its queries. We demonstrate that our approach is capable of
learning tasks of higher complexity than previously considered by
human-in-the-loop methods, including a variety of locomotion and robotic
manipulation skills. We also show that our method is able to utilize real-time
human feedback to effectively prevent reward exploitation and learn new
behaviors that are difficult to specify with standard reward functions.
