Batch Exploration with Examples for Scalable Robotic Reinforcement
  Learning
  Learning from diverse offline datasets is a promising path towards learning
general purpose robotic agents. However, a core challenge in this paradigm lies
in collecting large amounts of meaningful data, while not depending on a human
in the loop for data collection. One way to address this challenge is through
task-agnostic exploration, where an agent attempts to explore without a
task-specific reward function, and collect data that can be useful for any
downstream task. While these approaches have shown some promise in simple
domains, they often struggle to explore the relevant regions of the state space
in more challenging settings, such as vision based robotic manipulation. This
challenge stems from an objective that encourages exploring everything in a
potentially vast state space. To mitigate this challenge, we propose to focus
exploration on the important parts of the state space using weak human
supervision. Concretely, we propose an exploration technique, Batch Exploration
with Examples (BEE), that explores relevant regions of the state-space, guided
by a modest number of human provided images of important states. These human
provided images only need to be collected once at the beginning of data
collection and can be collected in a matter of minutes, allowing us to scalably
collect diverse datasets, which can then be combined with any batch RL
algorithm. We find that BEE is able to tackle challenging vision-based
manipulation tasks both in simulation and on a real Franka robot, and observe
that compared to task-agnostic and weakly-supervised exploration techniques, it
(1) interacts more than twice as often with relevant objects, and (2) improves
downstream task performance when used in conjunction with offline RL.
