How to Spend Your Robot Time: Bridging Kickstarting and Offline
  Reinforcement Learning for Vision-based Robotic Manipulation
  Reinforcement learning (RL) has been shown to be effective at learning
control from experience. However, RL typically requires a large amount of
online interaction with the environment. This limits its applicability to
real-world settings, such as in robotics, where such interaction is expensive.
In this work we investigate ways to minimize online interactions in a target
task, by reusing a suboptimal policy we might have access to, for example from
training on related prior tasks, or in simulation. To this end, we develop two
RL algorithms that can speed up training by using not only the action
distributions of teacher policies, but also data collected by such policies on
the task at hand. We conduct a thorough experimental study of how to use
suboptimal teachers on a challenging robotic manipulation benchmark on
vision-based stacking with diverse objects. We compare our methods to offline,
online, offline-to-online, and kickstarting RL algorithms. By doing so, we find
that training on data from both the teacher and student, enables the best
performance for limited data budgets. We examine how to best allocate a limited
data budget -- on the target task -- between the teacher and the student
policy, and report experiments using varying budgets, two teachers with
different degrees of suboptimality, and five stacking tasks that require a
diverse set of behaviors. Our analysis, both in simulation and in the real
world, shows that our approach is the best across data budgets, while standard
offline RL from teacher rollouts is surprisingly effective when enough data is
given.
