COG: Connecting New Skills to Past Experience with Offline Reinforcement
  Learning
  Reinforcement learning has been applied to a wide variety of robotics
problems, but most of such applications involve collecting data from scratch
for each new task. Since the amount of robot data we can collect for any single
task is limited by time and cost considerations, the learned behavior is
typically narrow: the policy can only execute the task in a handful of
scenarios that it was trained on. What if there was a way to incorporate a
large amount of prior data, either from previously solved tasks or from
unsupervised or undirected environment interaction, to extend and generalize
learned behaviors? While most prior work on extending robotic skills using
pre-collected data focuses on building explicit hierarchies or skill
decompositions, we show in this paper that we can reuse prior data to extend
new skills simply through dynamic programming. We show that even when the prior
data does not actually succeed at solving the new task, it can still be
utilized for learning a better policy, by providing the agent with a broader
understanding of the mechanics of its environment. We demonstrate the
effectiveness of our approach by chaining together several behaviors seen in
prior datasets for solving a new task, with our hardest experimental setting
involving composing four robotic skills in a row: picking, placing, drawer
opening, and grasping, where a +1/0 sparse reward is provided only on task
completion. We train our policies in an end-to-end fashion, mapping
high-dimensional image observations to low-level robot control commands, and
present results in both simulated and real world domains. Additional materials
and source code can be found on our project website:
https://sites.google.com/view/cog-rl
