A Workflow for Offline Model-Free Robotic Reinforcement Learning
  Offline reinforcement learning (RL) enables learning control policies by
utilizing only prior experience, without any online interaction. This can allow
robots to acquire generalizable skills from large and diverse datasets, without
any costly or unsafe online data collection. Despite recent algorithmic
advances in offline RL, applying these methods to real-world problems has
proven challenging. Although offline RL methods can learn from prior data,
there is no clear and well-understood process for making various design
choices, from model architecture to algorithm hyperparameters, without actually
evaluating the learned policies online. In this paper, our aim is to develop a
practical workflow for using offline RL analogous to the relatively
well-understood workflows for supervised learning problems. To this end, we
devise a set of metrics and conditions that can be tracked over the course of
offline training, and can inform the practitioner about how the algorithm and
model architecture should be adjusted to improve final performance. Our
workflow is derived from a conceptual understanding of the behavior of
conservative offline RL algorithms and cross-validation in supervised learning.
We demonstrate the efficacy of this workflow in producing effective policies
without any online tuning, both in several simulated robotic learning scenarios
and for three tasks on two distinct real robots, focusing on learning
manipulation skills with raw image observations with sparse binary rewards.
Explanatory video and additional results can be found at
sites.google.com/view/offline-rl-workflow
