Offline Reinforcement Learning Hands-On
  Offline Reinforcement Learning (RL) aims to turn large datasets into powerful
decision-making engines without any online interactions with the environment.
This great promise has motivated a large amount of research that hopes to
replicate the success RL has experienced in simulation settings. This work
ambitions to reflect upon these efforts from a practitioner viewpoint. We start
by discussing the dataset properties that we hypothesise can characterise the
type of offline methods that will be the most successful. We then verify these
claims through a set of experiments and designed datasets generated from
environments with both discrete and continuous action spaces. We experimentally
validate that diversity and high-return examples in the data are crucial to the
success of offline RL and show that behavioural cloning remains a strong
contender compared to its contemporaries. Overall, this work stands as a
tutorial to help people build their intuition on today's offline RL methods and
their applicability.
