D4RL: Datasets for Deep Data-Driven Reinforcement Learning
  The offline reinforcement learning (RL) setting (also known as full batch
RL), where a policy is learned from a static dataset, is compelling as progress
enables RL methods to take advantage of large, previously-collected datasets,
much like how the rise of large datasets has fueled results in supervised
learning. However, existing online RL benchmarks are not tailored towards the
offline setting and existing offline RL benchmarks are restricted to data
generated by partially-trained agents, making progress in offline RL difficult
to measure. In this work, we introduce benchmarks specifically designed for the
offline setting, guided by key properties of datasets relevant to real-world
applications of offline RL. With a focus on dataset collection, examples of
such properties include: datasets generated via hand-designed controllers and
human demonstrators, multitask datasets where an agent performs different tasks
in the same environment, and datasets collected with mixtures of policies. By
moving beyond simple benchmark tasks and data collected by partially-trained RL
agents, we reveal important and unappreciated deficiencies of existing
algorithms. To facilitate research, we have released our benchmark tasks and
datasets with a comprehensive evaluation of existing algorithms, an evaluation
protocol, and open-source examples. This serves as a common starting point for
the community to identify shortcomings in existing offline RL methods and a
collaborative route for progress in this emerging area.
