When Should We Prefer Offline Reinforcement Learning Over Behavioral
  Cloning?
  Offline reinforcement learning (RL) algorithms can acquire effective policies
by utilizing previously collected experience, without any online interaction.
It is widely understood that offline RL is able to extract good policies even
from highly suboptimal data, a scenario where imitation learning finds
suboptimal solutions that do not improve over the demonstrator that generated
the dataset. However, another common use case for practitioners is to learn
from data that resembles demonstrations. In this case, one can choose to apply
offline RL, but can also use behavioral cloning (BC) algorithms, which mimic a
subset of the dataset via supervised learning. Therefore, it seems natural to
ask: when can an offline RL method outperform BC with an equal amount of expert
data, even when BC is a natural choice? To answer this question, we
characterize the properties of environments that allow offline RL methods to
perform better than BC methods, even when only provided with expert data.
Additionally, we show that policies trained on sufficiently noisy suboptimal
data can attain better performance than even BC algorithms with expert data,
especially on long-horizon problems. We validate our theoretical results via
extensive experiments on both diagnostic and high-dimensional domains including
robotic manipulation, maze navigation, and Atari games, with a variety of data
distributions. We observe that, under specific but common conditions such as
sparse rewards or noisy data sources, modern offline RL methods can
significantly outperform BC.
