Curriculum Offline Imitation Learning
  Offline reinforcement learning (RL) tasks require the agent to learn from a
pre-collected dataset with no further interactions with the environment.
Despite the potential to surpass the behavioral policies, RL-based methods are
generally impractical due to the training instability and bootstrapping the
extrapolation errors, which always require careful hyperparameter tuning via
online evaluation. In contrast, offline imitation learning (IL) has no such
issues since it learns the policy directly without estimating the value
function by bootstrapping. However, IL is usually limited in the capability of
the behavioral policy and tends to learn a mediocre behavior from the dataset
collected by the mixture of policies. In this paper, we aim to take advantage
of IL but mitigate such a drawback. Observing that behavior cloning is able to
imitate neighboring policies with less data, we propose \textit{Curriculum
Offline Imitation Learning (COIL)}, which utilizes an experience picking
strategy for imitating from adaptive neighboring policies with a higher return,
and improves the current policy along curriculum stages. On continuous control
benchmarks, we compare COIL against both imitation-based and RL-based methods,
showing that it not only avoids just learning a mediocre behavior on mixed
datasets but is also even competitive with state-of-the-art offline RL methods.
