DCUR: Data Curriculum for Teaching via Samples with Reinforcement
  Learning
  Deep reinforcement learning (RL) has shown great empirical successes, but
suffers from brittleness and sample inefficiency. A potential remedy is to use
a previously-trained policy as a source of supervision. In this work, we refer
to these policies as teachers and study how to transfer their expertise to new
student policies by focusing on data usage. We propose a framework, Data
CUrriculum for Reinforcement learning (DCUR), which first trains teachers using
online deep RL, and stores the logged environment interaction history. Then,
students learn by running either offline RL or by using teacher data in
combination with a small amount of self-generated data. DCUR's central idea
involves defining a class of data curricula which, as a function of training
time, limits the student to sampling from a fixed subset of the full teacher
data. We test teachers and students using state-of-the-art deep RL algorithms
across a variety of data curricula. Results suggest that the choice of data
curricula significantly impacts student learning, and that it is beneficial to
limit the data during early training stages while gradually letting the data
availability grow over time. We identify when the student can learn offline and
match teacher performance without relying on specialized offline RL algorithms.
Furthermore, we show that collecting a small fraction of online data provides
complementary benefits with the data curriculum. Supplementary material is
available at https://tinyurl.com/teach-dcur.
