Automatic Curricula via Expert Demonstrations
  We propose Automatic Curricula via Expert Demonstrations (ACED), a
reinforcement learning (RL) approach that combines the ideas of imitation
learning and curriculum learning in order to solve challenging robotic
manipulation tasks with sparse reward functions. Curriculum learning solves
complicated RL tasks by introducing a sequence of auxiliary tasks with
increasing difficulty, yet how to automatically design effective and
generalizable curricula remains a challenging research problem. ACED extracts
curricula from a small amount of expert demonstration trajectories by dividing
demonstrations into sections and initializing training episodes to states
sampled from different sections of demonstrations. Through moving the reset
states from the end to the beginning of demonstrations as the learning agent
improves its performance, ACED not only learns challenging manipulation tasks
with unseen initializations and goals, but also discovers novel solutions that
are distinct from the demonstrations. In addition, ACED can be naturally
combined with other imitation learning methods to utilize expert demonstrations
in a more efficient manner, and we show that a combination of ACED with
behavior cloning allows pick-and-place tasks to be learned with as few as 1
demonstration and block stacking tasks to be learned with 20 demonstrations.
