CoMPS: Continual Meta Policy Search
  We develop a new continual meta-learning method to address challenges in
sequential multi-task learning. In this setting, the agent's goal is to achieve
high reward over any sequence of tasks quickly. Prior meta-reinforcement
learning algorithms have demonstrated promising results in accelerating the
acquisition of new tasks. However, they require access to all tasks during
training. Beyond simply transferring past experience to new tasks, our goal is
to devise continual reinforcement learning algorithms that learn to learn,
using their experience on previous tasks to learn new tasks more quickly. We
introduce a new method, continual meta-policy search (CoMPS), that removes this
limitation by meta-training in an incremental fashion, over each task in a
sequence, without revisiting prior tasks. CoMPS continuously repeats two
subroutines: learning a new task using RL and using the experience from RL to
perform completely offline meta-learning to prepare for subsequent task
learning. We find that CoMPS outperforms prior continual learning and
off-policy meta-reinforcement methods on several sequences of challenging
continuous control tasks.
