It Takes Four to Tango: Multiagent Selfplay for Automatic Curriculum
  Generation
  We are interested in training general-purpose reinforcement learning agents
that can solve a wide variety of goals. Training such agents efficiently
requires automatic generation of a goal curriculum. This is challenging as it
requires (a) exploring goals of increasing difficulty, while ensuring that the
agent (b) is exposed to a diverse set of goals in a sample efficient manner and
(c) does not catastrophically forget previously solved goals. We propose
Curriculum Self Play (CuSP), an automated goal generation framework that seeks
to satisfy these desiderata by virtue of a multi-player game with four agents.
We extend the asymmetric curricula learning in PAIRED (Dennis et al., 2020) to
a symmetrized game that carefully balances cooperation and competition between
two off-policy student learners and two regret-maximizing teachers. CuSP
additionally introduces entropic goal coverage and accounts for the
non-stationary nature of the students, allowing us to automatically induce a
curriculum that balances progressive exploration with anti-catastrophic
exploitation. We demonstrate that our method succeeds at generating an
effective curricula of goals for a range of control tasks, outperforming other
methods at zero-shot test-time generalization to novel out-of-distribution
goals.
