Replay-Guided Adversarial Environment Design
  Deep reinforcement learning (RL) agents may successfully generalize to new
settings if trained on an appropriately diverse set of environment and task
configurations. Unsupervised Environment Design (UED) is a promising
self-supervised RL paradigm, wherein the free parameters of an underspecified
environment are automatically adapted during training to the agent's
capabilities, leading to the emergence of diverse training environments. Here,
we cast Prioritized Level Replay (PLR), an empirically successful but
theoretically unmotivated method that selectively samples randomly-generated
training levels, as UED. We argue that by curating completely random levels,
PLR, too, can generate novel and complex levels for effective training. This
insight reveals a natural class of UED methods we call Dual Curriculum Design
(DCD). Crucially, DCD includes both PLR and a popular UED algorithm, PAIRED, as
special cases and inherits similar theoretical guarantees. This connection
allows us to develop novel theory for PLR, providing a version with a
robustness guarantee at Nash equilibria. Furthermore, our theory suggests a
highly counterintuitive improvement to PLR: by stopping the agent from updating
its policy on uncurated levels (training on less data), we can improve the
convergence to Nash equilibria. Indeed, our experiments confirm that our new
method, PLR$^{\perp}$, obtains better results on a suite of
out-of-distribution, zero-shot transfer tasks, in addition to demonstrating
that PLR$^{\perp}$ improves the performance of PAIRED, from which it inherited
its theoretical framework.
