Emergent Complexity and Zero-shot Transfer via Unsupervised Environment
  Design
  A wide range of reinforcement learning (RL) problems - including robustness,
transfer learning, unsupervised RL, and emergent complexity - require
specifying a distribution of tasks or environments in which a policy will be
trained. However, creating a useful distribution of environments is error
prone, and takes a significant amount of developer time and effort. We propose
Unsupervised Environment Design (UED) as an alternative paradigm, where
developers provide environments with unknown parameters, and these parameters
are used to automatically produce a distribution over valid, solvable
environments. Existing approaches to automatically generating environments
suffer from common failure modes: domain randomization cannot generate
structure or adapt the difficulty of the environment to the agent's learning
progress, and minimax adversarial training leads to worst-case environments
that are often unsolvable. To generate structured, solvable environments for
our protagonist agent, we introduce a second, antagonist agent that is allied
with the environment-generating adversary. The adversary is motivated to
generate environments which maximize regret, defined as the difference between
the protagonist and antagonist agent's return. We call our technique
Protagonist Antagonist Induced Regret Environment Design (PAIRED). Our
experiments demonstrate that PAIRED produces a natural curriculum of
increasingly complex environments, and PAIRED agents achieve higher zero-shot
transfer performance when tested in highly novel environments.
