Teacher algorithms for curriculum learning of Deep RL in continuously
  parameterized environments
  We consider the problem of how a teacher algorithm can enable an unknown Deep
Reinforcement Learning (DRL) student to become good at a skill over a wide
range of diverse environments. To do so, we study how a teacher algorithm can
learn to generate a learning curriculum, whereby it sequentially samples
parameters controlling a stochastic procedural generation of environments.
Because it does not initially know the capacities of its student, a key
challenge for the teacher is to discover which environments are easy, difficult
or unlearnable, and in what order to propose them to maximize the efficiency of
learning over the learnable ones. To achieve this, this problem is transformed
into a surrogate continuous bandit problem where the teacher samples
environments in order to maximize absolute learning progress of its student. We
present a new algorithm modeling absolute learning progress with Gaussian
mixture models (ALP-GMM). We also adapt existing algorithms and provide a
complete study in the context of DRL. Using parameterized variants of the
BipedalWalker environment, we study their efficiency to personalize a learning
curriculum for different learners (embodiments), their robustness to the ratio
of learnable/unlearnable environments, and their scalability to non-linear and
high-dimensional parameter spaces. Videos and code are available at
https://github.com/flowersteam/teachDeepRL.
