Learning a subspace of policies for online adaptation in Reinforcement
  Learning
  Deep Reinforcement Learning (RL) is mainly studied in a setting where the
training and the testing environments are similar. But in many practical
applications, these environments may differ. For instance, in control systems,
the robot(s) on which a policy is learned might differ from the robot(s) on
which a policy will run. It can be caused by different internal factors (e.g.,
calibration issues, system attrition, defective modules) or also by external
changes (e.g., weather conditions). There is a need to develop RL methods that
generalize well to variations of the training conditions. In this article, we
consider the simplest yet hard to tackle generalization setting where the test
environment is unknown at train time, forcing the agent to adapt to the
system's new dynamics. This online adaptation process can be computationally
expensive (e.g., fine-tuning) and cannot rely on meta-RL techniques since there
is just a single train environment. To do so, we propose an approach where we
learn a subspace of policies within the parameter space. This subspace contains
an infinite number of policies that are trained to solve the training
environment while having different parameter values. As a consequence, two
policies in that subspace process information differently and exhibit different
behaviors when facing variations of the train environment. Our experiments
carried out over a large variety of benchmarks compare our approach with
baselines, including diversity-based methods. In comparison, our approach is
simple to tune, does not need any extra component (e.g., discriminator) and
learns policies able to gather a high reward on unseen environments.
