Learning Adaptive Exploration Strategies in Dynamic Environments Through
  Informed Policy Regularization
  We study the problem of learning exploration-exploitation strategies that
effectively adapt to dynamic environments, where the task may change over time.
While RNN-based policies could in principle represent such strategies, in
practice their training time is prohibitive and the learning process often
converges to poor solutions. In this paper, we consider the case where the
agent has access to a description of the task (e.g., a task id or task
parameters) at training time, but not at test time. We propose a novel
algorithm that regularizes the training of an RNN-based policy using informed
policies trained to maximize the reward in each task. This dramatically reduces
the sample complexity of training RNN-based policies, without losing their
representational power. As a result, our method learns exploration strategies
that efficiently balance between gathering information about the unknown and
changing task and maximizing the reward over time. We test the performance of
our algorithm in a variety of environments where tasks may vary within each
episode.
