Adapting Behaviour for Learning Progress
  Determining what experience to generate to best facilitate learning (i.e.
exploration) is one of the distinguishing features and open challenges in
reinforcement learning. The advent of distributed agents that interact with
parallel instances of the environment has enabled larger scales and greater
flexibility, but has not removed the need to tune exploration to the task,
because the ideal data for the learning algorithm necessarily depends on its
process of learning. We propose to dynamically adapt the data generation by
using a non-stationary multi-armed bandit to optimize a proxy of the learning
progress. The data distribution is controlled by modulating multiple parameters
of the policy (such as stochasticity, consistency or optimism) without
significant overhead. The adaptation speed of the bandit can be increased by
exploiting the factored modulation structure. We demonstrate on a suite of
Atari 2600 games how this unified approach produces results comparable to
per-task tuning at a fraction of the cost.
