Beyond Fine-Tuning: Transferring Behavior in Reinforcement Learning
  Designing agents that acquire knowledge autonomously and use it to solve new
tasks efficiently is an important challenge in reinforcement learning.
Knowledge acquired during an unsupervised pre-training phase is often
transferred by fine-tuning neural network weights once rewards are exposed, as
is common practice in supervised domains. Given the nature of the reinforcement
learning problem, we argue that standard fine-tuning strategies alone are not
enough for efficient transfer in challenging domains. We introduce Behavior
Transfer (BT), a technique that leverages pre-trained policies for exploration
and that is complementary to transferring neural network weights. Our
experiments show that, when combined with large-scale pre-training in the
absence of rewards, existing intrinsic motivation objectives can lead to the
emergence of complex behaviors. These pre-trained policies can then be
leveraged by BT to discover better solutions than without pre-training, and
combining BT with standard fine-tuning strategies results in additional
benefits. The largest gains are generally observed in domains requiring
structured exploration, including settings where the behavior of the
pre-trained policies is misaligned with the downstream task.
