Disentangled Cumulants Help Successor Representations Transfer to New
  Tasks
  Biological intelligence can learn to solve many diverse tasks in a data
efficient manner by re-using basic knowledge and skills from one task to
another. Furthermore, many of such skills are acquired without explicit
supervision in an intrinsically driven fashion. This is in contrast to the
state-of-the-art reinforcement learning agents, which typically start learning
each new task from scratch and struggle with knowledge transfer. In this paper
we propose a principled way to learn a basis set of policies, which, when
recombined through generalised policy improvement, come with guarantees on the
coverage of the final task space. In particular, we concentrate on solving
goal-based downstream tasks where the execution order of actions is not
important. We demonstrate both theoretically and empirically that learning a
small number of policies that reach intrinsically specified goal regions in a
disentangled latent space can be re-used to quickly achieve a high level of
performance on an exponentially larger number of externally specified, often
significantly more complex downstream tasks. Our learning pipeline consists of
two stages. First, the agent learns to perform intrinsically generated,
goal-based tasks in the total absence of environmental rewards. Second, the
agent leverages this experience to quickly achieve a high level of performance
on numerous diverse externally specified tasks.
