The Option Keyboard: Combining Skills in Reinforcement Learning
  The ability to combine known skills to create new ones may be crucial in the
solution of complex reinforcement learning problems that unfold over extended
periods. We argue that a robust way of combining skills is to define and
manipulate them in the space of pseudo-rewards (or "cumulants"). Based on this
premise, we propose a framework for combining skills using the formalism of
options. We show that every deterministic option can be unambiguously
represented as a cumulant defined in an extended domain. Building on this
insight and on previous results on transfer learning, we show how to
approximate options whose cumulants are linear combinations of the cumulants of
known options. This means that, once we have learned options associated with a
set of cumulants, we can instantaneously synthesise options induced by any
linear combination of them, without any learning involved. We describe how this
framework provides a hierarchical interface to the environment whose abstract
actions correspond to combinations of basic skills. We demonstrate the
practical benefits of our approach in a resource management problem and a
navigation task involving a quadrupedal simulated robot.
