Temporal Abstraction in Reinforcement Learning with the Successor
  Representation
  Reasoning at multiple levels of temporal abstraction is one of the key
attributes of intelligence. In reinforcement learning, this is often modeled
through temporally extended courses of actions called options. Options allow
agents to make predictions and to operate at different levels of abstraction
within an environment. Nevertheless, approaches based on the options framework
often start with the assumption that a reasonable set of options is known
beforehand. When this is not the case, there are no definitive answers for
which options one should consider. In this paper, we argue that the successor
representation (SR), which encodes states based on the pattern of state
visitation that follows them, can be seen as a natural substrate for the
discovery and use of temporal abstractions. To support our claim, we take a big
picture view of recent results, showing how the SR can be used to discover
options that facilitate either temporally-extended exploration or planning. We
cast these results as instantiations of a general framework for option
discovery in which the agent's representation is used to identify useful
options, which are then used to further improve its representation. This
results in a virtuous, never-ending, cycle in which both the representation and
the options are constantly refined based on each other. Beyond option discovery
itself, we discuss how the SR allows us to augment a set of options into a
combinatorially large counterpart without additional learning. This is achieved
through the combination of previously learned options. Our empirical evaluation
focuses on options discovered for temporally-extended exploration and on the
use of the SR to combine them. The results of our experiments shed light on
design decisions involved in the definition of options and demonstrate the
synergy of different methods based on the SR, such as eigenoptions and the
option keyboard.
