Compositional Planning Using Optimal Option Models
  In this paper we introduce a framework for option model composition. Option
models are temporal abstractions that, like macro-operators in classical
planning, jump directly from a start state to an end state. Prior work has
focused on constructing option models from primitive actions, by intra-option
model learning; or on using option models to construct a value function, by
inter-option planning. We present a unified view of intra- and inter-option
model learning, based on a major generalisation of the Bellman equation. Our
fundamental operation is the recursive composition of option models into other
option models. This key idea enables compositional planning over many levels of
abstraction. We illustrate our framework using a dynamic programming algorithm
that simultaneously constructs optimal option models for multiple subgoals, and
also searches over those option models to provide rapid progress towards other
subgoals.
