Flexible Option Learning
  Temporal abstraction in reinforcement learning (RL), offers the promise of
improving generalization and knowledge transfer in complex environments, by
propagating information more efficiently over time. Although option learning
was initially formulated in a way that allows updating many options
simultaneously, using off-policy, intra-option learning (Sutton, Precup &
Singh, 1999), many of the recent hierarchical reinforcement learning approaches
only update a single option at a time: the option currently executing. We
revisit and extend intra-option learning in the context of deep reinforcement
learning, in order to enable updating all options consistent with current
primitive action choices, without introducing any additional estimates. Our
method can therefore be naturally adopted in most hierarchical RL frameworks.
When we combine our approach with the option-critic algorithm for option
discovery, we obtain significant improvements in performance and
data-efficiency across a wide variety of domains.
