Hierarchical Deep Reinforcement Learning: Integrating Temporal
  Abstraction and Intrinsic Motivation
  Learning goal-directed behavior in environments with sparse feedback is a
major challenge for reinforcement learning algorithms. The primary difficulty
arises due to insufficient exploration, resulting in an agent being unable to
learn robust value functions. Intrinsically motivated agents can explore new
behavior for its own sake rather than to directly solve problems. Such
intrinsic behaviors could eventually help the agent solve tasks posed by the
environment. We present hierarchical-DQN (h-DQN), a framework to integrate
hierarchical value functions, operating at different temporal scales, with
intrinsically motivated deep reinforcement learning. A top-level value function
learns a policy over intrinsic goals, and a lower-level function learns a
policy over atomic actions to satisfy the given goals. h-DQN allows for
flexible goal specifications, such as functions over entities and relations.
This provides an efficient space for exploration in complicated environments.
We demonstrate the strength of our approach on two problems with very sparse,
delayed feedback: (1) a complex discrete stochastic decision process, and (2)
the classic ATARI game `Montezuma's Revenge'.
