On Credit Assignment in Hierarchical Reinforcement Learning
  Hierarchical Reinforcement Learning (HRL) has held longstanding promise to
advance reinforcement learning. Yet, it has remained a considerable challenge
to develop practical algorithms that exhibit some of these promises. To improve
our fundamental understanding of HRL, we investigate hierarchical credit
assignment from the perspective of conventional multistep reinforcement
learning. We show how e.g., a 1-step `hierarchical backup' can be seen as a
conventional multistep backup with $n$ skip connections over time connecting
each subsequent state to the first independent of actions inbetween.
Furthermore, we find that generalizing hierarchy to multistep return estimation
methods requires us to consider how to partition the environment trace, in
order to construct backup paths. We leverage these insight to develop a new
hierarchical algorithm Hier$Q_k(\lambda)$, for which we demonstrate that
hierarchical credit assignment alone can already boost agent performance (i.e.,
when eliminating generalization or exploration). Altogether, our work yields
fundamental insight into the nature of hierarchical backups and distinguishes
this as an additional basis for reinforcement learning research.
