Compositional Transfer in Hierarchical Reinforcement Learning
  The successful application of general reinforcement learning algorithms to
real-world robotics applications is often limited by their high data
requirements. We introduce Regularized Hierarchical Policy Optimization (RHPO)
to improve data-efficiency for domains with multiple dominant tasks and
ultimately reduce required platform time. To this end, we employ compositional
inductive biases on multiple levels and corresponding mechanisms for sharing
off-policy transition data across low-level controllers and tasks as well as
scheduling of tasks. The presented algorithm enables stable and fast learning
for complex, real-world domains in the parallel multitask and sequential
transfer case. We show that the investigated types of hierarchy enable positive
transfer while partially mitigating negative interference and evaluate the
benefits of additional incentives for efficient, compositional task solutions
in single task domains. Finally, we demonstrate substantial data-efficiency and
final performance gains over competitive baselines in a week-long, physical
robot stacking experiment.
