Learning and Exploiting Multiple Subgoals for Fast Exploration in
  Hierarchical Reinforcement Learning
  Hierarchical Reinforcement Learning (HRL) exploits temporally extended
actions, or options, to make decisions from a higher-dimensional perspective to
alleviate the sparse reward problem, one of the most challenging problems in
reinforcement learning. The majority of existing HRL algorithms require either
significant manual design with respect to the specific environment or enormous
exploration to automatically learn options from data. To achieve fast
exploration without using manual design, we devise a multi-goal HRL algorithm,
consisting of a high-level policy Manager and a low-level policy Worker. The
Manager provides the Worker multiple subgoals at each time step. Each subgoal
corresponds to an option to control the environment. Although the agent may
show some confusion at the beginning of training since it is guided by three
diverse subgoals, the agent's behavior policy will quickly learn how to respond
to multiple subgoals from the high-level controller on different occasions. By
exploiting multiple subgoals, the exploration efficiency is significantly
improved. We conduct experiments in Atari's Montezuma's Revenge environment, a
well-known sparse reward environment, and in doing so achieve the same
performance as state-of-the-art HRL methods with substantially reduced training
time cost.
