Automatic Curriculum Learning through Value Disagreement
  Continually solving new, unsolved tasks is the key to learning diverse
behaviors. Through reinforcement learning (RL), we have made massive strides
towards solving tasks that have a single goal. However, in the multi-task
domain, where an agent needs to reach multiple goals, the choice of training
goals can largely affect sample efficiency. When biological agents learn, there
is often an organized and meaningful order to which learning happens. Inspired
by this, we propose setting up an automatic curriculum for goals that the agent
needs to solve. Our key insight is that if we can sample goals at the frontier
of the set of goals that an agent is able to reach, it will provide a
significantly stronger learning signal compared to randomly sampled goals. To
operationalize this idea, we introduce a goal proposal module that prioritizes
goals that maximize the epistemic uncertainty of the Q-function of the policy.
This simple technique samples goals that are neither too hard nor too easy for
the agent to solve, hence enabling continual improvement. We evaluate our
method across 13 multi-goal robotic tasks and 5 navigation tasks, and
demonstrate performance gains over current state-of-the-art methods.
