C-Planning: An Automatic Curriculum for Learning Goal-Reaching Tasks
  Goal-conditioned reinforcement learning (RL) can solve tasks in a wide range
of domains, including navigation and manipulation, but learning to reach
distant goals remains a central challenge to the field. Learning to reach such
goals is particularly hard without any offline data, expert demonstrations, and
reward shaping. In this paper, we propose an algorithm to solve the distant
goal-reaching task by using search at training time to automatically generate a
curriculum of intermediate states. Our algorithm, Classifier-Planning
(C-Planning), frames the learning of the goal-conditioned policies as
expectation maximization: the E-step corresponds to planning an optimal
sequence of waypoints using graph search, while the M-step aims to learn a
goal-conditioned policy to reach those waypoints. Unlike prior methods that
combine goal-conditioned RL with graph search, ours performs search only during
training and not testing, significantly decreasing the compute costs of
deploying the learned policy. Empirically, we demonstrate that our method is
more sample efficient than prior methods. Moreover, it is able to solve very
long horizons manipulation and navigation tasks, tasks that prior
goal-conditioned methods and methods based on graph search fail to solve.
