Rethinking Goal-conditioned Supervised Learning and Its Connection to
  Offline RL
  Solving goal-conditioned tasks with sparse rewards using self-supervised
learning is promising because of its simplicity and stability over current
reinforcement learning (RL) algorithms. A recent work, called Goal-Conditioned
Supervised Learning (GCSL), provides a new learning framework by iteratively
relabeling and imitating self-generated experiences. In this paper, we revisit
the theoretical property of GCSL -- optimizing a lower bound of the goal
reaching objective, and extend GCSL as a novel offline goal-conditioned RL
algorithm. The proposed method is named Weighted GCSL (WGCSL), in which we
introduce an advanced compound weight consisting of three parts (1) discounted
weight for goal relabeling, (2) goal-conditioned exponential advantage weight,
and (3) best-advantage weight. Theoretically, WGCSL is proved to optimize an
equivalent lower bound of the goal-conditioned RL objective and generates
monotonically improved policies via an iterated scheme. The monotonic property
holds for any behavior policies, and therefore WGCSL can be applied to both
online and offline settings. To evaluate algorithms in the offline
goal-conditioned RL setting, we provide a benchmark including a range of point
and simulated robot domains. Experiments in the introduced benchmark
demonstrate that WGCSL can consistently outperform GCSL and existing
state-of-the-art offline methods in the fully offline goal-conditioned setting.
