Phasic Self-Imitative Reduction for Sparse-Reward Goal-Conditioned Reinforcement Learning
It has been a recent trend to leverage the power of supervised learning (SL)
towards more effective reinforcement learning (RL) methods. We propose a novel
phasic approach by alternating online RL and offline SL for tackling
sparse-reward goal-conditioned problems. In the online phase, we perform RL
training and collect rollout data while in the offline phase, we perform SL on
those successful trajectories from the dataset. To further improve sample
efficiency, we adopt additional techniques in the online phase including task
reduction to generate more feasible trajectories and a value-difference-based
intrinsic reward to alleviate the sparse-reward issue. We call this overall
algorithm, PhAsic self-Imitative Reduction (PAIR). PAIR substantially
outperforms both non-phasic RL and phasic SL baselines on sparse-reward
goal-conditioned robotic control problems, including a challenging stacking
task. PAIR is the first RL method that learns to stack 6 cubes with only 0/1
success rewards from scratch.