Learning from demonstrations with SACR2: Soft Actor-Critic with Reward
  Relabeling
  During recent years, deep reinforcement learning (DRL) has made successful
incursions into complex decision-making applications such as robotics,
autonomous driving or video games. Off-policy algorithms tend to be more
sample-efficient than their on-policy counterparts, and can additionally
benefit from any off-policy data stored in the replay buffer. Expert
demonstrations are a popular source for such data: the agent is exposed to
successful states and actions early on, which can accelerate the learning
process and improve performance. In the past, multiple ideas have been proposed
to make good use of the demonstrations in the buffer, such as pretraining on
demonstrations only or minimizing additional cost functions. We carry on a
study to evaluate several of these ideas in isolation, to see which of them
have the most significant impact. We also present a new method for
sparse-reward tasks, based on a reward bonus given to demonstrations and
successful episodes. First, we give a reward bonus to the transitions coming
from demonstrations to encourage the agent to match the demonstrated behaviour.
Then, upon collecting a successful episode, we relabel its transitions with the
same bonus before adding them to the replay buffer, encouraging the agent to
also match its previous successes. The base algorithm for our experiments is
the popular Soft Actor-Critic (SAC), a state-of-the-art off-policy algorithm
for continuous action spaces. Our experiments focus on manipulation robotics,
specifically on a 3D reaching task for a robotic arm in simulation. We show
that our method SACR2 based on reward relabeling improves the performance on
this task, even in the absence of demonstrations.
