SURF: Semi-supervised Reward Learning with Data Augmentation for
  Feedback-efficient Preference-based Reinforcement Learning
  Preference-based reinforcement learning (RL) has shown potential for teaching
agents to perform the target tasks without a costly, pre-defined reward
function by learning the reward with a supervisor's preference between the two
agent behaviors. However, preference-based learning often requires a large
amount of human feedback, making it difficult to apply this approach to various
applications. This data-efficiency problem, on the other hand, has been
typically addressed by using unlabeled samples or data augmentation techniques
in the context of supervised learning. Motivated by the recent success of these
approaches, we present SURF, a semi-supervised reward learning framework that
utilizes a large amount of unlabeled samples with data augmentation. In order
to leverage unlabeled samples for reward learning, we infer pseudo-labels of
the unlabeled samples based on the confidence of the preference predictor. To
further improve the label-efficiency of reward learning, we introduce a new
data augmentation that temporally crops consecutive subsequences from the
original behaviors. Our experiments demonstrate that our approach significantly
improves the feedback-efficiency of the state-of-the-art preference-based
method on a variety of locomotion and robotic manipulation tasks.
