CCLF: A Contrastive-Curiosity-Driven Learning Framework for
  Sample-Efficient Reinforcement Learning
  In reinforcement learning (RL), it is challenging to learn directly from
high-dimensional observations, where data augmentation has recently been shown
to remedy this via encoding invariances from raw pixels. Nevertheless, we
empirically find that not all samples are equally important and hence simply
injecting more augmented inputs may instead cause instability in Q-learning. In
this paper, we approach this problem systematically by developing a
model-agnostic Contrastive-Curiosity-Driven Learning Framework (CCLF), which
can fully exploit sample importance and improve learning efficiency in a
self-supervised manner. Facilitated by the proposed contrastive curiosity, CCLF
is capable of prioritizing the experience replay, selecting the most
informative augmented inputs, and more importantly regularizing the Q-function
as well as the encoder to concentrate more on under-learned data. Moreover, it
encourages the agent to explore with a curiosity-based reward. As a result, the
agent can focus on more informative samples and learn representation
invariances more efficiently, with significantly reduced augmented inputs. We
apply CCLF to several base RL algorithms and evaluate on the DeepMind Control
Suite, Atari, and MiniGrid benchmarks, where our approach demonstrates superior
sample efficiency and learning performances compared with other
state-of-the-art methods.
