Learning Temporally-Consistent Representations for Data-Efficient
  Reinforcement Learning
  Deep reinforcement learning (RL) agents that exist in high-dimensional state
spaces, such as those composed of images, have interconnected learning burdens.
Agents must learn an action-selection policy that completes their given task,
which requires them to learn a representation of the state space that discerns
between useful and useless information. The reward function is the only
supervised feedback that RL agents receive, which causes a representation
learning bottleneck that can manifest in poor sample efficiency. We present
$k$-Step Latent (KSL), a new representation learning method that enforces
temporal consistency of representations via a self-supervised auxiliary task
wherein agents learn to recurrently predict action-conditioned representations
of the state space. The state encoder learned by KSL produces low-dimensional
representations that make optimization of the RL task more sample efficient.
Altogether, KSL produces state-of-the-art results in both data efficiency and
asymptotic performance in the popular PlaNet benchmark suite. Our analyses show
that KSL produces encoders that generalize better to new tasks unseen during
training, and its representations are more strongly tied to reward, are more
invariant to perturbations in the state space, and move more smoothly through
the temporal axis of the RL problem than other methods such as DrQ, RAD, CURL,
and SAC-AE.
