Jointly Pre-training with Supervised, Autoencoder, and Value Losses for
  Deep Reinforcement Learning
  Deep Reinforcement Learning (DRL) algorithms are known to be data
inefficient. One reason is that a DRL agent learns both the feature and the
policy tabula rasa. Integrating prior knowledge into DRL algorithms is one way
to improve learning efficiency since it helps to build helpful representations.
In this work, we consider incorporating human knowledge to accelerate the
asynchronous advantage actor-critic (A3C) algorithm by pre-training a small
amount of non-expert human demonstrations. We leverage the supervised
autoencoder framework and propose a novel pre-training strategy that jointly
trains a weighted supervised classification loss, an unsupervised
reconstruction loss, and an expected return loss. The resulting pre-trained
model learns more useful features compared to independently training in
supervised or unsupervised fashion. Our pre-training method drastically
improved the learning performance of the A3C agent in Atari games of Pong and
MsPacman, exceeding the performance of the state-of-the-art algorithms at a
much smaller number of game interactions. Our method is light-weight and easy
to implement in a single machine. For reproducibility, our code is available at
github.com/gabrieledcjr/DeepRL/tree/A3C-ALA2019
