Learning State Representations from Random Deep Action-conditional
  Predictions
  Our main contribution in this work is an empirical finding that random
General Value Functions (GVFs), i.e., deep action-conditional predictions --
random both in what feature of observations they predict as well as in the
sequence of actions the predictions are conditioned upon -- form good auxiliary
tasks for reinforcement learning (RL) problems. In particular, we show that
random deep action-conditional predictions when used as auxiliary tasks yield
state representations that produce control performance competitive with
state-of-the-art hand-crafted auxiliary tasks like value prediction, pixel
control, and CURL in both Atari and DeepMind Lab tasks. In another set of
experiments we stop the gradients from the RL part of the network to the state
representation learning part of the network and show, perhaps surprisingly,
that the auxiliary tasks alone are sufficient to learn state representations
good enough to outperform an end-to-end trained actor-critic baseline. We
opensourced our code at https://github.com/Hwhitetooth/random_gvfs.
