Investigating the Properties of Neural Network Representations in
  Reinforcement Learning
  In this paper we investigate the properties of representations learned by
deep reinforcement learning systems. Much of the earlier work in representation
learning for reinforcement learning focused on designing fixed-basis
architectures to achieve properties thought to be desirable, such as
orthogonality and sparsity. In contrast, the idea behind deep reinforcement
learning methods is that the agent designer should not encode representational
properties, but rather that the data stream should determine the properties of
the representation -- good representations emerge under appropriate training
schemes. In this paper we bring these two perspectives together, empirically
investigating the properties of representations that support transfer in
reinforcement learning. This analysis allows us to provide novel hypotheses
regarding impact of auxiliary tasks in end-to-end training of non-linear
reinforcement learning methods. We introduce and measure six representational
properties over more than 25 thousand agent-task settings. We consider DQN
agents with convolutional networks in a pixel-based navigation environment. We
develop a method to better understand \emph{why} some representations work
better for transfer, through a systematic approach varying task similarity and
measuring and correlating representation properties with transfer performance.
