Influence-aware Memory Architectures for Deep Reinforcement Learning
  Due to its perceptual limitations, an agent may have too little information
about the state of the environment to act optimally. In such cases, it is
important to keep track of the observation history to uncover hidden state.
Recent deep reinforcement learning methods use recurrent neural networks (RNN)
to memorize past observations. However, these models are expensive to train and
have convergence difficulties, especially when dealing with high dimensional
input spaces. In this paper, we propose influence-aware memory (IAM), a
theoretically inspired memory architecture that tries to alleviate the training
difficulties by restricting the input of the recurrent layers to those
variables that influence the hidden state information. Moreover, as opposed to
standard RNNs, in which every piece of information used for estimating Q values
is inevitably fed back into the network for the next prediction, our model
allows information to flow without being necessarily stored in the RNN's
internal memory. Results indicate that, by letting the recurrent layers focus
on a small fraction of the observation variables while processing the rest of
the information with a feedforward neural network, we can outperform standard
recurrent architectures both in training speed and policy performance. This
approach also reduces runtime and obtains better scores than methods that stack
multiple observations to remove partial observability.
