On Predictive Information in RNNs
  Certain biological neurons demonstrate a remarkable capability to optimally
compress the history of sensory inputs while being maximally informative about
the future. In this work, we investigate if the same can be said of artificial
neurons in recurrent neural networks (RNNs) trained with maximum likelihood.
Empirically, we find that RNNs are suboptimal in the information plane. Instead
of optimally compressing past information, they extract additional information
that is not relevant for predicting the future. We show that constraining past
information by injecting noise into the hidden state can improve RNNs in
several ways: optimality in the predictive information plane, sample quality,
heldout likelihood, and downstream classification performance.
