Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables
  Signal Propagation in Recurrent Neural Networks
  Recurrent neural networks have gained widespread use in modeling sequence
data across various domains. While many successful recurrent architectures
employ a notion of gating, the exact mechanism that enables such remarkable
performance is not well understood. We develop a theory for signal propagation
in recurrent networks after random initialization using a combination of mean
field theory and random matrix theory. To simplify our discussion, we introduce
a new RNN cell with a simple gating mechanism that we call the minimalRNN and
compare it with vanilla RNNs. Our theory allows us to define a maximum
timescale over which RNNs can remember an input. We show that this theory
predicts trainability for both recurrent architectures. We show that gated
recurrent networks feature a much broader, more robust, trainable region than
vanilla RNNs, which corroborates recent experimental findings. Finally, we
develop a closed-form critical initialization scheme that achieves dynamical
isometry in both vanilla RNNs and minimalRNNs. We show that this results in
significantly improvement in training dynamics. Finally, we demonstrate that
the minimalRNN achieves comparable performance to its more complex
counterparts, such as LSTMs or GRUs, on a language modeling task.
