Dynamical Isometry and a Mean Field Theory of LSTMs and GRUs
  Training recurrent neural networks (RNNs) on long sequence tasks is plagued
with difficulties arising from the exponential explosion or vanishing of
signals as they propagate forward or backward through the network. Many
techniques have been proposed to ameliorate these issues, including various
algorithmic and architectural modifications. Two of the most successful RNN
architectures, the LSTM and the GRU, do exhibit modest improvements over
vanilla RNN cells, but they still suffer from instabilities when trained on
very long sequences. In this work, we develop a mean field theory of signal
propagation in LSTMs and GRUs that enables us to calculate the time scales for
signal propagation as well as the spectral properties of the state-to-state
Jacobians. By optimizing these quantities in terms of the initialization
hyperparameters, we derive a novel initialization scheme that eliminates or
reduces training instabilities. We demonstrate the efficacy of our
initialization scheme on multiple sequence tasks, on which it enables
successful training while a standard initialization either fails completely or
is orders of magnitude slower. We also observe a beneficial effect on
generalization performance using this new initialization.
