On Lyapunov Exponents for RNNs: Understanding Information Propagation
  Using Dynamical Systems Tools
  Recurrent neural networks (RNNs) have been successfully applied to a variety
of problems involving sequential data, but their optimization is sensitive to
parameter initialization, architecture, and optimizer hyperparameters.
Considering RNNs as dynamical systems, a natural way to capture stability,
i.e., the growth and decay over long iterates, are the Lyapunov Exponents
(LEs), which form the Lyapunov spectrum. The LEs have a bearing on stability of
RNN training dynamics because forward propagation of information is related to
the backward propagation of error gradients. LEs measure the asymptotic rates
of expansion and contraction of nonlinear system trajectories, and generalize
stability analysis to the time-varying attractors structuring the
non-autonomous dynamics of data-driven RNNs. As a tool to understand and
exploit stability of training dynamics, the Lyapunov spectrum fills an existing
gap between prescriptive mathematical approaches of limited scope and
computationally-expensive empirical approaches. To leverage this tool, we
implement an efficient way to compute LEs for RNNs during training, discuss the
aspects specific to standard RNN architectures driven by typical sequential
datasets, and show that the Lyapunov spectrum can serve as a robust readout of
training stability across hyperparameters. With this exposition-oriented
contribution, we hope to draw attention to this understudied, but theoretically
grounded tool for understanding training stability in RNNs.
