Lyapunov-Guided Embedding for Hyperparameter Selection in Recurrent
  Neural Networks
  Recurrent Neural Networks (RNN) are ubiquitous computing systems for
sequences and multivariate time series data. While several robust architectures
of RNN are known, it is unclear how to relate RNN initialization, architecture,
and other hyperparameters with accuracy for a given task. In this work, we
propose to treat RNN as dynamical systems and to correlate hyperparameters with
accuracy through Lyapunov spectral analysis, a methodology specifically
designed for nonlinear dynamical systems. To address the fact that RNN features
go beyond the existing Lyapunov spectral analysis, we propose to infer relevant
features from the Lyapunov spectrum with an Autoencoder and an embedding of its
latent representation (AeLLE). Our studies of various RNN architectures show
that AeLLE successfully correlates RNN Lyapunov spectrum with accuracy.
Furthermore, the latent representation learned by AeLLE is generalizable to
novel inputs from the same task and is formed early in the process of RNN
training. The latter property allows for the prediction of the accuracy to
which RNN would converge when training is complete. We conclude that
representation of RNN through Lyapunov spectrum along with AeLLE, and assists
with hyperparameter selection of RNN, provides a novel method for organization
and interpretation of variants of RNN architectures.
