Constructing Gradient Controllable Recurrent Neural Networks Using
  Hamiltonian Dynamics
  Recurrent neural networks (RNNs) have gained a great deal of attention in
solving sequential learning problems. The learning of long-term dependencies,
however, remains challenging due to the problem of a vanishing or exploding
hidden states gradient. By exploring further the recently established
connections between RNNs and dynamical systems we propose a novel RNN
architecture, which we call a Hamiltonian recurrent neural network (Hamiltonian
RNN), based on a symplectic discretization of an appropriately chosen
Hamiltonian system. The key benefit of this approach is that the corresponding
RNN inherits the favorable long time properties of the Hamiltonian system,
which in turn allows us to control the hidden states gradient with a
hyperparameter of the Hamiltonian RNN architecture. This enables us to handle
sequential learning problems with arbitrary sequence lengths, since for a range
of values of this hyperparameter the gradient neither vanishes nor explodes.
Additionally, we provide a heuristic for the optimal choice of the
hyperparameter, which we use in our numerical simulations to illustrate that
the Hamiltonian RNN is able to outperform other state-of-the-art RNNs without
the need of computationally intensive hyperparameter optimization.
