MomentumRNN: Integrating Momentum into Recurrent Neural Networks
  Designing deep neural networks is an art that often involves an expensive
search over candidate architectures. To overcome this for recurrent neural nets
(RNNs), we establish a connection between the hidden state dynamics in an RNN
and gradient descent (GD). We then integrate momentum into this framework and
propose a new family of RNNs, called {\em MomentumRNNs}. We theoretically prove
and numerically demonstrate that MomentumRNNs alleviate the vanishing gradient
issue in training RNNs. We study the momentum long-short term memory
(MomentumLSTM) and verify its advantages in convergence speed and accuracy over
its LSTM counterpart across a variety of benchmarks. We also demonstrate that
MomentumRNN is applicable to many types of recurrent cells, including those in
the state-of-the-art orthogonal RNNs. Finally, we show that other advanced
momentum-based optimization methods, such as Adam and Nesterov accelerated
gradients with a restart, can be easily incorporated into the MomentumRNN
framework for designing new recurrent cells with even better performance. The
code is available at https://github.com/minhtannguyen/MomentumRNN.
