An Adaptive Stochastic Nesterov Accelerated Quasi Newton Method for
  Training RNNs
  A common problem in training neural networks is the vanishing and/or
exploding gradient problem which is more prominently seen in training of
Recurrent Neural Networks (RNNs). Thus several algorithms have been proposed
for training RNNs. This paper proposes a novel adaptive stochastic Nesterov
accelerated quasiNewton (aSNAQ) method for training RNNs. The proposed method
aSNAQ is an accelerated method that uses the Nesterov's gradient term along
with second order curvature information. The performance of the proposed method
is evaluated in Tensorflow on benchmark sequence modeling problems. The results
show an improved performance while maintaining a low per-iteration cost and
thus can be effectively used to train RNNs.
