A Stochastic Variance Reduced Nesterov's Accelerated Quasi-Newton Method
  Recently algorithms incorporating second order curvature information have
become popular in training neural networks. The Nesterov's Accelerated
Quasi-Newton (NAQ) method has shown to effectively accelerate the BFGS
quasi-Newton method by incorporating the momentum term and Nesterov's
accelerated gradient vector. A stochastic version of NAQ method was proposed
for training of large-scale problems. However, this method incurs high
stochastic variance noise. This paper proposes a stochastic variance reduced
Nesterov's Accelerated Quasi-Newton method in full (SVR-NAQ) and limited
(SVRLNAQ) memory forms. The performance of the proposed method is evaluated in
Tensorflow on four benchmark problems - two regression and two classification
problems respectively. The results show improved performance compared to
conventional methods.
