A Distributed Optimisation Framework Combining Natural Gradient with
  Hessian-Free for Discriminative Sequence Training
  This paper presents a novel natural gradient and Hessian-free (NGHF)
optimisation framework for neural network training that can operate efficiently
in a distributed manner. It relies on the linear conjugate gradient (CG)
algorithm to combine the natural gradient (NG) method with local curvature
information from Hessian-free (HF) or other second-order methods. A solution to
a numerical issue in CG allows effective parameter updates to be generated with
far fewer CG iterations than usually used (e.g. 5-8 instead of 200). This work
also presents a novel preconditioning approach to improve the progress made by
individual CG iterations for models with shared parameters. Although applicable
to other training losses and model structures, NGHF is investigated in this
paper for lattice-based discriminative sequence training for hybrid hidden
Markov model acoustic models using a standard recurrent neural network, long
short-term memory, and time delay neural network models for output probability
calculation. Automatic speech recognition experiments are reported on the
multi-genre broadcast data set for a range of different acoustic model types.
These experiments show that NGHF achieves larger word error rate reductions
than standard stochastic gradient descent or Adam, while requiring orders of
magnitude fewer parameter updates.
