Sequence Training of DNN Acoustic Models With Natural Gradient
  Deep Neural Network (DNN) acoustic models often use discriminative sequence
training that optimises an objective function that better approximates the word
error rate (WER) than frame-based training. Sequence training is normally
implemented using Stochastic Gradient Descent (SGD) or Hessian Free (HF)
training. This paper proposes an alternative batch style optimisation framework
that employs a Natural Gradient (NG) approach to traverse through the parameter
space. By correcting the gradient according to the local curvature of the
KL-divergence, the NG optimisation process converges more quickly than HF.
Furthermore, the proposed NG approach can be applied to any sequence
discriminative training criterion. The efficacy of the NG method is shown using
experiments on a Multi-Genre Broadcast (MGB) transcription task that
demonstrates both the computational efficiency and the accuracy of the
resulting DNN models.
