Accelerating Hessian-free optimization for deep neural networks by
  implicit preconditioning and sampling
  Hessian-free training has become a popular parallel second or- der
optimization technique for Deep Neural Network training. This study aims at
speeding up Hessian-free training, both by means of decreasing the amount of
data used for training, as well as through reduction of the number of Krylov
subspace solver iterations used for implicit estimation of the Hessian. In this
paper, we develop an L-BFGS based preconditioning scheme that avoids the need
to access the Hessian explicitly. Since L-BFGS cannot be regarded as a
fixed-point iteration, we further propose the employment of flexible Krylov
subspace solvers that retain the desired theoretical convergence guarantees of
their conventional counterparts. Second, we propose a new sampling algorithm,
which geometrically increases the amount of data utilized for gradient and
Krylov subspace iteration calculations. On a 50-hr English Broadcast News task,
we find that these methodologies provide roughly a 1.5x speed-up, whereas, on a
300-hr Switchboard task, these techniques provide over a 2.3x speedup, with no
loss in WER. These results suggest that even further speed-up is expected, as
problems scale and complexity grows.
