Trust-Region Algorithms for Training Responses: Machine Learning Methods
  Using Indefinite Hessian Approximations
  Machine learning (ML) problems are often posed as highly nonlinear and
nonconvex unconstrained optimization problems. Methods for solving ML problems
based on stochastic gradient descent are easily scaled for very large problems
but may involve fine-tuning many hyper-parameters. Quasi-Newton approaches
based on the limited-memory Broyden-Fletcher-Goldfarb-Shanno (BFGS) update
typically do not require manually tuning hyper-parameters but suffer from
approximating a potentially indefinite Hessian with a positive-definite matrix.
Hessian-free methods leverage the ability to perform Hessian-vector
multiplication without needing the entire Hessian matrix, but each iteration's
complexity is significantly greater than quasi-Newton methods. In this paper we
propose an alternative approach for solving ML problems based on a quasi-Newton
trust-region framework for solving large-scale optimization problems that allow
for indefinite Hessian approximations. Numerical experiments on a standard
testing data set show that with a fixed computational time budget, the proposed
methods achieve better results than the traditional limited-memory BFGS and the
Hessian-free methods.
