A Novel Fast Exact Subproblem Solver for Stochastic Quasi-Newton Cubic
  Regularized Optimization
  In this work we describe an Adaptive Regularization using Cubics (ARC) method
for large-scale nonconvex unconstrained optimization using Limited-memory
Quasi-Newton (LQN) matrices. ARC methods are a relatively new family of
optimization strategies that utilize a cubic-regularization (CR) term in place
of trust-regions and line-searches. LQN methods offer a large-scale alternative
to using explicit second-order information by taking identical inputs to those
used by popular first-order methods such as stochastic gradient descent (SGD).
Solving the CR subproblem exactly requires Newton's method, yet using
properties of the internal structure of LQN matrices, we are able to find exact
solutions to the CR subproblem in a matrix-free manner, providing large
speedups and scaling into modern size requirements. Additionally, we expand
upon previous ARC work and explicitly incorporate first-order updates into our
algorithm. We provide experimental results when the SR1 update is used, which
show substantial speed-ups and competitive performance compared to Adam and
other second order optimizers on deep neural networks (DNNs). We find that our
new approach, ARCLQN, compares to modern optimizers with minimal tuning, a
common pain-point for second order methods.
