Self-Tuning Stochastic Optimization with Curvature-Aware Gradient
  Filtering
  Standard first-order stochastic optimization algorithms base their updates
solely on the average mini-batch gradient, and it has been shown that tracking
additional quantities such as the curvature can help de-sensitize common
hyperparameters. Based on this intuition, we explore the use of exact
per-sample Hessian-vector products and gradients to construct optimizers that
are self-tuning and hyperparameter-free. Based on a dynamics model of the
gradient, we derive a process which leads to a curvature-corrected,
noise-adaptive online gradient estimate. The smoothness of our updates makes it
more amenable to simple step size selection schemes, which we also base off of
our estimates quantities. We prove that our model-based procedure converges in
the noisy quadratic setting. Though we do not see similar gains in deep
learning tasks, we can match the performance of well-tuned optimizers and
ultimately, this is an interesting step for constructing self-tuning
optimizers.
