Doubly Adaptive Scaled Algorithm for Machine Learning Using Second-Order
  Information
  We present a novel adaptive optimization algorithm for large-scale machine
learning problems. Equipped with a low-cost estimate of local curvature and
Lipschitz smoothness, our method dynamically adapts the search direction and
step-size. The search direction contains gradient information preconditioned by
a well-scaled diagonal preconditioning matrix that captures the local curvature
information. Our methodology does not require the tedious task of learning rate
tuning, as the learning rate is updated automatically without adding an extra
hyperparameter. We provide convergence guarantees on a comprehensive collection
of optimization problems, including convex, strongly convex, and nonconvex
problems, in both deterministic and stochastic regimes. We also conduct an
extensive empirical evaluation on standard machine learning problems,
justifying our algorithm's versatility and demonstrating its strong performance
compared to other start-of-the-art first-order and second-order methods.
