Adaptive Newton Sketch: Linear-time Optimization with Quadratic
  Convergence and Effective Hessian Dimensionality
  We propose a randomized algorithm with quadratic convergence rate for convex
optimization problems with a self-concordant, composite, strongly convex
objective function. Our method is based on performing an approximate Newton
step using a random projection of the Hessian. Our first contribution is to
show that, at each iteration, the embedding dimension (or sketch size) can be
as small as the effective dimension of the Hessian matrix. Leveraging this
novel fundamental result, we design an algorithm with a sketch size
proportional to the effective dimension and which exhibits a quadratic rate of
convergence. This result dramatically improves on the classical
linear-quadratic convergence rates of state-of-the-art sub-sampled Newton
methods. However, in most practical cases, the effective dimension is not known
beforehand, and this raises the question of how to pick a sketch size as small
as the effective dimension while preserving a quadratic convergence rate. Our
second and main contribution is thus to propose an adaptive sketch size
algorithm with quadratic convergence rate and which does not require prior
knowledge or estimation of the effective dimension: at each iteration, it
starts with a small sketch size, and increases it until quadratic progress is
achieved. Importantly, we show that the embedding dimension remains
proportional to the effective dimension throughout the entire path and that our
method achieves state-of-the-art computational complexity for solving convex
optimization programs with a strongly convex component.
