Iterative Hessian sketch: Fast and accurate solution approximation for
  constrained least-squares
  We study randomized sketching methods for approximately solving least-squares
problem with a general convex constraint. The quality of a least-squares
approximation can be assessed in different ways: either in terms of the value
of the quadratic objective function (cost approximation), or in terms of some
distance measure between the approximate minimizer and the true minimizer
(solution approximation). Focusing on the latter criterion, our first main
result provides a general lower bound on any randomized method that sketches
both the data matrix and vector in a least-squares problem; as a surprising
consequence, the most widely used least-squares sketch is sub-optimal for
solution approximation. We then present a new method known as the iterative
Hessian sketch, and show that it can be used to obtain approximations to the
original least-squares problem using a projection dimension proportional to the
statistical complexity of the least-squares minimizer, and a logarithmic number
of iterations. We illustrate our general theory with simulations for both
unconstrained and constrained versions of least-squares, including
$\ell_1$-regularization and nuclear norm constraints. We also numerically
demonstrate the practicality of our approach in a real face expression
classification experiment.
