Optimal Iterative Sketching with the Subsampled Randomized Hadamard
  Transform
  Random projections or sketching are widely used in many algorithmic and
learning contexts. Here we study the performance of iterative Hessian sketch
for least-squares problems. By leveraging and extending recent results from
random matrix theory on the limiting spectrum of matrices randomly projected
with the subsampled randomized Hadamard transform, and truncated Haar matrices,
we can study and compare the resulting algorithms to a level of precision that
has not been possible before. Our technical contributions include a novel
formula for the second moment of the inverse of projected matrices. We also
find simple closed-form expressions for asymptotically optimal step-sizes and
convergence rates. These show that the convergence rate for Haar and randomized
Hadamard matrices are identical, and asymptotically improve upon Gaussian
random projections. These techniques may be applied to other algorithms that
employ randomized dimension reduction.
