Adaptive Iterative Hessian Sketch via A-Optimal Subsampling
  Iterative Hessian sketch (IHS) is an effective sketching method for modeling
large-scale data. It was originally proposed by Pilanci and Wainwright (2016;
JMLR) based on randomized sketching matrices. However, it is computationally
intensive due to the iterative sketch process. In this paper, we analyze the
IHS algorithm under the unconstrained least squares problem setting, then
propose a deterministic approach for improving IHS via A-optimal subsampling.
Our contributions are three-fold: (1) a good initial estimator based on the
A-optimal design is suggested; (2) a novel ridged preconditioner is developed
for repeated sketching; and (3) an exact line search method is proposed for
determining the optimal step length adaptively. Extensive experimental results
demonstrate that our proposed A-optimal IHS algorithm outperforms the existing
accelerated IHS methods.
