Stability and Risk Bounds of Iterative Hard Thresholding
  In this paper, we analyze the generalization performance of the Iterative
Hard Thresholding (IHT) algorithm widely used for sparse recovery problems. The
parameter estimation and sparsity recovery consistency of IHT has long been
known in compressed sensing. From the perspective of statistical learning,
another fundamental question is how well the IHT estimation would predict on
unseen data. This paper makes progress towards answering this open question by
introducing a novel sparse generalization theory for IHT under the notion of
algorithmic stability. Our theory reveals that: 1) under natural conditions on
the empirical risk function over $n$ samples of dimension $p$, IHT with
sparsity level $k$ enjoys an $\mathcal{\tilde
O}(n^{-1/2}\sqrt{k\log(n)\log(p)})$ rate of convergence in sparse excess risk;
2) a tighter $\mathcal{\tilde O}(n^{-1/2}\sqrt{\log(n)})$ bound can be
established by imposing an additional iteration stability condition on a
hypothetical IHT procedure invoked to the population risk; and 3) a fast rate
of order $\mathcal{\tilde O}\left(n^{-1}k(\log^3(n)+\log(p))\right)$ can be
derived for strongly convex risk function under proper strong-signal
conditions. The results have been substantialized to sparse linear regression
and sparse logistic regression models to demonstrate the applicability of our
theory. Preliminary numerical evidence is provided to confirm our theoretical
predictions.
