Consistent Risk Estimation in Moderately High-Dimensional Linear
  Regression
  Risk estimation is at the core of many learning systems. The importance of
this problem has motivated researchers to propose different schemes, such as
cross validation, generalized cross validation, and Bootstrap. The theoretical
properties of such estimates have been extensively studied in the
low-dimensional settings, where the number of predictors $p$ is much smaller
than the number of observations $n$. However, a unifying methodology
accompanied with a rigorous theory is lacking in high-dimensional settings.
This paper studies the problem of risk estimation under the moderately
high-dimensional asymptotic setting $n,p \rightarrow \infty$ and $n/p
\rightarrow \delta>1$ ($\delta$ is a fixed number), and proves the consistency
of three risk estimates that have been successful in numerical studies, i.e.,
leave-one-out cross validation (LOOCV), approximate leave-one-out (ALO), and
approximate message passing (AMP)-based techniques. A corner stone of our
analysis is a bound that we obtain on the discrepancy of the `residuals'
obtained from AMP and LOOCV. This connection not only enables us to obtain a
more refined information on the estimates of AMP, ALO, and LOOCV, but also
offers an upper bound on the convergence rate of each estimate.
