On Optimal Interpolation In Linear Regression
  Understanding when and why interpolating methods generalize well has recently
been a topic of interest in statistical learning theory. However,
systematically connecting interpolating methods to achievable notions of
optimality has only received partial attention. In this paper, we investigate
the question of what is the optimal way to interpolate in linear regression
using functions that are linear in the response variable (as the case for the
Bayes optimal estimator in ridge regression) and depend on the data, the
population covariance of the data, the signal-to-noise ratio and the covariance
of the prior for the signal, but do not depend on the value of the signal
itself nor the noise vector in the training data. We provide a closed-form
expression for the interpolator that achieves this notion of optimality and
show that it can be derived as the limit of preconditioned gradient descent
with a specific initialization. We identify a regime where the minimum-norm
interpolator provably generalizes arbitrarily worse than the optimal
response-linear achievable interpolator that we introduce, and validate with
numerical experiments that the notion of optimality we consider can be achieved
by interpolating methods that only use the training data as input in the case
of an isotropic prior. Finally, we extend the notion of optimal response-linear
interpolation to random features regression under a linear data-generating
model that has been previously studied in the literature.
