Memorize to Generalize: on the Necessity of Interpolation in High
  Dimensional Linear Regression
  We examine the necessity of interpolation in overparameterized models, that
is, when achieving optimal predictive risk in machine learning problems
requires (nearly) interpolating the training data. In particular, we consider
simple overparameterized linear regression $y = X \theta + w$ with random
design $X \in \mathbb{R}^{n \times d}$ under the proportional asymptotics $d/n
\to \gamma \in (1, \infty)$. We precisely characterize how prediction (test)
error necessarily scales with training error in this setting. An implication of
this characterization is that as the label noise variance $\sigma^2 \to 0$, any
estimator that incurs at least $\mathsf{c}\sigma^4$ training error for some
constant $\mathsf{c}$ is necessarily suboptimal and will suffer growth in
excess prediction error at least linear in the training error. Thus, optimal
performance requires fitting training data to substantially higher accuracy
than the inherent noise floor of the problem.
