Optimistic Rates: A Unifying Theory for Interpolation Learning and
  Regularization in Linear Regression
  We study a localized notion of uniform convergence known as an "optimistic
rate" (Panchenko 2002; Srebro et al. 2010) for linear regression with Gaussian
data. Our refined analysis avoids the hidden constant and logarithmic factor in
existing results, which are known to be crucial in high-dimensional settings,
especially for understanding interpolation learning. As a special case, our
analysis recovers the guarantee from Koehler et al. (2021), which tightly
characterizes the population risk of low-norm interpolators under the benign
overfitting conditions. Our optimistic rate bound, though, also analyzes
predictors with arbitrary training error. This allows us to recover some
classical statistical guarantees for ridge and LASSO regression under random
designs, and helps us obtain a precise understanding of the excess risk of
near-interpolators in the over-parameterized regime.
