On Uniform Convergence and Low-Norm Interpolation Learning
  We consider an underdetermined noisy linear regression model where the
minimum-norm interpolating predictor is known to be consistent, and ask: can
uniform convergence in a norm ball, or at least (following Nagarajan and
Kolter) the subset of a norm ball that the algorithm selects on a typical input
set, explain this success? We show that uniformly bounding the difference
between empirical and population errors cannot show any learning in the norm
ball, and cannot show consistency for any set, even one depending on the exact
algorithm and distribution. But we argue we can explain the consistency of the
minimal-norm interpolator with a slightly weaker, yet standard, notion: uniform
convergence of zero-error predictors in a norm ball. We use this to bound the
generalization error of low- (but not minimal-) norm interpolating predictors.
