Overparameterization and generalization error: weighted trigonometric
  interpolation
  Motivated by surprisingly good generalization properties of learned deep
neural networks in overparameterized scenarios and by the related double
descent phenomenon, this paper analyzes the relation between smoothness and low
generalization error in an overparameterized linear learning problem. We study
a random Fourier series model, where the task is to estimate the unknown
Fourier coefficients from equidistant samples. We derive exact expressions for
the generalization error of both plain and weighted least squares estimators.
We show precisely how a bias towards smooth interpolants, in the form of
weighted trigonometric interpolation, can lead to smaller generalization error
in the overparameterized regime compared to the underparameterized regime. This
provides insight into the power of overparameterization, which is common in
modern machine learning.
