A Theory of the Inductive Bias and Generalization of Kernel Regression
  and Wide Neural Networks
  Kernel regression is an important nonparametric learning algorithm with an
equivalence to neural networks in the infinite-width limit. Understanding its
generalization behavior is thus an important task for machine learning theory.
In this work, we provide a theory of the inductive bias and generalization of
kernel regression using a new measure characterizing the "learnability" of a
given target function. We prove that a kernel's inductive bias can be
characterized as a fixed budget of learnability, allocated to its eigenmodes,
that can only be increased with the addition of more training data. We then use
this rule to derive expressions for the mean and covariance of the predicted
function and gain insight into the overfitting and adversarial robustness of
kernel regression and the hardness of the classic parity problem. We show
agreement between our theoretical results and both kernel regression and wide
finite networks on real and synthetic learning tasks.
