Just Interpolate: Kernel "Ridgeless" Regression Can Generalize
  In the absence of explicit regularization, Kernel "Ridgeless" Regression with
nonlinear kernels has the potential to fit the training data perfectly. It has
been observed empirically, however, that such interpolated solutions can still
generalize well on test data. We isolate a phenomenon of implicit
regularization for minimum-norm interpolated solutions which is due to a
combination of high dimensionality of the input data, curvature of the kernel
function, and favorable geometric properties of the data such as an eigenvalue
decay of the empirical covariance and kernel matrices. In addition to deriving
a data-dependent upper bound on the out-of-sample error, we present
experimental evidence suggesting that the phenomenon occurs in the MNIST
dataset.
