Kernel regression in high dimensions: Refined analysis beyond double
  descent
  In this paper, we provide a precise characterization of generalization
properties of high dimensional kernel ridge regression across the under- and
over-parameterized regimes, depending on whether the number of training data n
exceeds the feature dimension d. By establishing a bias-variance decomposition
of the expected excess risk, we show that, while the bias is (almost)
independent of d and monotonically decreases with n, the variance depends on n,
d and can be unimodal or monotonically decreasing under different
regularization schemes. Our refined analysis goes beyond the double descent
theory by showing that, depending on the data eigen-profile and the level of
regularization, the kernel regression risk curve can be a double-descent-like,
bell-shaped, or monotonic function of n. Experiments on synthetic and real data
are conducted to support our theoretical findings.
