The Neural Tangent Kernel in High Dimensions: Triple Descent and a
  Multi-Scale Theory of Generalization
  Modern deep learning models employ considerably more parameters than required
to fit the training data. Whereas conventional statistical wisdom suggests such
models should drastically overfit, in practice these models generalize
remarkably well. An emerging paradigm for describing this unexpected behavior
is in terms of a \emph{double descent} curve, in which increasing a model's
capacity causes its test error to first decrease, then increase to a maximum
near the interpolation threshold, and then decrease again in the
overparameterized regime. Recent efforts to explain this phenomenon
theoretically have focused on simple settings, such as linear regression or
kernel regression with unstructured random features, which we argue are too
coarse to reveal important nuances of actual neural networks. We provide a
precise high-dimensional asymptotic analysis of generalization under kernel
regression with the Neural Tangent Kernel, which characterizes the behavior of
wide neural networks optimized with gradient descent. Our results reveal that
the test error has non-monotonic behavior deep in the overparameterized regime
and can even exhibit additional peaks and descents when the number of
parameters scales quadratically with the dataset size.
