The Three Stages of Learning Dynamics in High-Dimensional Kernel Methods
  To understand how deep learning works, it is crucial to understand the
training dynamics of neural networks. Several interesting hypotheses about
these dynamics have been made based on empirically observed phenomena, but
there exists a limited theoretical understanding of when and why such phenomena
occur.
  In this paper, we consider the training dynamics of gradient flow on kernel
least-squares objectives, which is a limiting dynamics of SGD trained neural
networks. Using precise high-dimensional asymptotics, we characterize the
dynamics of the fitted model in two "worlds": in the Oracle World the model is
trained on the population distribution and in the Empirical World the model is
trained on a sampled dataset. We show that under mild conditions on the kernel
and $L^2$ target regression function the training dynamics undergo three stages
characterized by the behaviors of the models in the two worlds. Our theoretical
results also mathematically formalize some interesting deep learning phenomena.
Specifically, in our setting we show that SGD progressively learns more complex
functions and that there is a "deep bootstrap" phenomenon: during the second
stage, the test error of both worlds remain close despite the empirical
training error being much smaller. Finally, we give a concrete example
comparing the dynamics of two different kernels which shows that faster
training is not necessary for better generalization.
