Neural Tangent Kernel: Convergence and Generalization in Neural Networks
  At initialization, artificial neural networks (ANNs) are equivalent to
Gaussian processes in the infinite-width limit, thus connecting them to kernel
methods. We prove that the evolution of an ANN during training can also be
described by a kernel: during gradient descent on the parameters of an ANN, the
network function $f_\theta$ (which maps input vectors to output vectors)
follows the kernel gradient of the functional cost (which is convex, in
contrast to the parameter cost) w.r.t. a new kernel: the Neural Tangent Kernel
(NTK). This kernel is central to describe the generalization features of ANNs.
While the NTK is random at initialization and varies during training, in the
infinite-width limit it converges to an explicit limiting kernel and it stays
constant during training. This makes it possible to study the training of ANNs
in function space instead of parameter space. Convergence of the training can
then be related to the positive-definiteness of the limiting NTK. We prove the
positive-definiteness of the limiting NTK when the data is supported on the
sphere and the non-linearity is non-polynomial. We then focus on the setting of
least-squares regression and show that in the infinite-width limit, the network
function $f_\theta$ follows a linear differential equation during training. The
convergence is fastest along the largest kernel principal components of the
input data with respect to the NTK, hence suggesting a theoretical motivation
for early stopping. Finally we study the NTK numerically, observe its behavior
for wide networks, and compare it to the infinite-width limit.
