Spectral Analysis of the Neural Tangent Kernel for Deep Residual
  Networks
  Deep residual network architectures have been shown to achieve superior
accuracy over classical feed-forward networks, yet their success is still not
fully understood. Focusing on massively over-parameterized, fully connected
residual networks with ReLU activation through their respective neural tangent
kernels (ResNTK), we provide here a spectral analysis of these kernels.
Specifically, we show that, much like NTK for fully connected networks
(FC-NTK), for input distributed uniformly on the hypersphere
$\mathbb{S}^{d-1}$, the eigenfunctions of ResNTK are the spherical harmonics
and the eigenvalues decay polynomially with frequency $k$ as $k^{-d}$. These in
turn imply that the set of functions in their Reproducing Kernel Hilbert Space
are identical to those of FC-NTK, and consequently also to those of the Laplace
kernel. We further show, by drawing on the analogy to the Laplace kernel, that
depending on the choice of a hyper-parameter that balances between the skip and
residual connections ResNTK can either become spiky with depth, as with FC-NTK,
or maintain a stable shape.
