Deep Equals Shallow for ReLU Networks in Kernel Regimes
  Deep networks are often considered to be more expressive than shallow ones in
terms of approximation. Indeed, certain functions can be approximated by deep
networks provably more efficiently than by shallow ones, however, no tractable
algorithms are known for learning such deep models. Separately, a recent line
of work has shown that deep networks trained with gradient descent may behave
like (tractable) kernel methods in a certain over-parameterized regime, where
the kernel is determined by the architecture and initialization, and this paper
focuses on approximation for such kernels. We show that for ReLU activations,
the kernels derived from deep fully-connected networks have essentially the
same approximation properties as their shallow two-layer counterpart, namely
the same eigenvalue decay for the corresponding integral operator. This
highlights the limitations of the kernel framework for understanding the
benefits of such deep architectures. Our main theoretical result relies on
characterizing such eigenvalue decays through differentiability properties of
the kernel function, which also easily applies to the study of other kernels
defined on the sphere.
