What Can ResNet Learn Efficiently, Going Beyond Kernels?
  How can neural networks such as ResNet efficiently learn CIFAR-10 with test
accuracy more than 96%, while other methods, especially kernel methods, fall
relatively behind? Can we more provide theoretical justifications for this gap?
  Recently, there is an influential line of work relating neural networks to
kernels in the over-parameterized regime, proving they can learn certain
concept class that is also learnable by kernels with similar test error. Yet,
can neural networks provably learn some concept class BETTER than kernels?
  We answer this positively in the distribution-free setting. We prove neural
networks can efficiently learn a notable class of functions, including those
defined by three-layer residual networks with smooth activations, without any
distributional assumption. At the same time, we prove there are simple
functions in this class such that with the same number of training examples,
the test error obtained by neural networks can be MUCH SMALLER than ANY kernel
method, including neural tangent kernels (NTK).
  The main intuition is that multi-layer neural networks can implicitly perform
hierarchical learning using different layers, which reduces the sample
complexity comparing to "one-shot" learning algorithms such as kernel methods.
In a follow-up work [2], this theory of hierarchical learning is further
strengthened to incorporate the "backward feature correction" process when
training deep networks.
  In the end, we also prove a computation complexity advantage of ResNet with
respect to other learning methods including linear regression over arbitrary
feature mappings.
