On the Power and Limitations of Random Features for Understanding Neural
  Networks
  Recently, a spate of papers have provided positive theoretical results for
training over-parameterized neural networks (where the network size is larger
than what is needed to achieve low error). The key insight is that with
sufficient over-parameterization, gradient-based methods will implicitly leave
some components of the network relatively unchanged, so the optimization
dynamics will behave as if those components are essentially fixed at their
initial random values. In fact, fixing these explicitly leads to the well-known
approach of learning with random features. In other words, these techniques
imply that we can successfully learn with neural networks, whenever we can
successfully learn with random features. In this paper, we first review these
techniques, providing a simple and self-contained analysis for one-hidden-layer
networks. We then argue that despite the impressive positive results, random
feature approaches are also inherently limited in what they can explain. In
particular, we rigorously show that random features cannot be used to learn
even a single ReLU neuron with standard Gaussian inputs, unless the network
size (or magnitude of the weights) is exponentially large. Since a single
neuron is learnable with gradient-based methods, we conclude that we are still
far from a satisfying general explanation for the empirical success of neural
networks.
