Over Parameterized Two-level Neural Networks Can Learn Near Optimal
  Feature Representations
  Recently, over-parameterized neural networks have been extensively analyzed
in the literature. However, the previous studies cannot satisfactorily explain
why fully trained neural networks are successful in practice. In this paper, we
present a new theoretical framework for analyzing over-parameterized neural
networks which we call neural feature repopulation. Our analysis can
satisfactorily explain the empirical success of two level neural networks that
are trained by standard learning algorithms. Our key theoretical result is that
in the limit of infinite number of hidden neurons, over-parameterized two-level
neural networks trained via the standard (noisy) gradient descent learns a
well-defined feature distribution (population), and the limiting feature
distribution is nearly optimal for the underlying learning task under certain
conditions. Empirical studies confirm that predictions of our theory are
consistent with the results observed in real practice.
