On the Power of Over-parametrization in Neural Networks with Quadratic
  Activation
  We provide new theoretical insights on why over-parametrization is effective
in learning neural networks. For a $k$ hidden node shallow network with
quadratic activation and $n$ training data points, we show as long as $ k \ge
\sqrt{2n}$, over-parametrization enables local search algorithms to find a
\emph{globally} optimal solution for general smooth and convex loss functions.
Further, despite that the number of parameters may exceed the sample size,
using theory of Rademacher complexity, we show with weight decay, the solution
also generalizes well if the data is sampled from a regular distribution such
as Gaussian. To prove when $k\ge \sqrt{2n}$, the loss function has benign
landscape properties, we adopt an idea from smoothed analysis, which may have
other applications in studying loss surfaces of neural networks.
