How Much Over-parameterization Is Sufficient to Learn Deep ReLU
  Networks?
  A recent line of research on deep learning focuses on the extremely
over-parameterized setting, and shows that when the network width is larger
than a high degree polynomial of the training sample size $n$ and the inverse
of the target error $\epsilon^{-1}$, deep neural networks learned by
(stochastic) gradient descent enjoy nice optimization and generalization
guarantees. Very recently, it is shown that under certain margin assumptions on
the training data, a polylogarithmic width condition suffices for two-layer
ReLU networks to converge and generalize (Ji and Telgarsky, 2019). However,
whether deep neural networks can be learned with such a mild
over-parameterization is still an open question. In this work, we answer this
question affirmatively and establish sharper learning guarantees for deep ReLU
networks trained by (stochastic) gradient descent. In specific, under certain
assumptions made in previous work, our optimization and generalization
guarantees hold with network width polylogarithmic in $n$ and $\epsilon^{-1}$.
Our results push the study of over-parameterized deep neural networks towards
more practical settings.
