Approximation bounds for norm constrained neural networks with
  applications to regression and GANs
  This paper studies the approximation capacity of ReLU neural networks with
norm constraint on the weights. We prove upper and lower bounds on the
approximation error of these networks for smooth function classes. The lower
bound is derived through the Rademacher complexity of neural networks, which
may be of independent interest. We apply these approximation bounds to analyze
the convergence of regression using norm constrained neural networks and
distribution estimation by GANs. In particular, we obtain convergence rates for
over-parameterized neural networks. It is also shown that GANs can achieve
optimal rate of learning probability distributions, when the discriminator is a
properly chosen norm constrained neural network.
