Learning and Generalization in Overparameterized Normalizing Flows
  In supervised learning, it is known that overparameterized neural networks
with one hidden layer provably and efficiently learn and generalize, when
trained using stochastic gradient descent with a sufficiently small learning
rate and suitable initialization. In contrast, the benefit of
overparameterization in unsupervised learning is not well understood.
Normalizing flows (NFs) constitute an important class of models in unsupervised
learning for sampling and density estimation. In this paper, we theoretically
and empirically analyze these models when the underlying neural network is a
one-hidden-layer overparametrized network. Our main contributions are two-fold:
(1) On the one hand, we provide theoretical and empirical evidence that for
constrained NFs (this class of NFs underlies many NF constructions) with the
one-hidden-layer network, overparametrization hurts training. (2) On the other
hand, we prove that unconstrained NFs, a recently introduced model, can
efficiently learn any reasonable data distribution under minimal assumptions
when the underlying network is overparametrized and has one hidden-layer.
