Max-Affine Spline Insights into Deep Generative Networks
  We connect a large class of Generative Deep Networks (GDNs) with spline
operators in order to derive their properties, limitations, and new
opportunities. By characterizing the latent space partition, dimension and
angularity of the generated manifold, we relate the manifold dimension and
approximation error to the sample size. The manifold-per-region affine subspace
defines a local coordinate basis; we provide necessary and sufficient
conditions relating those basis vectors with disentanglement. We also derive
the output probability density mapped onto the generated manifold in terms of
the latent space density, which enables the computation of key statistics such
as its Shannon entropy. This finding also enables the computation of the GDN
likelihood, which provides a new mechanism for model comparison as well as
providing a quality measure for (generated) samples under the learned
distribution. We demonstrate how low entropy and/or multimodal distributions
are not naturally modeled by DGNs and are a cause of training instabilities.
