Diagnosing and Fixing Manifold Overfitting in Deep Generative Models
  Likelihood-based, or explicit, deep generative models use neural networks to
construct flexible high-dimensional densities. This formulation directly
contradicts the manifold hypothesis, which states that observed data lies on a
low-dimensional manifold embedded in high-dimensional ambient space. In this
paper we investigate the pathologies of maximum-likelihood training in the
presence of this dimensionality mismatch. We formally prove that degenerate
optima are achieved wherein the manifold itself is learned but not the
distribution on it, a phenomenon we call manifold overfitting. We propose a
class of two-step procedures consisting of a dimensionality reduction step
followed by maximum-likelihood density estimation, and prove that they recover
the data-generating distribution in the nonparametric regime, thus avoiding
manifold overfitting. We also show that these procedures enable density
estimation on the manifolds learned by implicit models, such as generative
adversarial networks, hence addressing a major shortcoming of these models.
Several recently proposed methods are instances of our two-step procedures; we
thus unify, extend, and theoretically justify a large class of models.
