A Universal Approximation Theorem of Deep Neural Networks for Expressing
  Probability Distributions
  This paper studies the universal approximation property of deep neural
networks for representing probability distributions. Given a target
distribution $\pi$ and a source distribution $p_z$ both defined on
$\mathbb{R}^d$, we prove under some assumptions that there exists a deep neural
network $g:\mathbb{R}^d\rightarrow \mathbb{R}$ with ReLU activation such that
the push-forward measure $(\nabla g)_\# p_z$ of $p_z$ under the map $\nabla g$
is arbitrarily close to the target measure $\pi$. The closeness are measured by
three classes of integral probability metrics between probability
distributions: $1$-Wasserstein distance, maximum mean distance (MMD) and
kernelized Stein discrepancy (KSD). We prove upper bounds for the size (width
and depth) of the deep neural network in terms of the dimension $d$ and the
approximation error $\varepsilon$ with respect to the three discrepancies. In
particular, the size of neural network can grow exponentially in $d$ when
$1$-Wasserstein distance is used as the discrepancy, whereas for both MMD and
KSD the size of neural network only depends on $d$ at most polynomially. Our
proof relies on convergence estimates of empirical measures under
aforementioned discrepancies and semi-discrete optimal transport.
