Kernel Stein Generative Modeling
  We are interested in gradient-based Explicit Generative Modeling where
samples can be derived from iterative gradient updates based on an estimate of
the score function of the data distribution. Recent advances in Stochastic
Gradient Langevin Dynamics (SGLD) demonstrates impressive results with
energy-based models on high-dimensional and complex data distributions. Stein
Variational Gradient Descent (SVGD) is a deterministic sampling algorithm that
iteratively transports a set of particles to approximate a given distribution,
based on functional gradient descent that decreases the KL divergence. SVGD has
promising results on several Bayesian inference applications. However, applying
SVGD on high dimensional problems is still under-explored. The goal of this
work is to study high dimensional inference with SVGD. We first identify key
challenges in practical kernel SVGD inference in high-dimension. We propose
noise conditional kernel SVGD (NCK-SVGD), that works in tandem with the
recently introduced Noise Conditional Score Network estimator. NCK is crucial
for successful inference with SVGD in high dimension, as it adapts the kernel
to the noise level of the score estimate. As we anneal the noise, NCK-SVGD
targets the real data distribution. We then extend the annealed SVGD with an
entropic regularization. We show that this offers a flexible control between
sample quality and diversity, and verify it empirically by precision and recall
evaluations. The NCK-SVGD produces samples comparable to GANs and annealed SGLD
on computer vision benchmarks, including MNIST and CIFAR-10.
