Generative Text Modeling through Short Run Inference
  Latent variable models for text, when trained successfully, accurately model
the data distribution and capture global semantic and syntactic features of
sentences. The prominent approach to train such models is variational
autoencoders (VAE). It is nevertheless challenging to train and often results
in a trivial local optimum where the latent variable is ignored and its
posterior collapses into the prior, an issue known as posterior collapse.
Various techniques have been proposed to mitigate this issue. Most of them
focus on improving the inference model to yield latent codes of higher quality.
The present work proposes a short run dynamics for inference. It is initialized
from the prior distribution of the latent variable and then runs a small number
(e.g., 20) of Langevin dynamics steps guided by its posterior distribution. The
major advantage of our method is that it does not require a separate inference
model or assume simple geometry of the posterior distribution, thus rendering
an automatic, natural and flexible inference engine. We show that the models
trained with short run dynamics more accurately model the data, compared to
strong language model and VAE baselines, and exhibit no sign of posterior
collapse. Analyses of the latent space show that interpolation in the latent
space is able to generate coherent sentences with smooth transition and
demonstrate improved classification over strong baselines with latent features
from unsupervised pretraining. These results together expose a well-structured
latent space of our generative model.
