On Posterior Collapse and Encoder Feature Dispersion in Sequence VAEs
  Variational autoencoders (VAEs) hold great potential for modelling text, as
they could in theory separate high-level semantic and syntactic properties from
local regularities of natural language. Practically, however, VAEs with
autoregressive decoders often suffer from posterior collapse, a phenomenon
where the model learns to ignore the latent variables, causing the sequence VAE
to degenerate into a language model. In this paper, we argue that posterior
collapse is in part caused by the lack of dispersion in encoder features. We
provide empirical evidence to verify this hypothesis, and propose a
straightforward fix using pooling. This simple technique effectively prevents
posterior collapse, allowing model to achieve significantly better data
log-likelihood than standard sequence VAEs. Comparing to existing work, our
proposed method is able to achieve comparable or superior performances while
being more computationally efficient.
