Educating Text Autoencoders: Latent Representation Guidance via
  Denoising
  Generative autoencoders offer a promising approach for controllable text
generation by leveraging their latent sentence representations. However,
current models struggle to maintain coherent latent spaces required to perform
meaningful text manipulations via latent vector operations. Specifically, we
demonstrate by example that neural encoders do not necessarily map similar
sentences to nearby latent vectors. A theoretical explanation for this
phenomenon establishes that high capacity autoencoders can learn an arbitrary
mapping between sequences and associated latent representations. To remedy this
issue, we augment adversarial autoencoders with a denoising objective where
original sentences are reconstructed from perturbed versions (referred to as
DAAE). We prove that this simple modification guides the latent space geometry
of the resulting model by encouraging the encoder to map similar texts to
similar latent representations. In empirical comparisons with various types of
autoencoders, our model provides the best trade-off between generation quality
and reconstruction capacity. Moreover, the improved geometry of the DAAE latent
space enables zero-shot text style transfer via simple latent vector
arithmetic.
