Improving Disentangled Text Representation Learning with
  Information-Theoretic Guidance
  Learning disentangled representations of natural language is essential for
many NLP tasks, e.g., conditional text generation, style transfer, personalized
dialogue systems, etc. Similar problems have been studied extensively for other
forms of data, such as images and videos. However, the discrete nature of
natural language makes the disentangling of textual representations more
challenging (e.g., the manipulation over the data space cannot be easily
achieved). Inspired by information theory, we propose a novel method that
effectively manifests disentangled representations of text, without any
supervision on semantics. A new mutual information upper bound is derived and
leveraged to measure dependence between style and content. By minimizing this
upper bound, the proposed method induces style and content embeddings into two
independent low-dimensional spaces. Experiments on both conditional text
generation and text-style transfer demonstrate the high quality of our
disentangled representation in terms of content and style preservation.
