{'fraction_non_alphanumeric': 0.01195219123505976, 'fraction_numerical': 0.0, 'mean_word_length': 5.444444444444445, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Challenges with Variational Autoencoders for Text

We study variational autoencoders for text data to build a generative model that can be used to conditionally generate text. We introduce a mutual information criterion to encourage the model to put semantic information into the latent representation, and compare its efficacy with other tricks explored in literature such as KL divergence cost annealing and word dropout. We compare the log-likelihood lowerbounds on held-out data using variational autoencoders with the log-likelihoods on an unconditional language model. We notice our models quickly learn to exploit the grammatical redundancies in our dataset, but it is more challenging to encode semantic information in the latent representation.