Disentangling semantics in language through VAEs and a certain
  architectural choice
  We present an unsupervised method to obtain disentangled representations of
sentences that single out semantic content. Using modified Transformers as
building blocks, we train a Variational Autoencoder to translate the sentence
to a fixed number of hierarchically structured latent variables. We study the
influence of each latent variable in generation on the dependency structure of
sentences, and on the predicate structure it yields when passed through an Open
Information Extraction model. Our model could separate verbs, subjects, direct
objects, and prepositional objects into latent variables we identified. We show
that varying the corresponding latent variables results in varying these
elements in sentences, and that swapping them between couples of sentences
leads to the expected partial semantic swap.
