Exploiting Inductive Bias in Transformers for Unsupervised Disentanglement of Syntax and Semantics with VAEs
We propose a generative model for text generation, which exhibits
disentangled latent representations of syntax and semantics. Contrary to
previous work, this model does not need syntactic information such as
constituency parses, or semantic information such as paraphrase pairs. Our
model relies solely on the inductive bias found in attention-based
architectures such as Transformers.
  In the attention of Transformers, keys handle information selection while
values specify what information is conveyed. Our model, dubbed QKVAE, uses
Attention in its decoder to read latent variables where one latent variable
infers keys while another infers values. We run experiments on latent
representations and experiments on syntax/semantics transfer which show that
QKVAE displays clear signs of disentangled syntax and semantics. We also show
that our model displays competitive syntax transfer capabilities when compared
to supervised models and that comparable supervised models need a fairly large
amount of data (more than 50K samples) to outperform it on both syntactic and
semantic transfer. The code for our experiments is publicly available.