AriEL: volume coding for sentence generation
  Mapping sequences of discrete data to a point in a continuous space makes it
difficult to retrieve those sequences via random sampling. Mapping the input to
a volume would make it easier to retrieve at test time, and that's the strategy
followed by the family of approaches based on Variational Autoencoder. However
the fact that they are at the same time optimizing for prediction and for
smoothness of representation, forces them to trade-off between the two. We
improve on the performance of some of the standard methods in deep learning to
generate sentences by uniformly sampling a continuous space. We do it by
proposing AriEL, that constructs volumes in a continuous space, without the
need of encouraging the creation of volumes through the loss function. We first
benchmark on a toy grammar, that allows to automatically evaluate the language
learned and generated by the models. Then, we benchmark on a real dataset of
human dialogues. Our results indicate that the random access to the stored
information is dramatically improved, and our method AriEL is able to generate
a wider variety of correct language by randomly sampling the latent space. VAE
follows in performance for the toy dataset while, AE and Transformer follow for
the real dataset. This partially supports to the hypothesis that encoding
information into volumes instead of into points, can lead to improved retrieval
of learned information with random sampling. This can lead to better generators
and we also discuss potential disadvantages.
