Can Unconditional Language Models Recover Arbitrary Sentences?
  Neural network-based generative language models like ELMo and BERT can work
effectively as general purpose sentence encoders in text classification without
further fine-tuning. Is it possible to adapt them in a similar way for use as
general-purpose decoders? For this to be possible, it would need to be the case
that for any target sentence of interest, there is some continuous
representation that can be passed to the language model to cause it to
reproduce that sentence. We set aside the difficult problem of designing an
encoder that can produce such representations and, instead, ask directly
whether such representations exist at all. To do this, we introduce a pair of
effective, complementary methods for feeding representations into pretrained
unconditional language models and a corresponding set of methods to map
sentences into and out of this representation space, the reparametrized
sentence space. We then investigate the conditions under which a language model
can be made to generate a sentence through the identification of a point in
such a space and find that it is possible to recover arbitrary sentences nearly
perfectly with language models and representations of moderate size without
modifying any model parameters.
