Multimodal Generative Models for Compositional Representation Learning
  As deep neural networks become more adept at traditional tasks, many of the
most exciting new challenges concern multimodality---observations that combine
diverse types, such as image and text. In this paper, we introduce a family of
multimodal deep generative models derived from variational bounds on the
evidence (data marginal likelihood). As part of our derivation we find that
many previous multimodal variational autoencoders used objectives that do not
correctly bound the joint marginal likelihood across modalities. We further
generalize our objective to work with several types of deep generative model
(VAE, GAN, and flow-based), and allow use of different model types for
different modalities. We benchmark our models across many image, label, and
text datasets, and find that our multimodal VAEs excel with and without weak
supervision. Additional improvements come from use of GAN image models with VAE
language models. Finally, we investigate the effect of language on learned
image representations through a variety of downstream tasks, such as
compositionally, bounding box prediction, and visual relation prediction. We
find evidence that these image representations are more abstract and
compositional than equivalent representations learned from only visual data.
