Variational methods for Conditional Multimodal Deep Learning
  In this paper, we address the problem of conditional modality learning,
whereby one is interested in generating one modality given the other. While it
is straightforward to learn a joint distribution over multiple modalities using
a deep multimodal architecture, we observe that such models aren't very
effective at conditional generation. Hence, we address the problem by learning
conditional distributions between the modalities. We use variational methods
for maximizing the corresponding conditional log-likelihood. The resultant deep
model, which we refer to as conditional multimodal autoencoder (CMMA), forces
the latent representation obtained from a single modality alone to be `close'
to the joint representation obtained from multiple modalities. We use the
proposed model to generate faces from attributes. We show that the faces
generated from attributes using the proposed model, are qualitatively and
quantitatively more representative of the attributes from which they were
generated, than those obtained by other deep generative models. We also propose
a secondary task, whereby the existing faces are modified by modifying the
corresponding attributes. We observe that the modifications in face introduced
by the proposed model are representative of the corresponding modifications in
attributes.
