M3D-GAN: Multi-Modal Multi-Domain Translation with Universal Attention
  Generative adversarial networks have led to significant advances in
cross-modal/domain translation. However, typically these networks are designed
for a specific task (e.g., dialogue generation or image synthesis, but not
both). We present a unified model, M3D-GAN, that can translate across a wide
range of modalities (e.g., text, image, and speech) and domains (e.g.,
attributes in images or emotions in speech). Our model consists of modality
subnets that convert data from different modalities into unified
representations, and a unified computing body where data from different
modalities share the same network architecture. We introduce a universal
attention module that is jointly trained with the whole network and learns to
encode a large range of domain information into a highly structured latent
space. We use this to control synthesis in novel ways, such as producing
diverse realistic pictures from a sketch or varying the emotion of synthesized
speech. We evaluate our approach on extensive benchmark tasks, including
image-to-image, text-to-image, image captioning, text-to-speech, speech
recognition, and machine translation. Our results show state-of-the-art
performance on some of the tasks.
