Latent Translation: Crossing Modalities by Bridging Generative Models
  End-to-end optimization has achieved state-of-the-art performance on many
specific problems, but there is no straight-forward way to combine pretrained
models for new problems. Here, we explore improving modularity by learning a
post-hoc interface between two existing models to solve a new task.
Specifically, we take inspiration from neural machine translation, and cast the
challenging problem of cross-modal domain transfer as unsupervised translation
between the latent spaces of pretrained deep generative models. By abstracting
away the data representation, we demonstrate that it is possible to transfer
across different modalities (e.g., image-to-audio) and even different types of
generative models (e.g., VAE-to-GAN). We compare to state-of-the-art techniques
and find that a straight-forward variational autoencoder is able to best bridge
the two generative models through learning a shared latent space. We can
further impose supervised alignment of attributes in both domains with a
classifier in the shared latent space. Through qualitative and quantitative
evaluations, we demonstrate that locality and semantic alignment are preserved
through the transfer process, as indicated by high transfer accuracies and
smooth interpolations within a class. Finally, we show this modular structure
speeds up training of new interface models by several orders of magnitude by
decoupling it from expensive retraining of base generative models.
