Sound2Sight: Generating Visual Dynamics from Sound and Context
  Learning associations across modalities is critical for robust multimodal
reasoning, especially when a modality may be missing during inference. In this
paper, we study this problem in the context of audio-conditioned visual
synthesis -- a task that is important, for example, in occlusion reasoning.
Specifically, our goal is to generate future video frames and their motion
dynamics conditioned on audio and a few past frames. To tackle this problem, we
present Sound2Sight, a deep variational framework, that is trained to learn a
per frame stochastic prior conditioned on a joint embedding of audio and past
frames. This embedding is learned via a multi-head attention-based audio-visual
transformer encoder. The learned prior is then sampled to further condition a
video forecasting module to generate future frames. The stochastic prior allows
the model to sample multiple plausible futures that are consistent with the
provided audio and the past context. Moreover, to improve the quality and
coherence of the generated frames, we propose a multimodal discriminator that
differentiates between a synthesized and a real audio-visual clip. We
empirically evaluate our approach, vis-\'a-vis closely-related prior methods,
on two new datasets viz. (i) Multimodal Stochastic Moving MNIST with a Surprise
Obstacle, (ii) Youtube Paintings; as well as on the existing Audio-Set Drums
dataset. Our extensive experiments demonstrate that Sound2Sight significantly
outperforms the state of the art in the generated video quality, while also
producing diverse video content.
