Vision-Infused Deep Audio Inpainting
  Multi-modality perception is essential to develop interactive intelligence.
In this work, we consider a new task of visual information-infused audio
inpainting, \ie synthesizing missing audio segments that correspond to their
accompanying videos. We identify two key aspects for a successful inpainter:
(1) It is desirable to operate on spectrograms instead of raw audios. Recent
advances in deep semantic image inpainting could be leveraged to go beyond the
limitations of traditional audio inpainting. (2) To synthesize visually
indicated audio, a visual-audio joint feature space needs to be learned with
synchronization of audio and video. To facilitate a large-scale study, we
collect a new multi-modality instrument-playing dataset called MUSIC-Extra-Solo
(MUSICES) by enriching MUSIC dataset. Extensive experiments demonstrate that
our framework is capable of inpainting realistic and varying audio segments
with or without visual contexts. More importantly, our synthesized audio
segments are coherent with their video counterparts, showing the effectiveness
of our proposed Vision-Infused Audio Inpainter (VIAI). Code, models, dataset
and video results are available at
https://hangz-nju-cuhk.github.io/projects/AudioInpainting
