Neural Dubber: Dubbing for Videos According to Scripts
  Dubbing is a post-production process of re-recording actors' dialogues, which
is extensively used in filmmaking and video production. It is usually performed
manually by professional voice actors who read lines with proper prosody, and
in synchronization with the pre-recorded videos. In this work, we propose
Neural Dubber, the first neural network model to solve a novel automatic video
dubbing (AVD) task: synthesizing human speech synchronized with the given video
from the text. Neural Dubber is a multi-modal text-to-speech (TTS) model that
utilizes the lip movement in the video to control the prosody of the generated
speech. Furthermore, an image-based speaker embedding (ISE) module is developed
for the multi-speaker setting, which enables Neural Dubber to generate speech
with a reasonable timbre according to the speaker's face. Experiments on the
chemistry lecture single-speaker dataset and LRS2 multi-speaker dataset show
that Neural Dubber can generate speech audios on par with state-of-the-art TTS
models in terms of speech quality. Most importantly, both qualitative and
quantitative evaluations show that Neural Dubber can control the prosody of
synthesized speech by the video, and generate high-fidelity speech temporally
synchronized with the video. Our project page is at
https://tsinghua-mars-lab.github.io/NeuralDubber/ .
