Vocoder-Based Speech Synthesis from Silent Videos
  Both acoustic and visual information influence human perception of speech.
For this reason, the lack of audio in a video sequence determines an extremely
low speech intelligibility for untrained lip readers. In this paper, we present
a way to synthesise speech from the silent video of a talker using deep
learning. The system learns a mapping function from raw video frames to
acoustic features and reconstructs the speech with a vocoder synthesis
algorithm. To improve speech reconstruction performance, our model is also
trained to predict text information in a multi-task learning fashion and it is
able to simultaneously reconstruct and recognise speech in real time. The
results in terms of estimated speech quality and intelligibility show the
effectiveness of our method, which exhibits an improvement over existing
video-to-speech approaches.
