Realistic Speech-Driven Facial Animation with GANs
  Speech-driven facial animation is the process that automatically synthesizes
talking characters based on speech signals. The majority of work in this domain
creates a mapping from audio features to visual features. This approach often
requires post-processing using computer graphics techniques to produce
realistic albeit subject dependent results. We present an end-to-end system
that generates videos of a talking head, using only a still image of a person
and an audio clip containing speech, without relying on handcrafted
intermediate features. Our method generates videos which have (a) lip movements
that are in sync with the audio and (b) natural facial expressions such as
blinks and eyebrow movements. Our temporal GAN uses 3 discriminators focused on
achieving detailed frames, audio-visual synchronization, and realistic
expressions. We quantify the contribution of each component in our model using
an ablation study and we provide insights into the latent representation of the
model. The generated videos are evaluated based on sharpness, reconstruction
quality, lip-reading accuracy, synchronization as well as their ability to
generate natural blinks.
