Speech2Video Synthesis with 3D Skeleton Regularization and Expressive
  Body Poses
  In this paper, we propose a novel approach to convert given speech audio to a
photo-realistic speaking video of a specific person, where the output video has
synchronized, realistic, and expressive rich body dynamics. We achieve this by
first generating 3D skeleton movements from the audio sequence using a
recurrent neural network (RNN), and then synthesizing the output video via a
conditional generative adversarial network (GAN). To make the skeleton movement
realistic and expressive, we embed the knowledge of an articulated 3D human
skeleton and a learned dictionary of personal speech iconic gestures into the
generation process in both learning and testing pipelines. The former prevents
the generation of unreasonable body distortion, while the later helps our model
quickly learn meaningful body movement through a few recorded videos. To
produce photo-realistic and high-resolution video with motion details, we
propose to insert part attention mechanisms in the conditional GAN, where each
detailed part, e.g. head and hand, is automatically zoomed in to have their own
discriminators. To validate our approach, we collect a dataset with 20
high-quality videos from 1 male and 1 female model reading various documents
under different topics. Compared with previous SoTA pipelines handling similar
tasks, our approach achieves better results by a user study.
