{'fraction_non_alphanumeric': 0.03301255771803392, 'fraction_numerical': 0.011996720321063307, 'mean_word_length': 4.541367766618842, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Talking Head from Speech Audio using a Pre-trained Image Generator

We propose a novel method for generating high-resolution videos of talking-heads from speech audio and a single 'identity' image. Our method is based on a convolutional neural network model that incorporates a pre-trained StyleGAN generator. We model each frame as a point in the latent space of StyleGAN so that a video corresponds to a trajectory through the latent space. Training the network is in two stages. The first stage is to model trajectories in the latent space conditioned on speech utterances. To do this, we use an existing encoder to invert the generator, mapping from each video frame into the latent space. We train a recurrent neural network to map from speech utterances to displacements in the latent space of the image generator. These displacements are relative to the back-projection into the latent space of an identity image chosen from the individuals depicted in the training dataset. In the second stage, we improve the visual quality of the generated videos by tuning the image generator on a single image or a short video of any chosen identity. We evaluate our model on standard measures (PSNR, SSIM, FID and LMD) and show that it significantly outperforms recent state-of-the-art methods on one of two commonly used datasets and gives comparable performance on the other. Finally, we report on ablation experiments that validate the components of the model. The code and videos from experiments can be found at https://mohammedalghamdi.github.io/talking-heads-acm-mm


INTRODUCTION
Synthesising video of talking heads from speech audio has many potential applications, such as video conferencing, video animation production and virtual assistants. Although there has been considerable prior work on this task, the quality of generated videos is typically limited in terms of overall realism and resolution. In this paper, we propose an audio-driven model that synthesises highresolution talking-head videos (1024 x 1024 in our experiments) from a single identity image.
Many previous models generate low-resolution video [31,32] or cropped faces [3,4]. Low resolution video is generally not suitable for deployment in many real world applications, such as a virtual assistant. A common approach has been to use intermediate features such as facial landmarks to map from audio to output video [3,7,41]. Another approach has been to edit an existing talkinghead video to blend in a new mouth region synthesised from audio [28,29].
Recent advances in image synthesis have been successful at generating high-resolution images from noise [12][13][14]. Karras et al. [14] propose a style-based generator StyleGAN that synthesises high quality images that are largely indistinguishable from real ones. Some works have studied the latent space of StyleGAN [1,9,22,24] and discovered meaningful semantics for manipulating images. Recent work has leveraged the richness of a pre-trained StyleGAN generator [15] to generate high-resolution videos from noise by decomposing (disentangling) the motion and content in the latent space [8,26,30]. Tian et al. [30] discover motion trajectories in the latent space to render high-resolution videos while image and motion generators are trained on different domain datasets.
Inspired by these advances, we propose a novel method for generating high-resolution videos conditioned on speech audio by constructing trajectories in the latent space of a pre-trained image generator [15]. Our framework uses a pre-trained image encoder [21] to find the latent code of a given identity image in the latent space of the generator. We then train a recurrent audio encoder along with a latent decoder to predict a sequence of latent displacements to the encoded identity image. In this stage, we show our approach can generate talking-head videos with accurate mouth movements conditioned on speech audio. To improve the visual quality of the generated videos further, we tune the generator on a single image or short video of a target subject using the PTI [22] method. We compare our approach with other state of the art approaches qualitatively and quantitatively using benchmark measures: LMD, SSIM, PSNR and FID. We show that it achieves performance at least as good as the state of the art on two commonly used datasets.
Our principal contributions are: • A method for generating high-resolution videos from speech audio by constructing motion trajectories in the latent space of a pre-trained image generator; • A comparative evaluation, including a user study, demonstrating the performance of the method on both quantitative and qualitative criteria.

RELATED WORK 2.1 Audio-driven talking-head generation
Various methods have been proposed to generate videos of talking heads. Given audio, the task is to lip-sync the head to the audio. The audio may itself be generated automatically from text or be extracted from a video clip of someone speaking. Some of these methods are generic, and can generate videos of any identity given one or more images of that identity [3,4,20,31,32,[39][40][41]. However, they can only generate low-resolution videos. Other methods that generate high-resolution videos from speech audio [11, 17, 18, 27-29, 34, 36, 38] can only work on a single subject. These approaches require retraining the models for each new subject.
Chung et al. [4] propose a model to generate videos using multiple images of the target face and an audio speech sample. The model consists of an audio encoder and identity encoder that learn a joint embedding of the face and audio, and a decoder that generates a frame that best represents the audio sample for the target identity. Prajwal et al. [20] adopt a similar approach, except that a pretrained lip-sync discriminator, and a visual discriminator are used in addition to 1 loss. Vougioukas et al. [31] expand the approach by training a recurrent-based decoder with a noise generator to model spontaneous facial expressions (e.g. blinks). Other models rely on 2D intermediate features (e.g. facial landmarks) to learn the mapping between audio input and video output [3,7,41]. Chen et al. propose a cascade approach that generates a talking face video given an image and audio. The method first transfers the given audio signal to facial landmarks and then generates video frames conditioned on the landmarks. Zhou et al. [41] adopt a similar approach that first disentangles the content and speaker information from the input audio. Then, these two components are mapped to content and speaker facial landmark spaces using a recurrent model on each.
Other methods rely on 3d intermediate features (e.g. through monocular reconstruction) to synthesise high-quality videos of a single subject. Some attempt to generate only the mouth region and blend it to a target video [17,[27][28][29]. Thies et al. [29] propose an approach that predicts the coefficients that drive a person-specific expression blendshape basis using audio features. A neural texture rendering network is then used to generate the mouth region. In addition, others modify the facial expressions, geometry or pose of a target video conditioned on the audio [11,36]. In contrast, our  Figure 2: Overview of the model: Given an identity image and a speech audio, the aim is to synthesise a video of the identity lip-synced with the audio. We first find the corresponding latent code of the identity image using the image encoder . We then encode the audio using the audio encoder . Next, we embed the identity latent code using the PCA basis . The decoder then takes both the embedded identity latent vector and encoded audio to predict a displacement to the identity latent vector in the latent space of a pre-trained image generator .
approach generates a full-frame video of a talking-head without editing a target video or relying on intermediate features.

Unconditional video generation using StyleGAN
Recently, there have been several works that use a pre-trained image generator (StyleGAN) to generate videos [8,26,30]. They all share the idea of discovering motion trajectories in the latent space of a StyleGAN generator without conditioning on any driving source. Tian et al. [30] propose a MOCOGAN-HD model that uses a motion generator to predict residual latent codes from an initial latent code sampled from the latent space of StyleGAN. The model is trained in the image space with a multi-scale video discriminator as well as contrastive image discriminator. Similarly, Fox et al. [8] reduce the training cost by training a Wasserstein GAN model in the latent space instead of image space. Although their model is trained on a single subject dataset, it can transfer the learned motion to a new subject using an offset trick. Skorokhodov et al. [26] modify the StyleGAN network to learn a continuous latent trajectory using a neural representation based approach. Our work differs from these methods by learning the motion trajectories in the latent space of StyleGAN conditioned on speech audio.

THE METHOD
Our method consists of four components: image encoder , audio encoder , latent decoder and image generator . Given an identity image I and speech audio a, the goal is to synthesise a video of the identity lip-synced with the audio. We first partition the audio clip into a sequence of fixed-duration audio segments { 1 , 2 , .., }. From this audio segment sequence, we target a video clip consisting of a sequence of video frames { 1 , 2 , .., }. There is therefore a one-to-one correspondence between input audio segments and output video frames. The inference pipeline is as follows. We take the identity image I and finds its latent code in the latent space + of the generator using the image encoder . Next, we encode an audio segment using the audio encoder and extract an audio embedding vector = ( ). We then feed in the identity latent code and the audio vector jointly to the latent decoder to predict a latent displacement = ( , ). We then calculate the displaced latent vector¯= + . Lastly, the image generator takes the displaced latent vector¯and generates the corresponding video frame¯. An overview of the model can be seen in figure 2. In the following section, we describe each component in detail.

Architecture
3.1.1 Image generator. For our image generator , we use the pre-trained StyleGAN [15] trained on the FFHQ dataset [14] to synthesise static images of faces. The StyleGAN architecture consists of mapping and synthesis networks. The mapping network is a non-linear 8-layer MLP which maps a latent code sampled from a latent space to an intermediate space + . The produced controls the synthesis network through an adaptive instance normalization (AdaIN) operation after each convolutional layer. We use only the synthesis part as our image generator .
3.1.2 Image encoder. We use pSp [21], an off-the-shelf pre-trained image encoder that inverts real images to the + space of StyleGAN. The encoder was trained by embedding the FFHQ [14] dataset to a fixed StyleGAN generator. Critically, the encoder produces latent vectors that preserve the mouth expression on synthesis with StyleGAN. Figure 3 shows frames from a video that has been inverted into the latent space + using the image encoder and then re-generated using .

Audio encoder.
We represent each audio segment using Mel-frequency cepstral coefficients (MFCC). The audio encoder is a recurrent model that takes a sequence of audio segments { 1 , 2 , .., } and produces a sequence of encoded audio segments { 1 , 2 , .., }. The encoder network consists of multiple convolutional layers followed by three LSTM layers.
3.1.4 Latent decoder. The latent decoder takes as input the identity latent vector and an encoded audio segment to predict a displacement to the identity vector in the latent space + . To reduce the high dimensionality of the latent space + , we first conduct principal component analysis (PCA) on the FFHQ dataset [14] mapped into + using the image decoder. We obtain a subspace from the components with the largest eigenvalues, giving a basis V. We project the identity latent input into the subspace defined by V and concatenate it with the encoded audio segment . This provides the input to the decoder. We map the decoder's output ℎ from the subspace to + to get the displacement vector . Thus, we obtain¯as follows:

Training
Our model is trained in two stages. In stage one, we are only interested in learning trajectories in the latent space + conditioned on the speech audio. The model predicts latent displacements to the identity in the latent space of a fixed image generator. This disentangles the mouth motion and the image content. The motion trajectories are learned by training the model using a talking-head dataset. Although the model learns to generate accurate mouth movements, the visual quality of the generated video exhibits some distortion (see section 4.5). The quality is determined by the pretrained StyleGAN generator, which has been trained on images of people who are typically making a static pose. In stage two, we tune the generator on a single image or short video of a target speaker.
3.2.1 Stage one. We train only the audio encoder and latent decoder while keeping the pre-trained image encoder and the pre-trained image generator fixed. For the loss function, we project the target frame into the generator's latent space + using the image encoder . We then have the corresponding latent code = ( ) for = 1, 2, ..., . We calculate an L2 loss between each target latent code and the predicted latent code¯. We define L latent as follow: In addition, we apply another loss in the image domain between the generated videox and the target video x. Since the pre-trained image generator has not seen the training data, applying the loss directly on the target video would affect the model's performance, enforcing it to focus on the facial appearance rather than mouth movements. For this, we invert the target video x using the image encoderˆ= ( ). We calculate the perceptual loss [37] between the generated video {¯1,¯2, ..,¯} and the inverted target video {ˆ1,ˆ2, ..,ˆ} as follows: where is based on the VGG neural network [25]. Thus, the overall loss for learning to predict latent displacements driven by audio is a weighted sum of the two losses: 3.2.2 Stage two. One could tune both the image encoder and the image generator on the target speaker using an auto-encoder. However, this would transform the latent space + and the learned model at stage one would consequently fail to generate correct mouth movements. To improve the visual quality of the generated video, we use the PTI method [22] to tune only the generator on a single image or video of a target speaker. In experiments, we implement stage two on short videos, and as a limiting case, on a single image.
Given a video x of a speaker, we tune the image generator on the video frames . For this task, we use the pre-trained encoder to encode the frames to the latent space + to get . Given = ( ; * ), we tune the weights of the generator while keeping the encoder fixed. We use the same objective loss used in PTI [22]: where L L2 is defined as : After tuning the generator, we can generate videos of talkingheads using our inference pipeline with the components trained in both stages.

EXPERIMENTS 4.1 Datasets
We evaluate our approach using two widely used datasets for synthesizing talking-head videos: GRID [5] and TCD-TIMIT [10]. The GRID dataset has 33 speakers uttering 1000 short sentences each containing 6 words. The TCD-TIMIT has 59 speakers each uttering 100 sentences. We hold-out ten speakers from each dataset for testing and use the remaining for training. The videos are resampled to 25 fps. To align the video frames, we use the same face alignment method used in preprocessing the FFHQ dataset [14] for training the original StyleGAN [15]. The input to the audio encoder is an audio segment of length 0.2 seconds which corresponds to a window of five frames. However, we choose only the middle frame as the the ground truth frame. The identity image is a randomly chosen frame out of this window. We represent the audio speech using MFCC values extracted from the raw values. Each audio segment is a window of size 12 x 28, where the columns represent MFCC features for each time step.

Implementation details
We perform our experiments using PyTorch [19]. For the image generator, we use an unofficial implementation of StyleGAN 1 . For the pre-trained image encoder, we use the official implementation of p2p [21]. For training the audio encoder and the decoder in stage one, we use an Adam optimiser [16] with a learning rate of 0.0002. In Eqn. 4, we set latent = 250 and LPIPS = 1. We tune the 1 https://github.com/rosinality/stylegan2-pytorch generator with a learning rate of 0.0003. The tuning process takes less than two minutes for a single identity image. All experiments use an NVIDIA V100 GPU with 32 GB of memory.

Results
In this section, we evaluate our model after tuning the generator. Figure 4 shows the quality of the generated videos from tuning the generator on a single frame (middle row) and on a short video (bottom row). The figure shows that tuning the generator on multiple frames has resulted in better visual quality. This can be seen in the mouth appearance highlighted in red.
To evaluate the quality of the generated videos, we use two common reconstruction measures: The peak signal-to-noise ratio (PSNR) and the structural similarity (SSIM) [35]. For these measures, a larger score is better. We use a landmarks distance metric (LMD) [2] to evaluate the synchronisation between the mouth movement and the speech audio. This metric computes the Euclidean distance between mouth landmarks of each generated frame and its corresponding true frame. It then averages the score on the number of frames and number of mouth landmark points. We also use a Fréchet Inception Distance (FID) to quantitatively evaluate generated videos. For LMD and FID measures, a lower score is better.
We compare our work against four state of the art models [3,20,31,41] and use the official available codes of these models to generate the videos and compute the evaluation measures. Table 1 shows that our approach outperforms other state of the art models on the TCD-TIMIT dataset [10]. On the GRID dataset [5], the model achieves better scores on the PSNR and FID measures. Figure 5 shows the visual quality of generated videos by our model in comparison with other models. The highlighted frames show that our model generates photo-realistic videos largely indistinguishable from the ground truth. Figure 6 shows a visual comparison of our model with others on a challenging mouth movement associated with the phoneme /p/ in the word "place". It can be seen that our model and Vougioukas et al. produce a closed-mouth shape (highlighted in green) in sync , Prajwal et al. [20] and Ours on audio samples form TCDTIMIT (left) and GRID (right). It can be seen that our model synthesises a lip-movement that is closer to the ground truth than the other methods.  Table 2 shows the number of trainable parameters, inference time and output frame size for the five methods. We ran all experiments on a V100 Nvidia GPU and report the achieved frame rate (FPS) as a measure of inference time. The source videos are sampled at 25 FPS. We can see that the method of Vougioukas, et al. [31] and ours are faster than real-time. In addition, our method generates much higher resolution videos compared to others.

User Study
We conducted a user study to compare our approach with related works using Amazon Mechanical Turk services. We evaluate both the audio-visual synchronisation and the visual quality of the stateof-the-art methods. We show participants a pair of videos: one generated by our method and the other generated by either Chen et al. [3] or Zhou et al. [41]. For each pair, we either ask which video looks more photo-realistic or which video has more accurate lip-sync with the audio. The choices for each question are "right", "left", "none", and "both". We randomly choose the order of videos in each pair. We obtained 80 answers from 20 participants for each question. Figure 7 shows the results of the user study. It can be seen our model achieves a better result in terms of both the audio-visual sync and the visual quality.

Ablation analysis
We analyse the effect of each loss in Eqn 4 on the performance of the model in generating talking-head videos. We train the model in stage one without L latent and L LPIPS separately and test it after tuning the generator in stage two. We observe that the choices of these losses do not affect the visual quality of the generated video but affect the lip-synchronisation accuracy. This is indicated in  We also compare the performance of the model at stage one without tuning the generator and after tuning the generator. Figure  8 shows a sample of generated video using the model trained in stage one. It can be seen that the model generates correct mouth positions but the visual quality inherits some distortion caused by the image generator. In stage two, we tune the generator on a single frame or short video of the target speaker. We can see from Table 4 that SSIM and PSNR are higher after tuning the generator, indicating that stage two is important to improve the quality of the generated video.

ETHICAL CONSIDERATION
Our framework can synthesise high quality videos from speech audio. This is perfect for video production animation, a talkinghead avatar and video-dubbing. Creative people may use our work to edit content in movies or generate new videos. However, the model can be misused to spread misinformation or manipulate   existing data. Generated videos using deep learning are becoming hard to distinguish from the real thing, although there have been promising advances in forensics on the detection of "deepfake" videos [6,23]. We will share generated videos of our framework with the community to help detecting manipulated videos.

CONCLUSION AND FUTURE WORKS
We propose a novel method for synthesising high-resolution videos from speech audio. The model can generate videos of a target speaker given a short video (or single image) of the speaker. Our model is built on top of a pre-trained image generator. We first learn to generate talking-head videos by constructing motion trajectories conditioned on speech audio. We then improve the image generator by tuning it on a short video of a target speaker. We show that the method significantly outperforms recent stateof-the-art methods on TCD-TIMIT in quantitative experiments and gives performance comparable to the state-of-the art on GRID. The method also performs best in the user study.
The generated faces depict only mouth movements because the training datasets (TCD-TIMIT and GRID) are neutral and expressionless. We anticipate our approach could in principle generate other facial expressions where these are present in the dataset (e.g. [33]), but have not yet demonstrated that this is the case.