{'fraction_non_alphanumeric': 0.03452256033578174, 'fraction_numerical': 0.011815320041972718, 'mean_word_length': 4.804019488428746, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Head2Head++: Deep Facial Attributes Re-Targeting

Facial video re-targeting is a challenging problem aiming to modify the facial attributes of a target subject in a seamless manner by a driving monocular sequence. We leverage the 3D geometry of faces and Generative Adversarial Networks (GANs) to design a novel deep learning architecture for the task of facial and head reenactment. Our method is different to purely 3D model-based approaches, or recent image-based methods that use Deep Convolutional Neural Networks (DCNNs) to generate individual frames. We manage to capture the complex non-rigid facial motion from the driving monocular performances and synthesise temporally consistent videos, with the aid of a sequential Generator and an ad-hoc Dynamics Discriminator network. We conduct a comprehensive set of quantitative and qualitative tests and demonstrate experimentally that our proposed method can successfully transfer facial expressions, head pose and eye gaze from a source video to a target subject, in a photo-realistic and faithful fashion, better than other state-of-the-art methods. Most importantly, our system performs end-to-end reenactment in nearly real-time speed (18 fps).


I MAGE and video synthesis receives a rapidly increasing amount of attention in Computer Vision and Deep
Learning research, as "synthetic" data appear more and more realistic, as well as especially promising for real-world applications. Video editing, film dubbing, social media content creation, teleconference and virtual assistance are some indicative examples. However, generating artificial human faces indistinguishable from real ones is a very challenging task, particularly when it comes to video data. Adding the extra dimension of time, might give rise to the so-called problem of temporal incoherence. As the uncanny valley effect suggests, people are extremely perceptible in unnatural facial and head movements, thus even small discontinuities can expose a synthetic video.
Over the past years, various methods that target human faces have emerged, taking advantage of different modalities, such as audio [1], [2] and video [3], [4], [5] to drive synthesis and dictate the movements of the generated subject. Facial reenactment is a widely-studied approach [3], [6], [7], which aims to transfer the facial expressions from a source to a target subject, by conditioning the generative process on the driving video of the source. In most cases [3], [7], this is done by modifying the deformations solely within the internal facial region of the target identity and placing the manipulated face back to the original target frames, using an image interpolation method. Given that face reenactment systems offer no control over the target's head pose and eye • * denotes equal contribution gaze, there might be cases where the person's expressions do not match with the overall head movement and therefore seem unnatural. Even so, face reenactment can be very useful for specific applications, such as video dubbing [8], which aims to alter the mouth motion of the target actor to match the audio track spoken by the dubber.
A more holistic approach on reenactment involves generating all pixels within frames, including the upper body, hair and background of the target identity. In contrast to video manipulation techniques, such as face reenactment, full head reenactment methods [4], [9] aim to transfer the entire head motion from a source identity to a target one, along with the eye blinking and gaze, providing complete control over the target subject. Most head reenactment approaches fall into two categories: a) Warping-based methods [2], [10], [11], which are mainly learning-based and do not involve computing priors such as facial landmarks or 3D face models, b) object-specific methods [3], [4], [9], [12], which assume knowledge (e.g. 3D reconstruction, 68 landmarks) of the source and target faces. On the one hand, warping-based methods usually suffer from distortions in the face and background [2]. On the other hand, objectspecific methods relying on facial keypoints are affected from the so-called identity preservation problem. As keypoints encapsulate identity attributes of the source (e.g, head geometry), the more the identity of the source diverges from that of the target, the more distorted the head shape of the generated subject might appear. On the contrary, 3D morphable models (3DMMs) [13], [14] have proven to be a reliable means of decoupling expression and identity from each other. Deep Video Portraits (DVP) [4] is a full head reenactment system that capitalises on 3DMMs and utilises a neural network to translate 3D face reconstructions to realistic frames. Nonetheless, DVP, as a purely image-based model, does not take into account temporal dependencies arXiv:2006.10199v2 [cs.CV] 28 Sep 2021 between frames.
Our recently proposed Head2Head [15] model overcomes the aforementioned limitations, as it combines the benefits of conditioning synthesis on 3D facial shapes with the advantages of a sequential, video-based, neural renderer. In this paper, we extend the work of Head2Head in the following directions: • We prevail over the limitations of the 3D reconstruction stage of Head2Head by designing a novel Dense-FaceReg network for the robust and fast estimation of semantic facial images based on 3D facial geometry. Our semantic facial images, referred to as Normalised Mean Face Coordinates (NMFC) images, capture pose, expression and identity information of the subject from the 3D facial mesh and are used to condition video synthesis. • We propose a simple yet fast method for detecting the eye gaze, based on 68 facial landmarks. When combined with 3D Facial Recovery and Video Rendering Network, our system can operate in nearly real-time speeds.
• We conduct an extensive set of experiments, including user, automated and ablation studies. Our results reveal the significance of video-based modeling.
• We make our code and dataset publicly available 1 .

3D Face Reconstruction
Recovering the 3D geometry of human faces from monocular images is a challenging problem that has attracted much attention due to its major role in many applications, ranging from facial reenactment, performance capture and tracking [16], facial expression recognition [17], [18], etc. Owing to their pose and illumination invariance, 3D facial data constitute an indispensable geometrical description of faces for various facial image processing systems. The Computer Vision field is rich in approaches targeting this problem under different assumptions and constraints. Some of these attempts [19], [20], named Shape from Shading, approximate the image formation process and make simplified assumptions about the lighting and illumination models leading to the formation of the image, while others, known as Structure from Motion (SfM), benefit from the geometric constraints in multiple images of the same object to solve the problem [21], [22]. One common approach for addressing this task are the 3D Morphable Models (3DMMs) of the human face. 3DMMs are linear statistical models that have been used substantially since the pioneering work of Blanz and Vetter [23], with many extensions [13], [14], [24]. With the rise of deep neural networks, nonlinear face models have been proposed to recover the 3D geometry and appearance of human faces from images or videos using deep Convolutional Neural Networks (CNNs) [25], [26], [27], [28]. For a very recent and comprehensive review of the state-of-the-art methods on monocular 3D face reconstruction and the open challenges of 3DMMs, we refer the readers to [29], [30].
In this work, we employ 3DMMs and a 3D shape regression network for: 1) estimating the 3D geometry of 1. https://github.com/michaildoukas/head2head faces appearing in videos, and 2) utilising the estimated 3D geometrical information for driving our Deep Video Rendering Neural Network.
Traditional methods of reenactment, transfer the facial expressions either with 2D warping techniques [10], [11], or by utilizing 3D face models [3], [6], [7]. These methods do not provide complete control over the generated video, as they manipulate only the interior of the face. Recently, there has been a substantial effort in the direction of both expression and pose transfer [2], [4], [9], [36], [37]. One of the first approaches, X2Face [2], designs an embedding and a set of auto-encoders. X2face follows a warping-based approach that causes deformations in the generated heads and inconsistent upper-body motions. Wang et al. [12] propose vid2vid, a GAN-based spatio-temporal approach for the video-to-video synthesis, relying on a Temporal Discriminator for improving the temporal quality of the synthesised videos. In a follow-up work, Wang et al. [38] extend their approach with an attention mechanism, making it trainable in a few-shot manner and leading to better generalisation performance. Both [12] and [38] can perform reenactment with face sketches drawn from landmarks, leading to a target identity preservation problem. Moreover,. the synthesised mouth regions do not look very realistic.
Zakharov et al. [9] propose a few-shot, image-based adversarial learning approach, as their network learns to generate unseen target identities even from a single image. Nonetheless, their neural network relies on landmarks, causing an identity distortion of the target. Siarohin et al. [39] animate objects via a deep motion transfer framework. Given a single image and a driving sequence, their method estimates a dense motion field appearing in the sequence and transfers it to the target image while preserving its appearance. When used for facial reenactment, the faces synthesised by this approach suffer from head distortions and non-naturalistic mouth and teeth areas. To tackle this issue, the authors extended their work [40] to account for complex motions with a first-order motion model and an occlusion-aware Generator. Although this extension considerably improved the results on various tasks, synthesised faces still exhibited visual artifacts appearing as expressiondependent continuous scale changes. This can be attributed to the estimated 2D dense motion field that does not fully describe the actual intricate 3D facial motion.
As far as we are aware, DVP [4] is the only learningbased head reenactment system, prior to our work, that uses 3D facial information to condition video synthesis. Their image-based model requires a long video footage of the target person, while training a new model for each target takes many hours. Moreover, generated mouths look unnatural, since the 3D reconstruction method they adopt does not encode the inner-mouth region or the teeth.  ) and camera (pt) parameters of the source for each time-step t, are passed to the NMFC renderer, which produces the NMFC video. At the same time, the eye landmarks of the source are used to detect the pupils and sketch the eyes video, which represents eyes movements. Finally, the combined NMFC and eyes video is given as conditional input to the Video Rendering Network, which computes the generated video (reenactment result). The Video Rendering Network is a person specific model, as it is trained using footage of the target person. During training, the source video coincides with the target video, enabling us to perform self-reenactment and get access to ground truth data.
Unlike other studies, our approach employs efficiently the 3D geometry and conditions frame generation on a compact and meaningful representation in the image space derived from 3D facial reconstruction. When combined with our novel video-based neural rendering stage, the result is a faithful and photo-realistic full head video reenactment, indistinguishable from real videos. Additionally, we focus specifically on the mouth area and improve its visual quality by designing a dedicated discriminator.

METHODOLOGY
Our Head2Head++ framework proposes a solution for the largely ill-posed full head reenactment problem. Our method is capable of transferring the time-varying head attributes (pose, expression, eye gaze) and mainly consists of two subsequent modules: a) 3D Facial Recovery (Sec. 3.1), and b) a Video Rendering Network (Sec. 3.4). Our video rendering network capitalises on a carefully designed GANbased framework. Please refer to Fig. 1 for an overview of our Head2Head++ pipeline.

3D Facial Recovery
We aim to generate a reliable estimation of the facial 3D geometry, capturing the temporal dynamics, while separating the identity and expression contributions of the pictured subject in each frame. We utilise this separability to effectively disentangle the human head characteristics in a transferable and photo-realistic way between different videos. Towards that aim, we benefit from the prior knowledge in our problem space and harness the power of 3DMMs [23] for reconstructing the faces that appear in the input monocular sequences. Given a sequence of T frames F 1:T = {f t | t = 1, . . . , T }, the 3D reconstruction stage produces two sets of parameters: 1) shape parameters S = {s t | s t ∈ IR ni+ne , t = 1, ..., T }, and 2) camera parameters P = {p t | p t ∈ IR 6 , t = 1, ..., T }, depicting rotation, translation and orthographic scale. Shape representation. Using 3DMMs, a 3D facial shape can be written mathematically as: wherex ∈ IR 3N is the mean shape of the morphable model, given byx =x id +x exp , withx id andx exp standing for the mean identity and expression of the model, respectively. U id ∈ IR 3N ×ni is the identity orthonormal basis with n i principal components (n i 3N ), U exp ∈ IR 3N ×ne is the expression orthonormal basis with the n e principal components (n e 3N ) and s i t ∈ IR ni , s e t ∈ IR ne are the identity and expression parameters of the morphable model. We designate the joint identity and expression parameters . In the adopted model (1), the 3D facial shape x is a function of both identity and expression coefficients (x(s i t , s e t )), where expression variations are effectively represented as offsets from a given identity shape. Video-based 3D reconstruction. The video fitting approach followed in Head2Head [15] to estimate the 3D facial geometry is based on a set of sparse landmarks extracted from the entire input sequence. This method has three main drawbacks: 1) the fidelity of the 3D reconstruction relies heavily on the accuracy of extracted landmarks which are also sparse (68 in total), 2) it might require a large number of frames with enough reconstruction cues (various rotations) to produce good accuracy, 3) it makes a quite strong assumption in the initialisation stage about the rigidity of the face to estimate the camera parameters. To overcome these limitations, we propose to perform the 3D facial reconstruction in this work by training a deep CNN, we call DenseFaceReg, the purpose thereof is to produce a dense 3D facial mesh from a single RGB frame. We use a thousand annotated videos from the Face3DVid dataset of Koujan et al. [16] to train this network in a supervised manner. The adopted loss function during training is: (2) Equation (2) penalises the deviation of each vertex from the corresponding ground-truth vertex (v i = [x, y, z] ). We use the camera parameters provided with the Face3DVid dataset to project the 3D ground-truth mesh, so that our DenseFaceReg produces 3D vertices (dense landmarks) directly in the image space. We use the dense 3D vertices (∼5K) estimated by our trained DenseFaceReg on each video frame to: 1) estimate the camera parameters, 2) generate the 3DMM identity and expression coefficients by projecting the dense shape onto the 3DMM bases. For all our experiments in this work, we use the same 3DMMs utilised in [15]. The analysis-by-synthesis approach, which is used by many state-of-the-art approaches [3], [4], estimates a lot of parameters (e.g. illumination, reflectance, shape, etc) and solves a highly ill-posed problem for fitting 3DMMs to images. On the contrary, our facial reconstruction stage is a fast CNNbased approach (6ms test runtime) trained on a large number of in-the-wild videos. The facial representation extracted with our method is informative enough to synthesise photorealistic and temporally smooth videos, eliminating the need for more elaborate and slower 3D facial reconstruction techniques.

Facial Semantic Representation -NMFC Rendering
Our video rendering network receives as input a facial semantic representation of the target subject, with the head pose and facial expression guided by the source frames. This representation disentangles identity from expression, allowing us to train our video rendering network on a specific target person. During test, we are able to transfer the expression and pose of any source, with different head characteristics, to the target. Given the recovered identity and expression parameters (facial shape) and camera parameters p t , at frame t, we rasterize the 3D shape, producing a visibility mask (M ∈ IR W ×H ) in the image space. Each pixel of M, stores the index of the corresponding visible triangle on the 3D face seen from this pixel. Thereafter, we store the normalised x-y-z coordinates of the centre of this triangle in the NMFC ∈ IR W ×H×3 image, which is the facial semantic representation that is utilised as conditional input to the video rendering network. Equation (3) details this process.
where R is the rasterizer, E is the encoding function and x is the normalised version of the 3DMM mean face (see (1)), so that the x-y-z coordinates of this face belong in [0, 1]. The NMFC image generation relies not only on the 3D reconstructed face and camera parameters of the current frame but also on an always-fixed normalised mean face (x) of the employed 3DMM. The (x, y, z) coordinates of the normalised mean face are used as constant colors to texture the 3D face of the current input frame f t . The NMFC image of a frame f t is generated then by rendering the corresponding textured 3D mesh. Since we texture any reconstructed 3D mesh with always a fixed set of distinctive colors, the same semantic point, say the tip of the nose, in any NMFC image (regardless of the subject and the target video) will always have the same color. This is why we refer to NMFCs as semantic images. The main advantage of using NMFCs over a UV 2D parameterisation is that it proved experimentally to be easier for the video renderer to learn the mapping to the output RGB images, since both the NMFCs and the output are in the same space. Additionally, this representation is more compact and easy-to-interpret by our video rendering network, since it associates well with the corresponding RGB frame to be generated, pixel by pixel, and, subsequently, leads to a realistic and novel video synthesis. Note that during test time, the expression coefficients s e t and camera parameters p t are estimated from the source video at frame t, while the identity coefficients s i are the average identity parameters estimated using all frames of the target video (see Fig. 1).

Eye Pupils Detection
We choose a real-time operating method for the extraction of the eye movements from the source frames, which does not require fitting an eye model [41], but is based on 68 facial landmarks [42]. Given the subset of landmark points L eye ∈ IR 6×2 that correspond to the left or right eye, we estimate one more landmark l pupil ∈ IR 2 , which corresponds to the eye pupil. The eye pupil is obtained as the centre of mass within the set of pixels Ω that are bounded by the polygon formed with the eye landmarks L eye by computing a weighted sum, using the inverse intensity of pixels in Ω as weights. Following [43], we are based on the assumption that eye pupils are the "darker" areas within the eye region in terms of pixel intensity. Mathematically, the eye centre of mass is given as where I(p) is the inverse intensity of pixel p in the source frame. In this way, l pupil corresponds to the center of "darker" pixels within the eye region.
Once the eyes and pupils landmarks are estimated, we create the eyes video E 1:T , which is a sequence of T RGB frames of size 256 × 256 × 3. An example of these frames is shown in Fig. 1. We connect the eye landmarks with white edges to create an outline of each eye. Then, two red circles are drawn on the same plane, using as centres the eye pupil coordinates of each eye, in order to indicate eye gaze.

Deep Video Rendering Neural Network
Our carefully-designed video rendering network receives as conditional input two sequences of images, namely: the NMFC video NMFC 1:T and the corresponding eye video E 1:T , collectively as With this input, the video rendering network yields a highly realistic and temporally coherent output videoỸ 1:T picturing the target actor mimicking exactly the same expressions, head pose and eyes reflected in X 1:T . While training, we follow a self reenactment setting where the source and target videos are the same footage. In this way, the reenacted video in the output should be a replication of the RGB training target video Y 1:T , which serves as ground truth. Our video rendering network is trained within a GAN-based structure with: 1) a Generator G, which is the video rendering network itself, 2) an Image Discriminator D I , 3) a multi-scale Dynamics Discriminator D D enforcing the temporal coherence and realism of the produced facial performance, and 4) a dedicated Mouth Discriminator D M , which further improves the visual quality of the mouth area. Generator G. To successfully synthesise smooth and convincing temporal facial performances, we condition the synthesis of the t-th frameỸ t not only on the conditional input X t , but also on the previous inputs X t−1 and X t−2 , as well as the previously generated framesỸ t−1 andỸ t−2 , thus:Ỹ The Generator is applied sequentially, producing the output frames one after the other, until the entire output sequence has been created. Similar to Head2Head [15], the Generator consists of two identical encoders, operating in parallel, as well as a decoder. The first encoder receives the concatenated NMFC and eye images X t−2:t , while the second is given the two previously generated framesỸ t−2:t−1 .
The two extracted feature maps are first added and then passed through the decoder, which brings the outputỸ t in a normalised [-1,+1] range, using a tanh activation function. Image Discriminator D I and Mouth Discriminator D M . Both of these networks aim at telling real and synthesised frames apart, and are used only during training. At timestep t, the Image Discriminator D I processes the real pair (X t , Y t ) and the fake one (X t ,Ỹ t ). Concurrently, the corresponding mouth regions (X m t , Y m t ) and (X m t ,Ỹ m t ) are cropped and sent to the Mouth Discriminator D M . Dynamics Discriminator D D . With the aim of learning complex facial dynamics in mind, we further equip our GAN framework with a Dynamics Discriminator D D . During training, D D receives a set of three consecutive real frames Y t:t+2 or fake framesỸ t:t+2 . They are passed to D D after being associated with the optical flow W 1:T −1 , computed from the real frames (training video Y 1:T of target subject). Following this, the Generator is encouraged to create fake frames showing the same flow (dynamics) as the corresponding real ones. The Dynamics Discriminator learns to differentiate between the real (W t:t+1 , Y t:t+2 ) and fake (W t:t+1 ,Ỹ t:t+2 ) pairs. In practice, we employ a multiple-scale Dynamics Discriminator, which operates in three different temporal scales. The first scale receives frame sequences in the original frame rate. Then, the two extra scales are formed by sub-sampling the frames by a factor of two for each scale. Objective function: In order to train our Generator to synthesise samples as real as possible, we formulate an adversarial loss. More specifically, we use LSGAN [44] with labels a = c = 1 for fake samples and label b = 0 for real ones, resulting in the following adversarial objective for the Generator: We add two more losses in the learning objective function of the Generator: 1) a VGG loss L G vgg and 2) a feature matching loss L G f eat , first proposed in the work of Xu et al. [?]. Our feature matching loss term is based on the activations of both the Image and Dynamics Discriminators. Given a ground-truth frame Y t and the synthesised framẽ Y t , we use the VGG network [46] to extract visual features in different layers for both frames and compute the VGG loss as in [45] and [12]. Likewise, the feature matching loss is computed by extracting features with the two Discriminators D I and D D and computing the 1 distance of these features for a fake frameỸ t and the corresponding ground truth Y t . For the computation of the overall feature matching loss, we compute a loss term L G−D I f eat using features extracted by D I , as well as another loss term L G−D D f eat , using the dynamics discriminator D I and then we add them: The total objective for G is given by: To achieve a balance between the loss terms above, we set λ vgg = λ f eat = 10. The Image, Mouth and Dynamics Discriminators are optimised under their corresponding adversarial objective functions (see Supplementary Material). Facial dynamics.
One key factor to consider while designing the Dynamics Discriminator is how it could learn the complex realistic facial muscular interactions observed in monocular videos of facial performances. As discussed and validated experimentally in [16], off-the-shelf state-of-theart optical flow methods solve this problem without any prior knowledge, since they target any moving objects in the scene. When applied to faces captured in the wild, the estimated flow does not reflect faithfully the non-rigid and composite facial deformations. To tackle this, we employ a state-of-the-art network, termed as FlowNet2, for the optical flow estimation [47]. The authors of [47] trained this network on publicly available images after rendering them with synthesised chairs modified by various affine transformations. Starting from the pretrained models of [47], we fine-tune their network on the 4DFAB dataset [48]. This dataset has dynamic high-resolution 4D videos of subjects eliciting spontaneous and posed facial behaviours. Our finetuned FlowNet2 network is therefore utilised to estimate the optical flow matrix W referred to in equation 6.

Dataset and Implementation Details
We train and test Head2Head++ on a newly collected video dataset, which we make publicly available, consisting of eight different individuals. Each individual is depicted in one video, which is at least 10 minutes long and presents the target in diverse head poses and facial expressions. First, we apply face detection on each video, as a pre-processing step. We utilise [49] to obtain a bounding box per frame, and then we compute the average bounding box throughout frames. We extract the facial ROI of 256 × 256 pixels for each frame, according to this fixed bounding box. This way, the background remains as stable and unchanged as possible, from the beginning of the video until the end. Finally, the frames of each subject are split into a training and a test set. Please refer to the Supplementary Material for more details on the dataset. We base our Generator's architecture on our previous work [15]. All discriminators have the same architecture, adopted by pip2pixHD [45]. We train a separate person-and videospecific Video Rendering Network for each one of the eight target identities. Each trained model at the end is dedicated for the target video used during training and does not generalise to synthesise the same target person under different cloths and backgrounds (only video specific). Given a target video footage of 10 minutes, extracting the eyes and NMFC conditional input sequences and training, the GAN framework requires around 20 hours. Networks are optimised with Adam [50], for 40 epochs with an initial learning rate η = 2 · 10 4 , β 1 = 0.5, β 2 = 0.999, on a single NVIDIA GeForce RTX2080 Ti. During test time, our Head2Head++ pipeline performs head reenactment from web-camera captures in nearly real-time speeds (18 fps).

Supported Reenactment Methods
As already mentioned, we have chosen to use 3DMMs for the retrieval of 3D facial shapes from the source and target videos in order to address the target identity preservation problem, allowing us to disentangle expression from identity seamlessly. Therefore, Head2Head++ can be used both for full head reenactment and simple facial reenactment, depending on which set of camera parameters (target or source) are used during the creation of the conditional NMFC sequence. Head Reenactmenent. This is the main functionality of our method and arguably the most challenging, since it involves transferring the complete head motion from any driving video to the desired target subject. A head reenactment example is demonstrated in Fig. 2. Given the driving sequence (Johnson), our method synthesises a highly photo-realistic and temporally coherent video of the targeted identity (Trump). Face Reenactmenent. Instead of transferring all head movements from the source video, we can simply pass the facial expressions to the target identity. This is done by utilizing the original camera parameters, estimated from the target training sequence. In this case, we use the original eye landmarks extracted from the training sequence to drive synthesis. The advantage of face reenactment is that our Generator has encountered the same head poses and positions during training, which is beneficial when it comes to synthesising the hair, background and upper body areas, where no conditional information is available in the input. Although prior works have mainly used it as a manipulation method [3], [7], we perform face reenactment by generating all pixels within frames, not only a masked area of the face. Self Reenactmenent. During self reenactment, the source identity coincides with the target one. This is extremely useful for training and evaluating our model, since we are given access to the ground truth frames. Note that the driving sequences used during test have not been seen during training, since they belong to the test data split. Fig. 3 shows a self reenactment experiment. Ideally, the generated frames should be identical with the source ones. For the evaluation of the reconstructive ability of models in the experiments that follow, we use metrics such as the average pixel distance (APD) or masked average pixel distance (MAPD) between the synthesised and ground truth frames. In Sec. 4.3, we provide more details on APD and MAPD.

Quantitative Evaluation Metrics
We evaluate the performance of our system both qualitatively and quantitatively. Most experiments were performed under self reenactment. That is, given a set of test frames Y (i) for each target identity i in the dataset, we generated the corresponding synthetic framesỸ (i) and used Y (i) as ground truth. We assessed our method's reconstructive ability, target identity preservation, pose and expression transferability as well as the photorealism of generated frames, using the metrics below: Average Pixel Distance (APD): is computed as the average L2-distance of RGB values across all spatial locations and frames, between the ground truth and generated data. Masked Average Pixel Distance (MAPD): similar to APD, it tests the reconstructive performance. A mask computed from NMFC frames is used to constrain the metric on the facial area, where conditional information is available. Distance between Average Identities (DAI): after performing 3D face reconstruction on Y (i) andỸ (i) sequences, we compute the average identity coefficients for each sequence, and then the L1-distance between those average identities. Average Expression Distance (AED): after reconstructing the fake and ground truth sequences, we measure the average L1-distance between the expression coefficients across frames. Average Rotation Distance (ARD): this metric is used to measure pose transferability. Using the estimated camera parameters from the generated and ground videos, we compute the Euler angles that correspond to head poses. Then, the average Euler angles discrepancy across sequences is determined in terms of degrees. Fréchet Inception Distance (FID): this is a widely used metric for evaluating the visual quality of individual frames, originally proposed in [51]. We use [52] as a feature extractor and we report the mean FID throughout the experiments. Maximum Mean Discrepancy (MMD 2 ): is a very useful metric for measuring the discrepancy between real and fake frames [53].

Ablation Study
We conducted an ablation study, in order to assess the significance of several "key" components of our Head2Head++ system. First, we evaluate numerically the contribution of the two reconstructive loss terms, namely the feature match-  ing and VGG loss. As can be seen in the first two rows of Table 1, all metrics demonstrate the contribution of these loss terms on the quality of the generated samples. The next row of the table validates the importance of a sequential Generator, as opposed to its non-sequential variation ("w/o seq. G"). The results indicate that considering previously generated frames in the Generator has a very positive effect on the visual quality of samples. Finally, the majority of metrics show that removing the Dynamics Discriminator ("w/o D D ") might slightly improve the quality of frames, when seen as individual images. We believe this is due to the fact that the proposed metrics analyse solely the spatial content of frames and do not evaluate temporal information.
Removing D D network would actually reduce the temporal stability and coherence of videos. Such behaviour is more apparent in areas where no conditional information is available (e.g. hair, background, upper body). In order to understand better the significance of the Dynamics Discriminator in the visual plausibility of the output video, please refer to our Supplementary Video. Arguably, the eyes and the mouth, are the facial areas that might be the first to expose the "fakeness" of a synthetic video. In Fig. 4, we illustrate the importance of the dedicated Mouth Discriminator, initially proposed in [15]. As can be  seen in the second row of Fig. 4, when the Mouth Discriminator is not included in GAN training, teeth details make the generated video appear less realistic, as opposed to the results in the first row, where the Generator was supervised by D M network. Fig. 5 demonstrates the significance of conditioning the Generator on the eyes input sequence. Given that NMFCs do not adequately reflect eye gaze and blinking, conditioning solely on the NMFC input sequence does not provide substantial information to the Generator, for transferring the eye movements of the source to the target. Additionally, we quantitatively evaluate the significance of the mouth discriminator and the eye gaze conditioning. Table 2 presents the numeric measures obtained when assessing their effect. In the second column of Table  2, we report the average pixel distance (APD) within the mouth area under self-reenactment, between the generated and ground truth regions of interest, for all eight identities in our dataset. As can be seen, the numeric results agree with the qualitative ones, in Fig. 4. In the third column of Table 2, we assess the importance of conditioning the Generator on the Eyes input (Eyes Video sketch shown in Fig. 1). For that, we use the landmarker to extract eye landmarks, both from the ground truth and synthetic frames, created under selfreenactment. Then, we compute the average eye landmarks L2-distance (AELD), between real and generated frames, across all eight identities and report the results in the Table  2. We observe that conditioning on the eyes helps to synthesise faces that better follow the eye movement and gaze appearing in the source video. Lastly, in order to evaluate the importance of the Dynamics Discriminator D D on the temporal consistency of generated frames, we use the Fréchet Video Distance (FVD) [?]. We also provide the FVD score for the non-sequential variation of Head2Head++ (w/o seq. G). The results of Table 3 indicate that our full model with a sequential Generator trained alongside a Dynamics Discriminator outperforms both variations by a significant margin. For that experiment, we trained five separate models per target identity. Each model was trained on frame sequences of different length: one, two, four, eight, up to sixteen thousand frames. We found that FID and MMD 2 scores, which indicate visual quality, are the metrics affected the most by the number of training frames. Furthermore, it is visible that there is still room for improvement in the performance of our Video Rendering Network, provided that even longer training videos are available. However, this would require additional footage with enough head pose variability, which in practice does not scale linearly with the number of training frames available. In Fig. 6, we demonstrate the impact of training video length on the visual quality of synthetic frames. We observe visually that as the number of available training frames increases, artifacts become less severe and eventually disappear. This behaviour comes in agreement with the decrease of evaluation metrics, shown in Fig. 7.

Pre-training on Face Forensics++ Dataset
We further explore how the generative performance of the Video Rendering Network can be improved in the case of limited video footage, for instance when training sequences are less than four thousand frames (about 2.5 minutes).
To that end, we pre-trained Head2Head++ on the largescale Face Forensics++ dataset [54], containing 1000 different identities. At this stage, the Generator learned to create a "average identity", since there is no mechanism to dictate which identity to be synthesised over this multi-person dataset. As a next step, we fine-tuned eight person-specific models, one for each target identity in our dataset, using a subset of their available footage (1K, 2K or 4K frames). In Table 4, we report a quantitative evaluation with and without pre-training on [54] data. As suggested by the results,  pre-training on Face Forensics++ data appears advantageous, especially for shorter sequences (1K or 2K frames). This could be explained if we consider pre-training as a head start for the GAN, where the Generator is already capable of creating a realistic "average identity" and then it remains to learn a specific one. The same trend can be seen in Fig.  8. Especially for a training footage of 1K or 2K frames, the quality of generated frames is clearly inferior in comparison with samples created with pre-training.

Comparison with the State of the Art
We compare our Head2Head++ system with the image-based method of DVP [4] and video-based Head2Head [15]. This is done by performing a full head reenactment experiment on the same source and target video footage. First, we trained our method and Head2Head on the target sequence, which was kindly provided by the authors of [4], and then we used the same source to drive synthesis. In Fig. 9 we display some Source Ours DVP Fig. 9. Comparison of our method with DVP. In many cases our method outperforms DVP, especially in terms of head pose transferability. examples, in which our method outperforms DVP in terms of head pose and expression transferability. For a quantitative comparison of the three methods, we performed 3D face reconstruction on the two synthetic videos and computed the average expression distance (AED) and average rotation distance (ARD) across frames, between the source video and the three generated videos. Then, we applied 3D face reconstruction on the target video and computed the distance between the average identities extracted from this real and each one of the three fake sequences. Finally, we used the FID and MMD 2 scores as a photo-realism metric, both computed between the fake videos and a set of frames from the target's original footage.
For the eye region, where significant visual improvements have been made over Head2Head, we computed the average eye landmarks L2-distance (AELD), between source and generated videos. The results presented in Table 5 suggest that our method outperforms DVP on every single metric and Head2Head in several. More specifically, data samples produced by Head2Head++ exhibit significantly lower FID and MMD 2 scores, when compared to these created by Head2Head++. This can be explained by the fact that the network (ArcFace [52]) we employ to extract facial features for FID and MMD 2 is very sensitive to the shape of the eyes and thus imperfections and inconsistencies in the eyes region generated by Head2Head are reflected in both photo-realism metrics.
Considering the AELD metric, the difference between the two methods is very small, which can be justified by the fact that the eye tracker used in Head2Head is quite reliable on eye region synthesis. We synthesise more photo-realistic eyes in comparison to [15]. Please refer to the supplementary video for more examples. and the video mostly frontal, therefore differences with Head2Head++ can be better understood in terms of photorealism. In Fig. 10, we show that our new eye motion extraction method results in more reliable eye synthesis, compared to Head2Head [15], in terms of photo-realism and eye shape. We observe that the eye regions generated by Head2Head seem unnatural, while Head2Head++ has successfully rendered photo-realistic eyes. The AED, ARD and DAI metric values obtained for Head2Head seem to be slightly better than Head2Head++. We attribute these results to the fact that Head2Head is equipped with a video-based 3D reconstruction approach that relies on all video frames to generate the 3D faces, taking a few minutes compared to few milliseconds in the case of Head2Head++. Nonetheless, we strongly believe that performing 3D face recovery in nearly real-time outweighs the small disadvantage we noticed in terms of facial expression accuracy.
We further compare our method with the warping-based X2Face method [2] as well as the one-shot, image-based First Order Motion Model for Image Animation (FOMM) [40] system. As suggested visually in Fig. 11 and the quantitative results in Table 6  the inherent ambiguity when the desired head pose is very different from the one displayed on that single reference image. Unavoidably, this makes the identity preservation problem more prominent in one-shot models.
In order to demonstrate the significance of the NMFC representation, we conducted a head reenactment experiment conditioning on landmarks. To that end, we trained vid2vid [12] on the task of landmark-to-RGB video translation. We demonstrate the results in Fig. 12. As can be seen, the facial shape of the source has been transferred along with the pose and expression to the generated target.

Automated Study
The progress recently made by generative deep learning methods is so impressive to the extent that manipulated videos (deepfakes) by such approaches are indistinguishable from the real ones. Consequently, a new research topic has emerged to tackle the detection of what is commonly known as 'deepfakes'. A recent attempt in that direction is the Face Forensics++ [54]. The authors of [54] collected a dataset of 1000 YouTube videos and manipulated them with graphics-based [3], [55] and learning-based [7], [56], facial reenactment methods. With this dataset, a CNN was trained to outrank the performance of human observers in detecting manipulated videos. Such a network was trained on 1.8 million facially manipulated frames and reached a very high detection accuracy on the test split of their dataset (around 99%). We use their pre-trained best-performing network and fine-tune it on around 14K frames synthesised from eight different subjects by our Head2Head++ approach and another 14K frames synthesised by Head2Head. While fine-tunning, we set the learning-rate to 0.0001 and batchsize to 32. Adam optimiser [50] was also used with the default parameters. We then test the fine-tuned network on 3.2K frames generated by Head2Head++ and Head2Head methods (1.6K frames each) and not seen during training. The trained network reports an accuracy of 76% vs 80.2% of fake frames detection for Head2Head++ and Head2Head, respectively. This indicates that the fine-tuned forgery detection network finds it easier to spot fake frames generated by Head2Head compared to Head2Head++. Moreover, the same network performs way better (around 99%) in detecting fake frames synthesised by any of the four headreenactment methods tested in [54]. Thanks to our carefully designed Head2Head++ framework, the trained forgery detection network finds it harder to identify our fake frames, which can be attributed to the increased realism of our results.

User Study
We further evaluate the photo-realism of our generated videos, by conducting a user study. For that, we synthesised five videos of 75 frames (3 seconds) each, using different reenactment methods (head, face and self reenactment). Then, we coupled each fake video with two real ones, with the same duration and same target identity, forming triplets of videos. Next, we asked 50 participants on MTurk to detect the fake sample of each triplet. We presented the video triplet to the participants, allowing them to watch them only once.
Our recorded results indicate a human fake detection accuracy of 43.1%, which demonstrates the strong photo-realism of samples created by our Head2Head++ system. In Fig. 13a, we show the relative frequency of participants on the number of fake videos that were successfully detected. As can be seen, the majority managed to spot between 4 and 8 out of the 15 synthetic samples we displayed. The comparison of our statistics with a Binomial distribution (red curve) indicates that user study participants performed slightly better than random selection. Nonetheless, a significant percentage of them, around 18%, managed to detect 9 or more fake samples. Finally, Fig. 13b displays the predictive accuracy of participants on each video triplet. It is evident that self reenactment samples were indistinguishable from real ones, with just 32.8% predictive accuracy. Videos generated under face and head reenactment were identified a little more successfully, with an accuracy of 47.2% and 49.2% respectively.

CONCLUSION AND FUTURE WORK
We proposed Head2Head++, a pipeline consisting of a novel 3D facial reconstruction system and a Video Rendering Network, able to perform full head reenactment from a source to a target identity. Our new 3D face recovery and eye tracking methods allow our system to operate in nearly real-time speeds. The extensive set of experiments we performed demonstrated the generative abilities of our system, as well as the significance of its components. Furthermore, qualitative and quantitative comparisons suggest that Head2Head++ outperforms other state-of-the-art methods, in terms of photo-realism, expression and pose transferability, as well as target identity preservation. For future work, we aim to make the network trainable in a few-shot manner, reducing the training time and eliminating the need for learning a different model per person. Additionally, we plan to transfer the facial expressions from the source in a way that preserves the target's speaking style.