{'fraction_non_alphanumeric': 0.030637108048731775, 'fraction_numerical': 0.010345516277211904, 'mean_word_length': 4.9995207284926915, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
FaceSwapNet: Landmark Guided Many-to-Many Face Reenactment

Recent face reenactment studies have achieved remarkable success either between two identities or in the many-to-one task. However, existing methods have limited scalability when the target person is not a predefined specific identity. To address this limitation, we present a novel many-to-many face reenactment framework, named FaceSwapNet, which allows transferring facial expressions and movements from one source face to arbitrary targets. Our proposed approach is composed of two main modules: the landmark swapper and the landmark-guided generator. Instead of maintaining independent models for each pair of person, the former module uses two encoders and one decoder to adapt anyone's face landmark to target persons. Using the neutral expression of the target person as a reference image, the latter module leverages geometry information from the swapped landmark to generate photo-realistic and emotion-alike images. In addition, a novel triplet perceptual loss is proposed to force the generator to learn geometry and appearance information simultaneously. We evaluate our model on RaFD dataset and the results demonstrate the superior quality of reenacted images as well as the flexibility of transferring facial movements between identities.


Introduction
Face reenactment is a task to transfer the facial expressions and movements from one source face to another target face, advancing huge potential applications such as face editing, movie making, and augmented reality, etc. Existing methods can generally be categorized into two classes: parametric face synthesis and data-driven face generation. In the first case, the human face is represented by a predefined parametric model for explicit manipulation of facial movements [28,27,16]. Given a source face, these methods fit it into facial pose and expression space with the predefined model and then render the target face. The parametric model can provide straightforward control over face movements. However, well-defined parametric face models require large efforts and delicate designs. On the other hand, data-driven methods [26,32,33] have natural advantages in learning patterns from a large-scale dataset. In early efforts, Xu et al. [33] directly apply image translation framework (CycleGAN [35]) to generate face images of the target character. This method does not explicitly capture facial geometric variances and thus it is unable to guarantee the movement consistency between its input and output. Among recent attempts of face reenactment, the best-performing method (ReenactGAN [32]) introduces facial boundaries as the latent representation to robustly encode facial movements. Given an arbitrary face under very different poses and expressions, such a state-of-the-art framework is capable of manipulating a target face. However, we argue that ReenactGAN has two limitations. Firstly, it is inefficient in many-to-many face reenactment task. The main reason is that it maintains a transformer network for each person in the dataset. More specifically, in order to learn all transformations among n human faces, n(n − 1) transformers have to be trained. Secondly, it is ineffective in the many-to-many task since its decoder module is designed for mapping a latent boundary to a specific person's face, i.e., the decoder simultaneously learns appearance and geometry information from a face boundary. This mapping performs well when the target person is one or a few predefined specific identities. However, similar landmarks will increase as the number of target persons grows, where the decoder will map similar landmarks to an average face. Figure 1 intuitively demonstrate this limitation.
To address above limitations, we propose FaceSwapNet to efficiently achieve many-to-many face reenactment. We first leverage face landmark detection technique [8] to encode two input images into landmark latent space. This latent space is a high-quality bridge for the next face adapting step since the facial geometry information is faithfully and efficiently preserved and the appearance information is omitted. Subsequently, our landmark swapper module is used to adapt any source person's landmark to the target person's landmark, for keeping identity consistency between generated persons and the target person, which is an essential requirement in face reenactment task. Finally, the landmark-guided generator simultaneously leverages geometry information from the swapped landmark and appearance information from the target person to generate the reenacted image. Such decoupling design is used to distinguish similar/same landmarks that are extracted from different persons with different appearances.
As far as we know, our work is the first one to successfully perform many-to-many face reenactment. Our contributions are three-fold. i) a unified swapping module to transform any source person's geometry to the reference person while keeping emotion information. ii) a landmark-guided generator that is designed to simultaneously use geometry and appearance information to generate the reenacted target face. iii) a novel triplet perceptual loss to force generator to learn geometry and appearance information simultaneously. Our code is available at https://github.com/zhangzjn/FaceSwapNet.

Related works 2.1 Image synthesis
Driven by remarkable results of GANs [7], researchers leveraged GANs to generate images in various domains, such as image translation [23,11,35,29], person image synthesis [4,21] as well as face generation [3,22,14,15]. Imposing L1 and adversarial losses between the generated image and its ground-truth, pix2pix [11] achieved incredible results in the paired image translation task, which includes edges to photo, aerial to map and semantic labels to scene, etc. With only imagelevel annotation, CycleGAN [35], DiscoGAN [17], and DualGAN [34] each proposed the cycle consistency loss to preserve key attributes between the input and the translated image. Furthermore, StarGAN [3] proposed a unified model for multi-domain facial attribute transferring and facial expression synthesis tasks, which learns the mappings between all available domains by only a single generator. Recently, Karras et al. [15] proposed a style-based generator that embeds the input latent code into an intermediate latent space, which can control the strength of image features at different scales and synthesis extremely naturalistic face images. These approaches are capable of learning to generate realistic images, but have limited scalability in handling face reenactment, where the facial  actions of output are required to be consistent with its input. Inspired by the above methods, we also leverage a generative adversarial network to synthesize realistic face images. Moreover, we introduce a face landmark latent space to capture facial movement.

Face reenactment
Benefiting from accurate 3D face modeling [2,6], large-scale face database collections [19,10,20], and reliable landmark detection techniques [8,31,5], numerous impressive face reenactment methods are proposed in recent years. Most of existing methods can be categorized into either model-based approaches or data-driven frameworks. In model-based branch, Blanz and Vetter [1] first proposed the 3D morphable model (3DMM) to characterize face geometry and textures, which learns affine parameters from 200 high-quality scans. Face2face [28] tracked both the expressions of the source and target actor's videos based on a statistical facial prior, and rendered the new video through morphing. Taking the coarse 3D face renderings of a target actor as input, Hyeongwoo et al. [16] proposed a rendering-to-video translation network that transforms face renderings into full photo-realistic portrait video outputs. In general, those parametric models can provide fully-controllable 3D face interior model over a target person. However, well-defined parametric face models require delicate designs. Therefore, researchers paid attention to data-driven approaches [26,32,33], which do not explicitly model hair and face appearance. In early efforts, [33,12] directly applied CycleGAN [35] to transfer face expressions between two identities. Recently, Wu et al. proposed ReenactGAN [32] that is capable of transferring facial movements from an arbitrary person's monocular video input to a target person's video. In contrast to the CycleGAN-based methods or ReenactGAN that can only transfer facial movements from source faces to one predefined target face, our framework aims at solving the harder many-to-many face reenactment problem, which permits more flexible application scenarios.

FaceSwapNet
Many-to-many face reenactment task aims at transferring facial expressions from one source person to a target person while keeping the identity of the target. Specifically, many-to-many here means that both the source and the target are not a predefined identity. In this paper, we propose a novel framework, FaceSwapNet, to efficiently achieve this task. As depicted in Figure 2, leveraging the face landmark detection technique [8], we first encode two input images I S,x , I T,r , where the first subscript means identity and the second is expression, to landmark latent space and get L S,x , L T,r . That space is very useful in face reenactment since it faithfully preserves the facial geometry information and omits the appearance information. The landmark swapper is subsequently used to adapt the source landmark to the target landmark, denoted as ψ : (L S,x , L T,r ) →L T,x . Finally, the landmark-guided generator simultaneously use geometry information of source person and appearance information of target person to generate the reenacted target face, denoted as φ : (L T,x , I T,r ) →Î T,x . We design the landmark swapper module to overcome the structural gap between the source face and the target. As mentioned in [32], it may lead to artifacts such as inapposite face outline that directly applying the source landmark to synthesis images about the target person. In contrast to existing methods [32,33], we use a unified module to adapt anyone's landmark to the target one. As shown in Figure 2 (top), the landmark swapper contains two landmark encoders and a landmark decoder. Encoders ψ α1 and ψ α2 extract features of target landmark and source landmark respectively. Then decoder ψ α3 fuses them and estimatesL T,x that has the face shape with L T,r while keeping the face emotion information of L S,x . This process is denoted as:

Landmark swapper
Our landmark swapper works based on a hypothesize that any person X has a specific and complete landmark space L X , i.e., same expressions of different persons S and T can find correspondence in the landmark space. For example, if we aim at acquiring L T,1 , we assume that there exists a corresponding L S,1 and both L S,1 and L T,r are leveraged to estimateL T,1 . This procedure is visualized in Figure 3. We design the landmark swapper as a fully-connected network with a vector input, and its training process is independent from the landmark-guided generator.

Landmark-guided generator
Given the swapped landmarkL T,x and the neutral emotion image of tartget person as a reference I T,r , the landmark-guided generator aims at generating the reenacted target faceÎ T,x . We design the landmark-guided generator with a decoupling thought, adapting network architecture from the stateof-the-art face generation framework [15]. Specifically, it is designed to learn geometry information from the swapped source landmark and the appearance information from the reference image. As shown in Figure 2 (bottom), the landmark-guided generator φ consists of an image encoder, a transformer, and an image decoder with parameters θ 1 , θ 3 , and θ 4 respectively. A landmark encoder with parameter θ 2 is added to extract landmark geometry information.
Ideally, the image encoder φ θ1 extracts the appearance features from the reference image I T,r and the landmark encoder φ θ2 uses swapped landmarkL T,x to extract geometry features in a multi-guided manner. The transformer φ θ3 integrates decoupled appearance features and geometry features. The image decoder φ θ4 serves as a decoder to reconstruct the reenacted imageÎ T,x that contains source geometry information while keeping the identity of I T,r . This process is denoted as: Such decoupling design avoids directly mapping a latent landmark to a specific image, whose limitation can be seen in Figure 1, since each input provides a specific information that contributes to the reenacted image.

Objective functions
Triplet perceptual loss. We find the training process of our generator is ill-conditioned if only under the supervision of adversarial loss [7] and L1 loss, where the generator probably only learns a mapping between the input landmark and the generated image. Especially, we conduct an experiment that only contains one target person. Its result (shown in the supplementary material) indicates that the generated image is almost associated with the input landmark not matter what the reference image we feed. For example, the generator ouputs the target person even feeding a black image. When more target identities are included in the training dataset, the phenomenon drops but still exists. If the given landmark is not in the training dataset, the generator will map it to an average face, which can be seen as a linear combination of similar identities in the dataset. It is harmful to achieve many-to-many face reenactment especially when the source person is not in the dataset.
This ill-condition comes down to two reasons: first, the RGB image and landmark have a different distribution where the generator tends to learn from the landmark since its distribution is more simple. So we design the landmark-guided generator to split the reference image and landmark. Second, our vanilla generator does not distinguish the image generated from similar landmarks. So we propose a novel triplet perceptual loss which maximizes inter-class perceptual variation and minimizes intra-class perceptual variation. During the training phase, two arbitrary emotion images I T,x1 and I T,x2 are randomly selected within the target person T while the third image I X,x3 is randomly selected with another person X in arbitrary emotion x 3 . Then imagesÎ T,x2 ,Î T,x3 ,Î X,x2 are generated by the landmark-guided generator where it takes (L T,x2 , I T,x1 ), (L T,x3 , I T,x2 ), and (L X,x2 , I X,x3 ) respectively. Following the original perceptual loss [13] and the triplet loss [24], we define a novel triplet perceptual loss to distinguish images generated by similar landmarks, denoted as: where m is the margin value for controlling intra and inter gaps; κ(·) means features extracted by VGG [25]; D(·, ·) means L1 distance.
Without triplet perceptual loss,Î T,x2 is only a intra-class transferring under the supervision of its ground-truth image, where it tend to couple the landmark and the generated image naturally. With triplet perceptual loss,Î T,x2 have additional intra-class/inter-class constrains from generated imageŝ I T,x3 andÎ X,x2 . In this case, the generator has to simultaneously leverage the reference image and input landmark to generate images, since triplet perceptual loss let one landmark participates in the generation of all target person.
Full objectives. During the training phase of the landmark-guided generator, we totally use three losses: pixel-wise L1 loss L pixel for intra-class swapping( (L T,x , I T,r ) →Î T,x ), adversarial loss [7] L adv for improving realism of generated images, and triplet perceptual loss L T P for intra-class and inter-class transferring. The full loss function L total is defined as: where λ i denotes the weight of i-loss, respectively.

Experiments
We first quantitatively and qualitatively evaluate our model in a famous RaFD dataset. Then, we conduct ablation studies to demonstrate the effect of each proposed component in our framework. Finally, a series of landmark interpolation and manipulation experiments are performed to highlight the decoupling design of our generator.

Datasets and implementation details
RaFD. The Radboud Faces Database (RaFD) [19] consists of 8,040 images collected from 67 participants with image size 512 × 512. Each participant makes eight facial expressions in three   Figure 5: Comparison against the baseline. The first column is reference images and the first row is source images (the right four are not in the dataset). Images on the second and third rows are generated by pix2pix and the rest are generated by our method. Zoom in for details. different gaze directions, which are captured from five different angles. We use all 45 • , 90 • and 135 • face images to perform face reenactment in this paper. All the images are cropped to 416 × 416 with face-centered and then resized to 256 × 256. The landmark with 106 keypoints for each face image is provided by HyperLandmark [8].
Evaluation metrics. We use the Amazon Mechanical Turk (AMT) to evaluate the visual quality of synthesized results. We also apply Structural SIMilarity (SSIM) [30] to measure the structural similarity between generated images and their targets, and Fréchet inception distance (FID) [9] to measure the realism and variation of generated samples.
Implementation details. For the landmark-guided generator, our network architecture is adapted from pix2pix [11]. We use Adam [18] optimizer and set β 1 = 0.5, β 2 = 0.999. The initial learning rate is set to 2e −4 and batch size is 4. For the discriminator, we use PatchGAN proposed in [11] and training settings are the same as the generator. For the landmark swapper, ψ α1 and ψ α2 consist of three and four linear layers respectively, and ψ α3 contains three linear layers. We use Adam optimizer with β 1 = 0.9, β 2 = 0.99. The initial learning rate is set to 1e −4 and batch size is 32. Detailed network architecture can be found in supplementary material.

Quantitative results
To verify the effectiveness of the proposed method, we use the slightly modified pix2pix [11] as our baseline where the landmark is seen as the fourth channel concatenated to the input image. From the comparison results shown in Table 1, we can find that our full method outperforms the baseline in reference source Figure 6: Ablation study on RaFD. The second row is generated by vanilla landmark-guided generator (LGG). The results after adding landmark swapper (LGG+LS) are shown in the third row and the last row is generated by our full method (LGG+LS+TP).
all metrics. Benefiting from the landmark swapper module, our method has achieved a significant improvement for SSIM metric. Our network also achieves lower score for FID metric. We further perform a user study on Amazon Mechanical Turk (AMT) to compare human perceptual error against the baseline. For each of 30 testers, 67 real images and 67 fake images are shown in random order with unlimited decision time. Overview, our generated images confuse testers on 74.9% trials, i.e., 74.9% generated images are recognized as real image, while this value in the baseline is only 41.3%. As a reference, the percentage of true positive samples is 71.1%, which is slightly lower than our generated images.
On the other hand, we compare against three baselines on model parameters to verify the efficiency of our method. As shown in Table 2, the model parameters of our method is slightly more than the modified pix2pix but much lower than the other two baselines. The reason is that FaceSwapNet requires only a single generator, discriminator and swapper to achieve many-to-many face reenactment, while in the case of cross-domain based methods such as ReenactGAN, independent models are needed to be trained for each source-target pair.

Qualitative results
We next present and discuss a series of qualitative results that demonstrate the high quality of generated images and the flexibility of our framework. As shown in Figure 5, those experiments are conducted in diverse source-target pairs where the source persons can be arbitrary and the targets are in various identities and poses. For image quality, comparing the baseline (the second and third rows) and our method (the fourth and fifth rows), we can find that FaceSwapNet can preserve geometry information of reference images and thus generate better face reenactment results. For example, the generated images of baseline at the column (k) are unable to keep the face outline as the reference images, while our method can achieve it. Moreover, our model performs better at details such as the upper lip (the column (l)), nose (the column (m)) and teeth (the column (h)). For transferring flexibility, we randomly choose eight identities with different emotions from the training dataset and four identities with a random expression outside the dataset as source images. Then their expressions and movements are transferred to three target persons in three poses. The generated results are photo-realistic and expressive where the face appearances and outlines are consistent with the reference images.

Ablation study
We conduct an ablation study to verify the impact of each component of our proposed framework. As shown in Table 1 and Figure 6, we report evaluation results of the different versions of the proposed method. Firstly, we observe that our vanilla landmark-guided generator can generate realistic images whether the source images are in the dataset or not. However, it is unable to preserve the geometry information of the target person though it gets the best score on FID. Secondly, comparing the second row and the third row in Figure 6, we can learn that transferring the landmark from source to reference significantly enhances the performance of face reenactment, since it notably improves the identity consistency between generated images and the reference. The effect of the landmark swapper is also Figure 7: Landmark interpolation results. In each row, the first is a reference image while the second and last images are used to provide two original landmarks. The rest images are generated by decoding interpolated landmarks on the reference image. Note that three input images are arbitrary. evaluated in quantitative results on SSIM, where the score is increased by a large margin with that module. Finallly, we examine the effectiveness of the proposed triplet perceptual loss. Observing the last row in Figure 6, we find that when triplet perceptual loss is added, the generated images are more various in details, e.g., brows, wrinkles and mouth corners. Those enhanced details improve the realism of the generated image, which is indicated by the score on FID.

Landmark interpolation and manipulation
We also present a series of qualitative results to highlight the advantages of decoupling design in our generator. Such design brings two convenient ways to manipulate image attributes: landmark interpolation and landmark manipulation. In contrast to traditional interpolation operating in pixel space, our method interpolates images in the landmark latent space and the transformations are then decoded on a reference image. More specifically, we extract two original landmarks from two source images and get a series of similar landmarks thorough interpolating two landmarks. The geometry information in those landmarks is then transferred to a reference image. As a result, we get a series of interpolation images about the reference. Theoretically, this processing spams a single point (the reference image) in image space to its local subspace. The visualized results are shown in Figure 7.
The other way to change image attributes is directly manipulating the input landmark. That provides a flexible pixel-level way to fine tune generated details. As shown in Figure 8, several landmark manipulation experiments are performed to change the generated person's partial attribute by operating corresponding points in the input landmark. The results demonstrate that our method can keep the identity and other attributes while manipulating a specific attribute, which intuitively indicates the advantages of decoupling appearance and geometry learning.

Conclusions
In this paper, we proposed FaceSwapNet to address a novel many-to-many face reenactment task, which aims at transferring the facial expressions from source persons to targets while keeping the identity consistency to the targets. A landmark swapper module is proposed to effectively adapt the landmark of any source person to the target person. A landmark-guided generator is designed to distinguish similar/same landmarks that are extracted from different persons. A triplet perceptual loss is further proposed to force the generator to learn geometry and appearance information simultaneously. Extensive experimental results demonstrate the high quality of generated images and the flexibility of the proposed framework.