{'fraction_non_alphanumeric': 0.03425273968976653, 'fraction_numerical': 0.011541108581714225, 'mean_word_length': 4.949606299212598, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Lip movements information disentanglement for lip sync

The lip movements information is critical for many audio-visual tasks. However, extracting lip movements information from videos is challenging, as it can be easily perturbed by factors like personal identities and head poses. This paper proposes utilizing the parametric 3D face model to disentangle lip movements information explicitly. Building on top of the recent 3D face reconstruction advances, we firstly offer a method that can consistently disentangle expression information, where the lip movements information lies. Then we demonstrate that once the influences of perturbing factors are alleviated by synthesizing faces with the disentangled lip movements information, the lip-sync task can be done better with much fewer data. Finally, we show its effectiveness in the wild by testing it on an unseen dataset for the active speaker detection task and achieving competitive performance.


INTRODUCTION
For audio-visual tasks such as lip-syncing and lip-reading, lip movements are the most valuable information inside the facial video for audio-visual tasks, as they directly correlate with speech.
Many previous works use mouth-centered video clips as visual inputs and train a visual encoder to extract the lip movements information. However, directly learning lip movements information from those loosely registered facial video clips is of challenges, as it is easily perturbed by factors like large head poses. In the literature, it has been noticed that, on several audio-visual tasks, there exists a considerable performance degradation when inputting faces are in large head poses. Thus a large amount of non-frontal faces, obtained either by collecting real images [1] or by synthesizing via data augmentation [2], are required for training.
However, the head pose is not the only perturbing factor. For tasks like lip-syncing or lip-reading, to be speakerindependent, the influences of person-specific attributes, like identity and appearance, should also be alleviated. Therefore, many data covering diverse combinations of attributes are required for learning.
Complementary to the utterly data-driven approach, explicitly alleviating the influences of perturbing factors are shown to be effective. Face frontalization, achieved using either generative models [3] or warping [4], is shown to be beneficial, as they explicitly alleviate the influence of head pose by synthesizing a frontal face. In addition, previous works demonstrate that by explicitly disentangling personindependent content information, the lip-reading performance is improved on unseen subjects [5].
The 3D Morphable Model (3DMM) [6] is attractive for the lip movements information disentanglement. In 3DMM, a parametric space is defined where lip movements information, lying in the expression space, is independent of other attributes. Ideally, the lip movements information of a video clip can be obtained by finding the proper expression parameters for its frames. However, as indicated by [7], in reality, the expression space is not strictly disentangled with the space of person-specific attributes, including identity and texture, thus causing ambiguities. For example, in the 3DMM space, a thick-lip subject in neutral expression may be modeled equally well with parameters of a thin-lip subject with a bit of pouting expression.
For resolving ambiguities, one practical constraint is that images of the same subject should be of the same person-specific attributes. With person-specific attributes constrained, all other non-rigid facial morphings can ascribe to the expression.
This constraint is widely used in the image-collections based 3D face reconstruction literatures. In recent advances, Deng et al. [8] propose to aggerate person-specific attributes estimated from multiple images of the same subject and treat the aggregated person-specific attributes as the final estimations for those images. Tewari et al. [7] aggregate low-level features from multiple images and use the fused features to estimate global person-specific attributes for all those images.
However, aggregation strategy methods can only achieve a weak consistency among images. Since the features of each image are utilized for estimating both the expression and the person-specific attributes, it is inevitable that on some images, they are prone to adjust the person-specific attributes for a better 3D face modeling, while on others, they are prone to use the expression, causing inconsistency.
Differently, other works [9][10] choose to do a joint estimation on multiple images, optimizing person-specific at-tributes shared among images together with other pre-image attributes. Unlike aggregation strategy methods, the features of each image focus on estimating the pre-image attributes, leaving the person-specific attributes image-independent. Hence those methods achieve a consistent disentanglement between expression and the person-specific attributes, despite its simplicity. However, existing approaches either need to retrain a network [10] for each specific video or are not deep-learning-based and thus are less robust in the wild [9].
In this paper, we show that by simply introducing the idea of treating the person-specific attributes as imageindependent variables into the existing image-collections based 3D face reconstruction algorithm like [8], a method that can consistently disentangle expression, where lip movements information lies, can be achieved. Then we demonstrate its effectiveness on the lip-sync task, whose objective is to find the time offsets between audio segments and facial video clips, by learning a competitive lip-sync model with much fewer data. Finally, we further show that the lip-sync model, trained with disentangled lip movements, can achieve competitive performance on an unseen in-the-wild dataset for the activate speaker detection(ASD) task.

PERLIMINARIES
In this section, we introduce models required for 3D face reconstruction.
The 3DMM. Within the 3DMM, the face shape S and the face texture T are represented as linear models: whereS andT are the mean face shape and the mean texture; B id , B exp , and B tex are three PCA bases for identity, expression, and texture, respectively. The Basel Face Model 2009 [11] and FaceWarehouse [12] built expression bases are adopted, resulting in identity coefficients α ∈ R 80 , expression coefficients β ∈ R 64 , and texture coefficients δ ∈ R 80 . The illumination model. Under the distant smooth illumination assumption and the Lambertian skin surface assumption, the Spherical Harmonics(SH) [13] is employed to model the per-vertex radiosity as where n i is the normal, t i is the texture color, Φ b are SH basis functions. We choose B=3 for each of the red, green, and blue channels, resulting in illumination coefficients γ ∈ R 27 . The image formation model. In the camera space, the 3D face pose p ∈ R 6 is represented by a rotation R ∈ SO(3), parameterized with three Euler angles, and a translation t ∈ R 3 . Thus, the image formation model can be presented as where Π : R 3 → R 2 is the perspective projection matrix, whose values, including image centers and focal length, are empirically defined, similar to [8].  In this section, we describe the proposed disentangled expression extraction algorithm and a face synthesis pipeline to alleviate the influences of perturbing factors.

Training Dataset Construction
A multi-image dataset is constructed for training, similar to [7]. First, many single-speaker videos are collected. For each video, we perform the face alignment and detect landmarks on its frames. Then, the good quality face crops and corresponding landmarks are normalized to 224x224 pixels. Among those, M face crops with enough pose and expression variations are roughly selected according to lip landmarks distances (LMD) [14]. Finally, the training dataset is structured as K sets, and each set contains M pairs of face crops and 3D landmarks from one video of one person, i.e.

Expression Extraction Model Architecture
As illustrated in Fig.1(a), the model consists of a regression network and two trainable matrices. We denote the regression network as the E-Net, responsible for estimating preframe attributes. Similar to [8], we use a ResNet-50 backbone with a fully connected layer to regress a 97-dimensional vector, which comprises expression coefficients (β ∈ R 64 ), illumination coefficients (γ ∈ R 27 ), and pose coefficients (p ∈ R 6 ).
Regarding the two trainable matrices, one is denoted as the shared identity matrix, which is of dimension 80-by-K, where 80 is the dimension of identity coefficients (α ∈ R 80 ) and K is the number of sets in the training dataset. It is denoted as shared because the k-th column of the matrix represents the identity coefficients shared among face crops of the k-th set. Similarly, the other matrix is denoted as the shared texture matrix, and it is of dimension 80-by-K since δ ∈ R 80 .
In the inference phase, only the E-Net is required to regress the expression coefficients. Note that we aim to extract disentangled expression rather than 3D face reconstruction, so not all attributes are required.

Loss Functions
Within the Analysis-by-Synthesis framework [6], for an estimated coefficientsx = {α k ,δ k ,β,p,γ}, several losses are utilized to measure the discrepancy between the synthesized data and the captured data, following [8].
The robust photometric loss. It measures the photometric consistency between the real image and the synthesized one, on the skin area.
where I i (x) is the synthesized image, M is a binary mask indicating the rendering area, and A is a mask indicating the per-pixel skin probability.
The landmarks reprojection loss. It measures the geometric consistency between reprojected model vertices and detected 3D landmarks.
where L n denotes reprojected landmarks, and ω n is the landmark weight. In particular, we set ω n = 10 for landmarks on face contour and nose to increase pose estimation accuracy, ω n = 20 for landmarks on the lip to emphasize the accurate lip fitting, and ω n = 1 for the rest.
The face recognition loss. It measures the consistency between the real face image and the rendered one at the perception level. In particular, a pretrained ArcFace [15] model is utilized to extract feature vectors of both images, and their similarity is measured by computing the cosine distance, as where f represents the ArcFace model. The regularization loss. It adds zero-mean Gaussian distribution prior constraints on 3DMM coefficients.
where we set ω α = 0.8, ω β = 0.8 and ω δ = 2.0. Compared to [8], we assign more constrains on the expression to prevent it from catching all non-rigid facial morphings and causing person-specific attributes become trivial.
The texture regularization loss. It adds a constraint preferring a uniform skin albedo, which helps disentangle texture and illumination.
where T is the set of skin vertices, T is the number of vertices in the set, and µ = 1 T i∈T (t i (δ)) is the estimated mean skin albedo.
During training, the loss is calculated batch-wise. In each batch B, we propose to sample multiple data from the same set, and hence, where W EN et denotes the learnable weights of the E-Net. Since data of the same set share the same person-specific attributes but are of different expressions, they provide constraints to each other and help resolve ambiguities between expression and person-specific attributes.

Face Synthesis
As Fig.1(b) depicts, we synthesize a new face with the expression extracted from the inputting face and all other attributes in default values (i.e., irrelevant to the inputting faces). In this way, we can explicitly emphasize the expression by alleviating the influences of perturbing attributes and background. In particular, we choose to set uniform white illumination, mean identity, mean texture, and frontal pose as default.

Datasets
For the proposed disentangled expression extraction algorithm and the lip-sync model, we use the VoxCeleb2 [16] for training and evaluation. The dev-split of VoxCeleb2 contains over 1 million video clips from 145569 videos of 5994 speakers, and its test-split containers 36237 video clips from 4911 videos of 118 speakers. We preprocess all video clips to be 25fps and the extracted audios to be 16kHz. For training the proposed disentangled expression extraction algorithm, as described in Sec. 3.1, a dataset with K = 80k sets is constructed, where each set is formed by merging video clips from the one video of a speaker, from which at most M = 50 frames are selected. Unlike [7], we construct sets at the video level instead of at the video-clip level, as many video clips are too short to cover diverse attributes combinations.
For training the lip-sync model, we create a minidev-split of 500 speakers sampled from the dev-split, and for each speaker, we evenly select 10 video clips from 2 videos. After eliminating video clips without enough duration for training [17], we get 4488 video clips in total. For the evaluation, we use the entire test split. Note that the minidev-split is only about 1/9 of the test-slipt in size.
Regarding the ASD task, we evaluate the trained lip-sync model on the test split of the Active Speaker in the Wild (ASW) dataset [18]. The test split comprises 53 in the wild videos, including 4.5 hours of active (i.e., speaking) face tracks (i.e., a sequence of face crops) and 3.4 hours of nonactive face tracks. In addition, we perform face preprocessing as described in Sec.3.1 within each face track to reduce jittering.

Results on Lip Movements Disentanglement
We first train the proposed disentangled expression extraction model with the constructed dataset. In particular, the E-Net is initialized with ImageNet [19] pretrained weights, and two matrices are randomly initialized with Gaussian noise N (µ = 0, σ = 0.01). Then, we train the model using Adam optimizer for 10 epochs, with a learning rate of 1.0e−4. For each batch, we sample 2 sets with 8 face crops from each. Common data augmentation methods, including affine transformation, jpeg compression, and Gaussian blur, are used.
We compare the proposed algorithm with one state-of-theart image-collections based 3D face reconstruction method [8], using its pretrained model. We quantitatively evaluate the expression disentanglement consistency by calculating the lip LMD on 100 video clips of different speakers randomly sampled from the test split. For [8], we report two results, one with person-specific attributes estimated preframe; the other with person-specific attributes aggregated from frames of high confidence. For the proposed algorithm, since the E-Net does not regress person-specific attributes during inference, we report one result obtained with the mean person-specific attributes and two results obtained with person-specific attributes randomly sampled from trained share matrices on trainset. As shown in Tab.1, for [8], the results obtained using aggregated person-specific attributes are worse than the results obtained using preframe estimations. It indicates that both expression and person-specific attributes are adjusted for lip fittings on most frames. In contrast, the proposed algorithm results are more consistent, indicating that non-rigid facial morphings are consistently ascribed to expression, thus less affected by changing person-specific attributes. Fig.2 shows some qualitative results from the test split. For [8], the faces synthesized with preframe estimated identities only, that with expressions only, and that with both attributes, are shown in the left three columns, respectively. It is clear that preframe estimated identities vary with the inputs, and both attributes contribute to the lip fittings. In contrast, illustrated in the fourth column, faces synthesized with expressions estimated by the proposed E-Net alone can consistently describe the lip movements well, indicating that nonrigid facial morphings are correctly ascribed to the expression attribute.
The quantitative and qualitative results demonstrate that the proposed algorithm can consistently disentangle the lip movements information.

Results on Audio-Visual Tasks
To demonstrate the benefits of explicitly alleviating the influences of perturbing factors, we consider the lip-sync and ASD tasks. For the lip-sync task, we follow the state-of-art method [17][20] to train a two-streams network, except that we use mel-spectrogram as audio features. First, the audio and visual features are extracted from every 0.2s segment, with a stride of 0.04s. Then, we compute the cosine similarities between each visual feature and all audio features with a ±15 visual frames range and determine the offset given the minimal distance on average. It is correct if the determined offset is within the ±1 visual frame from the ground truth offset given in the annotations, following [17].
Three variants are trained on the minidev split, with mouth-centered video clips, synthesized video clips using both expressions and person-specific attributes pre-frame estimated by [8], and synthesized video clips using expressions   [8], indicating that further alleviating the influences of person-specific attributes is beneficial. For the ASD task, we directly apply the best lip-sync model, the one trained with the proposed method, on the ASW test split. Following [18], an audio-visual pair with cosine similarity higher than a threshold is determined to be active. In our experiments, we first align the audio-visual streams using the offset predicted on this track. Then we calculate the pairwise cosine similarities with the aligned streams. Finally, similarities are averaged over 15 frames to improve estimation robustness as [17].
As Tab.3 shows, our method shows strong generalization ability by achieving performances on par with the strong baseline [18], in metrics including the average precision(AP), the area under the receiver operating characteristic(AUROC), and equal error rate(EER). Note that we do not finetune on the ASW dataset.

CONCLUSION
This paper offers a method for the lip movements information disentanglement and demonstrates its effectiveness on two audio-visual tasks. First, we show that using the parametric space of 3DMM, a disentangled expression extraction algorithm can be achieved by introducing the idea of personspecific attributes as image-independent variables into the existing image-collections based 3D face reconstruction algorithm. With the expression extracted from input, the lip movements information can be emphasized by synthesizing a new face with all other attributes in default values. Finally, we show that the lip-sync model trained with synthesized video clips achieves competitive performance on lip-sync and ASD tasks using much fewer data.