{'fraction_non_alphanumeric': 0.0339498104822899, 'fraction_numerical': 0.014507907463076722, 'mean_word_length': 4.827303884234578, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Learning Complete 3D Morphable Face Models from Images and Videos

Most 3D face reconstruction methods rely on 3D morphable models, which disentangle the space of facial deformations into identity geometry, expressions and skin reflectance. These models are typically learned from a limited number of 3D scans and thus do not generalize well across different identities and expressions. We present the first approach to learn complete 3D models of face identity geometry, albedo and expression just from images and videos. The virtually endless collection of such data, in combination with our self-supervised learning-based approach allows for learning face models that generalize beyond the span of existing approaches. Our network design and loss functions ensure a disentangled parameterization of not only identity and albedo, but also, for the first time, an expression basis. Our method also allows for in-the-wild monocular reconstruction at test time. We show that our learned models better generalize and lead to higher quality image-based reconstructions than existing approaches.


Introduction
Monocular 3D face reconstruction is defined as recovering the dense 3D facial geometry and skin reflectance of a face from a monocular image. It has applications in several domains such as VR/AR, entertainment, medicine, and human computer interaction. We are concerned with in-the-wild images which can include faces of many different identities with varied expressions and poses, in unconstrained environments with widely different illumniation. This problem has been well-studied, where a lot of success can be owed to the emergence of 3D Morphable Models [5]. These morphable models define the space of deformations for faces as separate disentangled models such as facial identity, expression and reflectance. These models are widely used in the literature to limit the search space for reconstruction [61,15]. However, these morphable models are often learned from a limited number of 3D scans, which constrains their generalizability to subjects and expressions outside the space of the scans.
Recent efforts examined learning face models with better Figure 1. We present the first approach for learning complete 3D morphable face models from in-the-wild images. Our approach learns models for identity geometry, expression and albedo, completely in a self-supervised manner. It achieves good disentanglement of the various facial components and produces high quality photorelism in the final overlay. generalizability from internet images or videos [51,52,54,55,56]. However, learning from in-the-wild data is highly challenging, requiring solutions for handling the strong inherent ambiguities and for ensuring disentanglement between different components of the reconstruction. Some approaches deal with a slightly easier problem of refining an initial morphable model pretrained on 3D data on in-thewild imagery [52,55,57,56,54]. Our objective is to learn face models without using any pretrained models to start with. The closest approach to ours is Tewari et al. [51], which learns only the models of facial identity geometry and reflectance from in-the-wild videos. However, they still use a pretrained expression model to help disentangle the identity and expression variations in geometry. We present the first approach that learns the the complete face model of identity geometry, albedo and expression from just from inthe-wild videos. We start just from a template face mesh without using any priors about deformations of the face, other than smoothness. This also makes ours the first approach to learn face expression models from 2D data. We achieve this through several technical contributions. We design a neural network architecture which, in combination with specially tailored self-supervised loss functions, enables (1) learning of face identity, expression and skin reflectance models, as well as (2) joint 3D reconstruction of faces from monocular images at state-of-the-art accuracy. We use a siamese network architecture which can pro-cess multiple frames of video during training, and enables consistent identity reconstructions and expression basis reconstruction.We use a differentiable renderer to render synthetic images of the network's reconstructions. To compare reconstructions to the input, we use a new combination of appearance-based and face segmentation losses that permit learning of overall face geometry and appearance, as well as a high-quality expression basis of detailed mouth and lip motion. Our novel lip segmentation consistency loss aligns the lip region in 3D with 2D segmentations. Our loss is robust to noisy outliers, leading to qualitatively better lip segmentations than the ground truth used. We also introduce a disentanglement loss which ensures that the expression component of a reconstructed mesh is small when the input image contains a neutral face. We show that the combination of these innovations is crucial to learn a full face model with proper component disentanglement from in-the-wild imagery, and out performs the state-of-the-art image-based face reconstruction.
In summary we make the following contributions: 1) the first approach for learning all components -identity, albedo and expression bases -of a morphable face model, trained on in-the-wild 2D data, 2) the first approach to learn 3D expression models of faces in a self-supervised manner, 3) a lip segmentation consistency loss to enforce accurate mouth modeling and reconstruction, 4) enforcing disentanglement of identity and expression geometry by utilizing a dataset of neutral images.

Face Modeling
Faces are typically modeled as a combination of several components: reflectance, identity geometry and expression. 3D parametric identity [5,3] and blendshapes [38,30,50] are used to represent identity (geometry and reflectance) and facial expressions. This generalizes active appearance models [12] from 2D to 3D space. To model the variations among different people, a parametric PCA space can be learned from a dataset of laser scans [5,3]. This represents deformations in a low-dimensional subspace. The resulting 3D morphable face model (3DMM) is the most widely used face model in literature [61]. Multi-linear face models extend this base concept; they often use tensor-based representations to better model the correlations between shape identity and expression [13,6,16].
Physics-based face models [23,49] have been proposed, however their complexity makes their use in real-time rendering or efficient reconstruction difficult. Animation artists can also manually create face rigs, with custom-designed control parameters. They often use blendshapes, linear combinations of designed base expressions, to control face expressions [30]. Recently, large collections of 3D and 4D (3D over time) scans have been used to learn face models. In [8] thousands of 4D scans are used to learn a parametric face model. Li et al. [32] used 33,000 3D scans to learn the FLAME face model. The model combines a linear shape space with articulated motions and semantic blendshapes.

Face Reconstruction
Image-based reconstruction methods [61] estimate face reflectance, geometry and/or expressions. 3DMMs [5,3] is often used as priors for this task. Methods differ in the type of input they use, such as monocular [42], multiframe [51] or unstructured photo collection input [43]. Current methods can be classified into 1) optimization-based and 2) learning-based. Optimization-based techniques rely on a personalized model [10,17,18,59,22] or a parametric prior [1,9,31,48] to estimate 3D geometry, often combined with texture and/or illumination, from a 2D video.
Learning-based approaches regress 3D face geometry from a single image by learning an image-to-parameter or image-to-geometry mapping [36,41,53,52,46,58,25]. These methods require ground truth face geometry [58,27], synthetic data generated from a morphable model [40,41,46,25], or a mixture of both [35,26,57]. Tewari et al. [53] propose a differentiable rendering-based loss which allows unsupervised training from 2D images. Genova et al. [20] learn to regress an image into 3D morphable model coordinates using unlabelled data. They impose identity similarity between the input and the output, in addition to a loop-back loss and a multi-view identity loss. Deng et al. [14] combined image-consistency and perceptual loss which leads to improved results. A new multi-image shape confidence learning scheme is also proposed which outperforms naive aggregation and other heuristics. RingNet [44] estimates the parameters of the FLAME model [32]. It utilizes multiple images during training, and enforces the shape to be the same for pictures of the same identity, and different for different people. While these techniques are fast and produce good results, reconstructing shape and appearance variations outside the pre-dedfined 3DMM space is difficult.

Joint Modeling and Reconstruction
Recent learning-based methods for monocular face reconstruction [52,56,55,7,47,51] allow for capturing variations outside of the 3DMM space by training from in-thewild data. Tran et al. [56] employ two separate convolutional decoders to learn a non-linear model that disentangles shape from appearance. Similarly, Sengupta et al. [47] propose residual blocks to produce a complete separation of surface normal and albedo features. Tewari et al. [52] learn a corrective space of albedo and geometry which generalizes beyond 3DMMs trained on 3D data. Lin et al. [34] refine the initial texture generated by the 3DMM albedo model. Lee et al. [29] learn a non-linear identity model, Figure 2. Our approach jointly learns identity, expression and albedo models along with the input-dependent parameters for these models. The network is trained in a siamese manner using differentiable renderer to compute self-supervised loss.
with a fixed existing expression model. Learning morphable models from scratch in a relatively less studied problem. Tewari et al. [51] learn the identity (shape and reflecance) model from community videos in a self-supervised manner. The learning starts from a neutral reflectance and coarse deformation graph, which are refined during training. It however relies on a learned expression model. Our method is the first to learn all dimensions-reflectance, identity geometry, and expression from in-the-wild data.

Method
We present the first method to learn a deformable face model that jointly learns all three of the following dimensions -identity geometry, expression and reflectance -from unlabelled community videos, without using a pre-defined 3DMM to start with. The starting point for our deformation models is a mesh which defines the topology of reconstructions, as well as the initial geometry and reflectance values for our networks. We design a multi-frame siamese network which processes the videos at training time. The training is self-supervised, without any 3D supervision. We use a differentiable renderer to define our loss functions in the image space. Our network design, in addition to loss functions enable disentangled learning of the face model subspaces. Our network also jointly learns to predict parameters of the models, thus enabling 3D reconstruction at test time, even from monocular images.

Model Representation
We learn a linear face models, similar to many existing face models [5,52,51], comprising linear models of identity geometry, expression geometry and albedo. (Stacked) Mesh vertex positions and reflectances are represented as V and R, |V | = |R| = 3N , where N is the number of vertices. We use the mesh topology of Tewari et al. [52] with N = 60, 000 vertices. Geometry Models 3D face deformations due to identity and expression can be represented using linear geometry models.
Here, M id ∈ R 3N ×mi and M exp ∈ R 3N ×me are the learnable linear identity and expression models. We use the mean face from [4] asV . α ∈ R mi and δ ∈ R me are the identity and expression parameters for the corresponding models. We use a low-dimensional embedded deformation graph to represent the linear models M id and M exp Here, M gid ∈ R 3G×mi and M gexp ∈ R 3G×me are linear models defined on a lower dimensional graph with G = 521 nodes. The fixed upsampling matrix U ∈ R 3N ×3G couples the deformation graph to the full face mesh and is precomputed before training. Learning the shape models in the graph-space reduces the number of learnable parameters in the model, and makes it easier to formulate smoothness constraints over the reconstructions.

Reflectance Model
We employ a linear model of diffuse face reflectance.
Here, M R ∈ R 3N ×mr is the learnable reflectance model, and β ∈ R mr are the estimated parameters. We use the mean face reflectance from [4] asR. Unlike geometry, we learn a per-vertex reflectance model on the full mesh resolution. This allows us to preserve photorealistic details of the face in the reconstructions.

Image Formation
Given a face mesh with positions V and reflectance values R, we additionally need the extrinsic camera parameters in order to render synthetic images. Rigid face pose is represented as φ(v) = Rot(v) + t, where t includes 3 translation parameters, and rotation Rot ∈ SO(3) is represented in 3 Euler angles. We use a perspective camera model, with projection function π : To define the color, we need to model the scene illuimination. We assume a lambertian surface, and use spherical harmonics (SH) coefficients γ to represent the illumination [39]. The color c of a point with reflectance r and position v can be computed as are the SH coefficients, n are the normals at point v and B = 3 .
Differentiable Rendering We implement a differentiable rasterizer to render 2D images from 3D face meshes. For each pixel, we first compute the 3D face points which project into the pixel. We use a z-buffering algorithm to select the visible triangles. Pixel color is computed by linearly interpolating between vertex colors using barycentric coordinates. We implement the renderer in a data-parallel fashion as a custom TensorFlow layer.
This implementation also allows for gradients to back propagate through the rendering step. The gradients computed at any pixel location can be distributed across the vertices of the relevant triangle according to the barycentric coordinates. While such an implementation cannot differentiate through the visibility check, it is appropriate in practice.

Network Architecture
Our network consists of siamese towers which take as input different frames of a video F i ∀i ∈ [0, N f ], where N f is the number of frames. Each such set of N f frames of one person identity is called a multi-frame image. The output of the siamese towers are the face parameters which are independent per-frame, i.e. expressions (δ i ), illumination (γ i ) and rigid pose (φ i ) ∀i ∈ [0, N f ]. We formulate multi-frame constraints for the identity component of the model. By design, the network only produces one output per multi-frame input for the identity shape (α) and reflectance (β). This is done through a multi-frame pooling of features from the siamese towers, followed by a small network. Thus, the network produces per-frame parameters, p i = (α, β, δ i , γ i , φ i ) In addition to the face parameters, we also learn the face models for expression (M exp ), identity shape (M id ) and albedo (M R ). These models are implemented as weights of the learnable network. More specifically, the position and reflectance of the face mesh, represented as V i (M id , M exp , α i , δ i ) and R(M R , β) are computed by applying the learned models to the predicted parameters as explained in Eqs. 1 and 3. Thus, for each multi-frame image in a mini-batch, an expression deformation per subframe, and consistent identity and reflectance deformations across all sub-frames in the multi-frame image, are computed. The computed reconstructions are then rendered using the differentiable renderer to produce synthetic images S i ∈ R 240X240X3 . We enforce orthogonality between the geometry and expression models to lead M id M exp = 0. This is done by dynamically constructing M id in a forward pass by projecting itself onto the orthogonal complement of M exp [51]. Please see Fig. 2 for more details.

Dataset
We use two datasets to train our approach: VoxCeleb [11] and EmotioNet [2]. VoxCeleb is a multi-frame dataset consisting of over 140k videos covering 6000 different celebrities crawled from YouTube. The multi-frame nature of Vox-Celeb allow us to train our Siamese network by feeding multiple frames (multi-frame images) for the same identity (see Fig. 2). We sample 4 images per identity from the same video clip to avoid unwanted variations due to aging, accessories and so on. This gives us a variety of head pose, expressions and illumination per identity. All our images are cropped around the face, and we discard images containing less than 200 pixels. We resize the crops to 240x240 pixels.
We also use EmotionNet [2]. It is a large-scale still image dataset of in-the-wild faces, covering a wide variety of expressions, automatically annotated with Action Units (AU) intensities. We use a subset of 7,000 images of neutral faces by selecting images with no active AU. We use these neutral faced images to enforce model disentanglement between the identity and expression geometry (Sec. 3.5.1).

Loss Functions
We formulate several loss functions to train our network. We perform self-supervised training, without using any 3D supervision. Let x be the learnable variables in the network, which includes all trainable weights in the neural network, as well as the learnable face models M id , M exp and M R . All the estimated parameters p i can be parametrized using We employ a segmentation loss which tries to move the vertices on the projected mesh contour (yellow) to a lower energy position in DT. In addition, each pixel in the boundary (red) attracts the nearest vertex on the mesh contour. these learnable variables. Our loss function consists of: The last two terms are regularizers and the first four are data terms. We used fixed λ • values to weigh the losses.
Landmark Consistency For each frame F i , we automatically annotate 66 sparse 2D keypoints [45] l ∈ R 66 . We compare these 2D landmarks with sparse vertices of the reconstruction which corresponds to these landmarks.
Here, v k (x) ∈ R 3 indicates the position of the kth landmark vertex, and p(v k (x) is its 2D projection (Sec. 3.2). While most face landmarks can be manually annotated once on the template mesh, the face contour is not fixed and thus has to be calculated dynamically (see supplemental for details).

Segmentation Consistency
The estimated keypoints are ambiguous in the inner lip regions, due to rolling lip contours. In addition, the accuracy of sparse keypoint prediction is inadequate to learn expressive expression models. We use a dense contour loss for the lip region, guided by automatic segmentation mask prediction for the lips [28]. The lip segmentation contours are converted into distance transform images D b a , where a ∈ {upper, lower} and b ∈ {outer, inner} corresponding to the outer and inner contours of both lips. We also compute the contours of both lips projected by the predicted reconstruction C b a (x), where each element of C b a (x) stores a 2D pixel location. For a given distance transform image and the corresponding contour of the predicted mesh, the loss function minimizes the distance between the mesh contours and segmentation contours, see Fig. 3. Figure 4. At test time, our approach produces plausible upper (gray) and lower (white) lip segmentation even when the images are of bad quality, contain extreme poses or occlusions. For such cases [28] struggles to produce acceptable segmentation (column 4).
Here, the first term minimizes the distance from every pixel in the mesh contour to the image contour. The second term is a symmetric term minimizing the distance between every pixel in the image contour to the closest mesh contour. closest(C b a (x), (x, y)) is a function which gives the position of the closest pixel in C b a to (x, y). We use our differentiable renderer to calculate C inner l (x) for the rolling inner contours. The outer contour C outer l (x) is computed as the projection of some manually annotated vertices on the template mesh. In practice, we ignore this loss term at pixels where the distance between the image and mesh contours is greater than a threshold. This helps in training with noisy segmentation labels.
Photometric Consistency We evaluate the dense photometric consistency between the reconstructions and the input. For each pixel, we minimize the color difference between the input image F and the rendered face S.
M i is a mask image with pixel value 1 where the reconstructed mesh projects to. Perceptual Loss We additionally employ a dense perceptual loss to help our networks learn higher quality models, including high-frequency reflectance details. In particular, we use a VGG network pretrained on ImageNet [24] to get the intermediate features for both input frames and the output synthetic frames. We then minimize the cosine distance between these features.
where f l (·) denotes the output of the lth intermediate layer for input x and < ·, · > denotes the inner product.

Geometry Smoothness
To ensure smoothness of the final geometry, we use a smoothness loss at the graph level. Let G i ∈ R Ng×3 with N g = 521 nodes denote the geometry reconstruction for frame F i at the graph level. We employ an 2 loss to constrain the difference between the deformation of adjacent nodes.
where N (g) is the neighbourhood of node g, g(x) and n(x) ∈ R 3 .

Model Disentanglement
Our goal is to learn deformation models for facial geometry, expression and reflectance. Disentangling these deformations in the absence of an initial 3DMM is challenging. We use a combination of network design choices and loss functions to enable simultaneous learning of these models. Siamese Networks: Our siamese network design ensures that the identity components of our reconstructions are consistent across all frames of the batch. Such a network architecture allows us to disentangle illumination from reflectance in addition to helping with disentanglement of expressions from identity geometry. Disentanglement Loss One example of a failure mode would be when M id collapses to a zero matrix. Here, all geometric deformations including identity would be learned by the expression model without any penalty from any loss function. To prevent such failure modes, we design a loss function to disentangle these components. As mentioned in 3.4, a subset of our dataset includes images which correspond to neutral expression. For these images, we employ a loss function which minimizes the geometry deformations due to expressions.
Since we do not have videos for these images, we simply duplicate the same image as input to the siamese towers. Finally, our training strategy further helps with disentanglement. Please refer to the supplemental for details.

Results
Training Details We implement our approach in Tensorflow and train it over three stages: 1) pose pretraining 2) identity pretraining and 3) combined training. We empirically found this curriculum learning to help with stable training and disentanglement of the identity and expression models. Pose Pretraining: We first train only for the rigid head pose. All other parameters are kept fixed to their initial value. Identity pretraining: Next, we train for the identity model. This step is only trained on the EmotionNet data with neutral expressions. We enforce the expression Figure 7. Our approach produces better geometry, including detailed mouth shapes compared to Tewari et al. [53] and FML [51]. Our albedo is also more detailed and better disentangled from the illumination.  [56,54] do not disentangle identity geometry from expressions. Our technique, however, estimates and disentagles all facial components. It also produces produces more accurate mouth shapes. parameters to be zero, enforcing all deformations to be induced by the identity model. Combined Training: Last, we train for the complete model with the loss functions as explained in (5). Similar to the first stage, we continue to impose the landmark loss term on the mean mesh throughout model learning. This helps in avoiding the geometric models learning the head pose. Our training data now consists of mini-batches sampled from EmotionNet and VoxCeleb with 1:3 ratio. We train for 650k iterations with a batch size of 1. This results in a training time of 117 hours on a TitanV. We use 80 basis vectors for identity geometry and albedo, and 64 for expression.  Fig. 6 shows all components of our reconstruction for several images. Our approach can handle different ethnicities, genders and scene conditions, and produces high-quality reconstruction, both in geometry and reflectance. Fig. 4 shows that our method can produce better lip segmentation than the approach used for generating the training data [28] in some cases. This is due to our segmentation loss function, L seg , where we selectively ignore unreliable segmentation estimates. Hence our final model is learned from only accurate segmentations in the trainingset. Fig. 10 shows that including perceptual loss in our training (Sec. 3.5) clearly improves the photorealism of the albedo and the final overlay. Comparisons: Fig. 8 -10 compare our approach to several state-of-the-art face reconstruction techniques. Tran et al. [56,54] learns a combined geometry for identity and expressions, while we learn a separate model for each (Fig. 8). MoFA [53] and GANFIT [19] geometry are limited by a pretrained 3DMM model and hence lead to less detailed shapes than ours (Fig. 9). While GANFIT produces detailed textures, it can contain artifacts. Like other 3DMM based approaches, RingNet [44], which estimates the parameters of a pretrained face model [33], struggles with out-of-space variations especially in the mouth region (Fig. 10). Tewari et al. [52] refines a pretrained 3DMM model on an image dataset. We can disentangle the reflectance and illumination better (Fig. 7). FML [51] is constrained by a pretrained expression model and thus produces less convincing shape reconstructions than us (Fig. 7). In addition, our reflectance estimates are more detailed compared to [51]. Thus, even though we start from just a template mesh without any deformation priors, we can produce high-quality results, better than the state-of-the-art.

Quantitative Evaluation
Geometric Error: To evaluate the geometric accuracy of our 3D reconstructions, we compute the per-vertex root mean square error between the ground-truth geometry and the geometry estimated using different techniques. We evaluate this metric on the BU3DFE dataset [60] where the ground-truth geometry is obtained using 3D scans. Tab. 1 reports the results over 324 images. Our approach outperforms the approaches of MoFA [53], Tewari et al. [52] and FML [51]. Note that none of the approaches in Tab. 1 learn a complete face model from images and videos. Segmentation Error: To specifically evaluate the quality of lip reconstructions, we use Intersection over Union (IoU) between our reconstructions and the input images over the lip regions. Since our approach learns an expression model from in-the-wild data, it can generalize better to different lip Figure 9. MoFA [53] and GANFIT [19] produce less accurate mouth shape than our technique. GANFIT [19] can produce artifacts in the albedo and final overlay, especially around the eyes. Figure 10. Left: Our technique better captures mouth shape and eye geometry than RingNet [44]. It also produces a photorelistic overlay. Right: Albedo and overlay is noticeably improved with perceptual loss Lper (see 3rd and 5th column).
shapes and significantly outperform FML [51] (see Tab. 2). Furthermore, Tab. 2 shows that removing the segmentation consistency term (Eq. 3.5) leads to lower quality results. Disentanglement Error: One of our main objectives is to obtain a disentangled representation for faces. To evaluate the disentanglement between the reconstructed expression and identity geometry, we design a metric which measures the average of expression deformation for images with neutral faces. We tested our approach on 1864 neutral faces mined using the same strategy described in Sec. 3.4. Tab. 3 reports the average 2 value of the expression deformations, as estimated using different approaches. The lower the expression strength, the better the disengagement. Our approach achieves significantly better expression and identity disentanglement over FML [51] and MoFA [53]. Verification Metric: To further evaluate disentanglement, we use the LFW dataset [21], which includes face image pairs of same as well as different identities. We render the identity component of the reconstructions with the predicted pose and lighting parameters. Face embeddings are computed as the average pooled version of conv5 3 output of VGG-Face [37]. We first compute the histogram distribution of cosine similarities between renderings of image pairs with the same identity in embedding space. Similarly, distribution of cosine similarities between rendering pairs of different identities is computed. Then, we compute the verification metric as the Earth Movers Distance (EMD) between these two distributions. Our method achieves an EMD of 0.15, compared to 0.09 of FML. A larger distance implies that the network can better represent the difference between different identities due to better disentanglement.

Conclusion
We presented the first approach for learning a full face model, including learned identity, reflectance and expres-  Table 3. Our identity disentanglement term results in lesser leakage of identity geometry into expression component in neutral faces. It performs better than FML [51] and MoFA [53] sion models from in-the-wild images and videos. Our method also learns to reconstruct faces on the basis of the learned model from monocular images. We introduced new training losses to enforce disentanglement between identity geometry and expressions, and to better capture detailed mouth shapes. Our approach outperforms existing methods, both in terms of the quality of image-based reconstruction, as well as disentanglement between the different model components. We hope that our work will inspire further research on building 3D models from 2D data.