{'fraction_non_alphanumeric': 0.04049702485124256, 'fraction_numerical': 0.016940847042352117, 'mean_word_length': 4.466041706523819, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
ACE-Net: Fine-Level Face Alignment through Anchors and Contours Estimation

We propose a novel facial Anchors and Contours Estimation framework, ACE-Net, for fine-level face alignment tasks. ACE-Net predicts facial anchors and contours that are richer than traditional facial landmarks while overcoming ambiguities and inconsistencies in their definitions. We introduce a weakly supervised loss enabling ACE-Net to learn from existing facial landmarks datasets without the need for reannotation. Instead, synthetic data, from which GT contours can be easily obtained, is used during training to bridge the density gap between landmarks and true facial contours. We evaluate the face alignment accuracy of ACE-Net with respect to the HELEN dataset which has 194 annotated facial landmarks, while it is trained with only 68 or 36 landmarks from the 300-W dataset. We show that ACE-Net generated contours are better than contours interpolated straight from the 68 GT landmarks and ACE-Net also outperforms models trained only with full supervision from GT landmarks-based contours.


Introduction
Face alignment is the basis for various types of face analysis such as face mesh reconstruction [2, 3, 42,33,53], facial behavior modeling for deepfake detection [1] and visual speech recognition [21,30,20], where fine-level alignment/tracking is crucial. Most of the existing approaches use facial landmarks to perform face alignment. While the accuracy of facial landmarks detection has greatly improved over the past few years, there remain three major issues with facial landmarks representation: First, only a small subset of landmarks are well defined and can be localized accurately, such as eye corners. We refer to these as anchors.
The rest of the landmarks are just discrete points sampled along various facial contours. We call those contour landmarks as their positions are ambiguous along the contours (Fig. 1a). Second, typical facial landmarks definitions are not dense enough to capture fine-level details of facial contours (Fig. 1c,1d). Third, facial landmarks definitions are inconsistent across existing datasets [23,32,46,25]. In order to address these issues, we propose a new representation that retains all the anchor landmarks but replaces all the contour landmarks with actual contours. We call this the Anchors and Contours (AC) representation (Fig. 1b). These face contours provide better constraints for alignment, for example when fitting meshes (Fig. 2). In this paper, we define a new framework which we call facial Anchors and Contours Estimation Network (ACE-Net) to extract the AC representations from images. ACE-Net consist of two modules, (i) the AC prediction module which predicts AC heatmaps from input images and (ii) the AC extraction module which converts the extracted AC heatmaps to the final AC representations with sub-pixel accuracy.
A new representation typically necessitates reannotation of the datasets with the new representation to generate the necessary training data for the new model. However, facial contours are extremely time consuming to annotate and  Mesh from [53] with AC overlaid show that there is room for improvement in the mesh fitting that AC rep can provide (see selected zoom-ins). For motivation only (not discussed in this paper.) presents a formidable bottleneck. Instead, we devise two strategies that allow us to bypass this bottleneck. First, we introduce a novel weakly supervised loss that enforces the predicted AC to pass through existing annotated facial landmarks. This allows us to use existing facial landmarks datasets as is. An added benefit is that this makes us independent of the specific landmarks definitions and allows us to use a multitude of datasets. We show later that a 36landmarks-based weak supervision (vs the typical 68) still works quite well. Second, since the density of contour landmarks are insufficient to fully characterize the contours, we augment the training with synthetic data that provides contours for full supervision. We find that in spite of the domain gap, synthetic data helps improve the accuracy of the contour localizations in the spaces between landmarks. Our ACE-Net is thus trained with weak supervision from the landmarks and full supervision from the synthetic contours.
We evaluate ACE-Net on the densely annotated HELEN dataset [25]. We compare ACE-Net results against linecontours (generated by connecting landmarks with line segments) and interpolated-contours (generated by spline fitting through the landmarks) derived straight from the GT landmarks. Since GT landmarks are arguably better than any SOTA landmark detectors, these contours can be considered the best AC results possible directly from landmarks. Furthermore, instead of using full supervision from the synthetic data, we also experiment with using these GT landmarks-derived contours to provide the full supervision (in which case, there is no need for the weak loss). Both our qualitative and quantitative results show that at a fine-level, ACE-Net outperforms landmarks based models, especially in regions lacking training annotations.
In summary, our main contributions include: • ACE-Net, the first facial alignment framework that predicts facial anchors and contours (with sub-pixel accuracy) instead of just facial landmarks.
• A novel "contourness" loss to effectively use existing facial landmarks as weak supervision.
• The incorporation of synthetic data to complement the training by bridging the gap between landmarks and contours (despite the domain gap) thus bypassing the need for re-annotation. We show that this is better than using landmark-interpolated contours.
• An evaluation metric for the AC representations with respect to densely annotated facial landmarks datasets.
Heatmap regression Heatmap regression methods [45,12,38,11,39] predict a heatmap for each facial landmark. stacked hourglass (HG) network [31] is the most widely used architecture among recent works [48,5,46]. Apart from the common L 1 or L 2 loss functions, [44] adapts wing loss from coordinate regression, and add different foreground and background weights to address the class imbalance in heatmaps. [18] introduces focal wing loss to adjust data sample weights. Currently, heatmap regression methods achieve higher accuracy compared to coordinate regression methods in general.
Facial boundary heatmaps To address the issue of illdefined contour landmarks, [46] introduces facial boundary heatmaps which is the heatmaps representation of contours interpolated from facial landmarks. Facial boundary heatmaps are also used in [44,18] to improve landmarks prediction accuracy. In existing works, facial boundary heatmaps have been used only as an internal intermediate representation to mitigate prediction errors caused by ill-defined landmarks. AC differs from facial boundary heatmaps in two aspects: (1) AC aims at modeling true facial contours instead of the interpolated contours from landmarks; and (2) AC can be explicitly extracted as final outputs and can be quantitative evaluated.
Face parsing Instead of predicting facial landmarks, face parsing methods [28,52,51,27,26] aim at segmenting faces into semantic face parts. Compared with facial landmarks models, face parsing models provide extra information such as skin masks and face part areas, but as segmentation models, they focus more on regions rather than landmarks or contours.   Fig. 3 shows an overview of our proposed ACE-Net framework. At test time (Fig. 3a,b), ACE-Net takes a face ROI image as the input and first predicts a separate heatmap for each facial anchor and contour, and then performs AC extraction to locate each anchor and contour with sub-pixel accuracy. At training time (Fig. 3c), ACE-Net is trained with both synthetic and real images. The synthetic images are rendered from 3D face meshes and thus the Ground Truth (GT) ACs are automatically available, while the real images only have annotated GT landmarks. We apply different losses to the synthetic and real data based on the different information available from each. For real data, we introduce a weakly supervised loss function which encourages the predicted contours to pass through available annotated GT landmarks. For the synthetic data, we use the fully supervised loss since complete GT ACs are available.

Methods
In this section we first define our AC representation followed by the definitions of our fully supervised and weakly supervised loss functions used to train the AC prediction module. Finally, we describe our AC extraction module that enables explicit AC localization.

The Anchor-Contour (AC) Representation
Anchor An anchor, a, is point feature tied to well defined local features on the face, e.g., corners of the eyes or lips. It is represented as a 2D point, (a x , a y ), on the image.
From GT anchor landmarks, anchor heatmaps are generated for training our network. The heatmap, H a , corresponding to an anchor a is defined as where p is a pixel on the heatmap and σ controls the width of the peak. We choose this definition of anchor heatmaps instead of the commonly used Gaussian heatmaps to keep it consistent with our contour heatmaps definition (below).
Contour A facial contour, c, is a 2-d curve that maps to well-defined facial contours, e.g., eyelids, lips. The occluding contour on the face (outer boundary) is also represented as a facial contour despite the fact that these are related to the viewing direction and not tied to specific features on the face. The contour, c, is represented as a collection of subpixel line segments, {s i }.
Similar to anchors, heatmaps are generated from contours for training our network. A contour heatmap H c is define as where p is any pixel on the heatmap, and dist(p, c) is a function measuring the minimum distance from point p to the contour c, specifically, the minimum distance between p and any of the line segments s i in c.

Fully Supervised Loss
In the fully supervised setting, for each image I, its anchors A = {a 1 , a 2 , . . . , a N A } and contours C = {c 1 , c 2 , . . . , c N C } are all available. We generate the ground truth AC heatmaps H = {H 1 a , . . . , H N A a , H 1 c , . . . , H N C c } following Eq. 1 and Eq. 2, and define the fully supervised loss L for predicted heatmapsĤ as a weighted root mean square loss: where |H| is the total number of pixels in heatmaps H, · is element-wise multiplication, and W is a weighting empha-sizing positive and hard negative examples: This weighting is necessary because most of the pixels in heatmaps are background pixels. Without W h the learned model is easily trapped in the trivial local minimumĤ = 0. In settings where line-contours,C, are generated by connecting GT contour landmarks with straight line segments, sinceC = C, applying this fully supervised loss L withC being the ground truth will result in significant loss at the fine-grained level.

Weakly Supervised Loss
In the weakly supervised setting, for each image, I, only its landmarks, L = {l 1 , l 2 , . . . , l N L }, are available. For predicted anchor heatmaps,Ĥ a , we just apply the fully supervised loss, L, in Eq. 3, because anchors are a subset of landmarks and we have access to all ground truth anchors A. However, the weakly supervised contour loss,L, needs a novel treatment.
Before we define our weakly supervised contour loss,L, we first introduce a "contourness score", C, that evaluates whether there is a contour passing through a given pixel p on a heatmap H. Assuming that a contour is locally a straight line with orientation θ and width σ, we construct its heatmap template T σ,θ based on Eq. 2: Examples of T σ,θ is shown in Fig. 4. If this contour passes through pixel p, then heatmap H cropped around p should match the template T σ,θ . Thus, based on template matching error, we define contourness score, C, as Here the negative sign at the beginning turns contourness C from an error to a score. H + = max(0, H) is a nonnegative clipping, and G σ (x, y) = e −(x 2 +y 2 )/σ 2 is the Gaussian weight. We will omit σ in the subscripts since it is a constant hyper-parameter.
The optimization in Eq. 6 has a closed-form solution through the use of steerable filters [15]. Therefore, the contourness score, C(H), for all pixels on heatmap H can be efficiently computed with convolutions and integrated into loss functions. Due to space limitation, please refer to supplementary materials for detailed description of this process. Apart from the contourness score map C(H), contour orientation map, O(H), can also be efficiently computed as the optimal θ in the closed-form solution to Eq. 6. Figure 4: Examples of contour template T σ,θ with various θ. Now, with the contourness score C defined, we can begin to define the weakly supervised loss. Recall that in this setting we only have the GT landmarks to work with. Letc be the a line-contour created from corresponding contour landmarks, L c . Note thatc can either be generated by simply connecting the adjacent GT landmarks with straight lines, or it can be generated by interpolating the landmarks with splines. LetĤ be a predicted contour heatmap representing a predicted contour,ĉ.
We design our weakly supervised loss to enforce the following three rules: (1) The predicted contourĉ must pass through all contour landmarks L c : where |L c | is the total number of contour landmarks, and f (·) is a mapping function that converts contourness score to a loss in [0, 1] range.
(2) The predicted contourĉ must be close to line-contour c. In other words, for each pixel p onc, there must exist a pixel q onĉ such that q − p is the line-contour normal at p, and p − q ≤ D for some constant threshold D: where |c| is the total number of pixels onc, Nc is the normal map of line-contourc and Nc(p) is the line-contour normal at pixel p.
(3) Pixels far away from line-contourc should have zero heat value: where M (c) is a binary mask selecting pixels far fromc: Eq. 9 follows L in Eq. 3 with ground truth being H = 0. It only adds an extra mask M (c) to select pixels far fromc. Finally, we define our weakly supervised contour lossL as the sum of the three losses above: (11) where λ landmark and λ line are constant weights.

AC Extraction
The AC Extraction module converts each predicted anchor heatmap, H a , to a 2D anchor position a = (a x , a y ), and each contour heatmap, H c , into to a contour, c = {s i }.
Extracting Anchors We adopt the local center-of-mass method: given an anchor heatmap H a , we find the pixel p * with highest heat value, and compute anchor position a as Extracting Contours Given a contour heatmap, H c , we compute its contourness map, C(H c ), as well as its contour orientation map, O(H c ), using the closed-form solution to Eq. 6. We then obtain the contour normal map N (H c ) = O(H c ) + π/2, and perform non-maximum suppression (NMS) on C(H c ) along directions specified by N (H c ), retaining just the maximal pixels which are then thresholded to obtain the binary contour mask, B c . During NMS, we also localize the points in B c to subpixel accuracy by fitting a parabola along the normal direction specified by N (H c ). After NMS and thresholding, each contour has a width of 1 pixel. Connected components analysis is performed with hysteresis to extract the contour trace, c, in the same way as Canny edge detection. (Fig. 5).

Synthetic Data Generation
Our synthetic training data is generated using a 3D morphable face model (3DMM) [4] created by [19]. We randomly select facial shape, expression and texture coefficients for the 3DMM and generate 50,000 3D face meshes with random accessories (hairs, glasses and headbands). Each mesh is rendered to a image with random lighting and head pose up to 60 • yaw, 30 • pitch and 30 • roll. An image from non-human categories of Caltech-256 dataset [16] is To obtain AC annotations for the synthetic data, we manually annotate the indices of all mesh vertices in the 3DMM corresponding to each facial anchor and contour, and then automatically generate the AC annotation for each synthetic image using the generated morph coefficients and head poses. Example synthetic images and their annotations are shown in Fig. 6.

Experiments
Following the assumption that accurate contour annotations are hard to obtain, we train our models with the most widely used facial landmarks dataset, 300-W, and evaluate the fine-level face alignment accuracy on the HELEN dataset where extremely dense landmarks annotations are available to approximate true facial contours. Fig. 7 illustrates the landmarks annotation density of the two datasets. [32] is by far the most widely used face alignment dataset where each image is annotated with 68 facial landmarks. Its training set consists of 3148 images.

300-W
HELEN [25] testset contains 330 test images, each annotated with 194 facial landmarks. We choose HELEN as our evaluation data because it is the only publicly available dataset with dense enough landmarks annotation to approx- imate true facial contours. We do not include HELEN training set for evaluation, because it overlaps with our training dataset 300-W.

Evaluation Metrics
Normalized Mean Error (NME AC ) Following the standard NME for facial landmarks, we define a similar metric, NME AC , for anchors and contours. Note that our ground truths are just landmarks, albeit a denser set than what was used to train ACE-Net. We first divide these ground truth landmarks, L, into anchor landmarks, L A = {a 1 , a 2 , . . . }, and contour landmarks, is the set of landmarks on ground truth contour c i . Then, given the predicted anchors, A = {â 1 ,â 2 , . . . }, and contours,Ĉ = {ĉ 1 ,ĉ 2 , . . . }, we define the prediction error of each GT landmark, l, as: where dist(·, ·) is the same point-to-contour distance as in Eq. 2. Finally, the metric NME AC is defined as the normalized mean error of all landmarks: where d is the normalization factor. For all experiments in this paper, we use inter-ocular distance (distance between the two outer eye corners) as the normalization factor. Area Under the Curve (AUC) measures the area under the cumulative error distribution (CED) curve with the error being NME AC . We cut off the error at 6%.

Implementation Details
We obtain face bounding boxes by padding the tight boxes from ground truth landmarks by 25%, 25%, 33%, 17% on left, right, top and bottom sides respectively, and then convert them into square boxes by keeping the longest edge length. Our training data augmentation includes random translation up to 25%, rotation up to 30 • , horizontal flipping with 50% chance, gamma correction with γ ∈ (2 −0.75 , 2 0.75 ) and Gaussian noise. We set anchor and contour width σ = 2, and contourness pooling radius D = 6 for our weakly supervised losses in Eq. 8 and Eq. 10. We choose the mapping function f (C) = 1 − 2 (C−4.92)/1.5 for Eq. 7 and Eq. 8 since the maximum contourness C is approximately 4.92 when σ = 2. We empirically set weights λ landmark = λ line = 0.1.
The CNN used in our experiments has an input size of 256 × 256 (grayscale) and an output size of 128 × 128 × (N A + N C ) where N A and N C are the number of anchors and contours, respectively. Details of our network architecture are described in our supplementary materials due to space limitations. ACE-Net does not rely on specific network architecture, so it can be replaced by any other commonly used CNNs. We train our models from scratch using Adam optimizer [22] with an initial learning rate of 2.5 × 10 −4 , fuzz factor at 10 −8 and learning rate decay at 10 −6 . All our models are trained with three 8GB GeForce GTX 1080 GPUs.
At test time, we use σ = 3 for AC extraction. We choose a larger σ than in training because the predicted heatmaps tend to be more "spread out" than the ground truth heatmaps.

Evaluation on the HELEN Dataset
In this paper, all models are trained on 300-W training set with 68 landmarks per image, and the resulting AC representations are evaluated on HELEN test set with 194 landmarks per image. During evaluation, eyebrow boundary landmarks and nose side landmarks are excluded since they are not annotated in 300-W. We compare the performance of the proposed ACE-Net with the following models: Lmk-line first predicts 68 facial landmarks through conventional heatmap regression, and then converts the predicted landmarks into anchors and line-contours. Lmk-line has the same backbone architecture as ACE-Net, and its training loss is L in Eq. 3. Note that this is just for baseline reference and is not at SOTA-level.
GT68-line converts the ground truth 68 facial landmarks (from the 300-W testset which are also available in the HELEN testset) into anchors and line-contours. We use the ground truth landmarks instead of any particular SOTA landmark extraction results because the GT landmarks represent the upper bound for landmarks detection accuracy and consequently these line-contours must represent an upper bound for landmarks-based line-contour models.
GT68-interp is similar to GT68-line but uses quadratic spline interpolation instead of straight lines to generate contours from landmarks. We also tried cubic interpolation but it is more sensitive to landmark noise than quadratic interpolation and so does not perform as well.
Line predicts AC with the same architecture as ACE-Net, but is trained only with real images and treats linecontours obtained from GT landmarks as GT contours.
Line * extends Line by including synthetic data in its training set.
To sum up, Lmk-line, GT68-line and GT68-interp are landmarks-based models with no special training to extract contours, while Line, Line * and ACE-Net are AC based models, specifically trained to extract contours. The differences between these AC based models are only in the training data and losses:  Here "Real" stands for the 300-W training set and "Syn" stands for the generated synthetic data. The results on HELEN testset are shown in Table 1 and Fig. 8. ACE-Net achieves significantly better accuracy than its landmarks detection based counterpart Lmk-line in all face parts (NME AC : 1.32 vs 2.20, AUC: 86.77% vs 77.95%). Notably, ACE-Net also outperforms ground truth landmarks based models GT68-line and GT68-interp (NME AC : 1.32 vs 1.48 and 1.39, AUC: 86.77% vs 85.20% and 86.17%), showing the effectiveness of AC based modeling at fine-level face alignment.
Among the AC based models, ACE-Net achieves higher accuracy than Line (NME AC : 1.32 vs 1.59, AUC: 86.77% vs 84.27%), especially in the small error region. For example, ACE-Net has a success rate of 56.4% at NME AC ≤ 1.25% whereas Line only has 36.7%. This is because ACE-Net's weakly supervised loss L prevents it from overfitting to the inaccurately hallucinated portions of line-contours, and instead learns from accurate synthetic contours. Without the weakly supervised loss L, synthetic data alone does not bring any noticeable improvement, since there is a large    domain gap between the synthetic and the real data, and the model tends to overfit to the inaccurate real line-contours. Therefore, model Line * has almost identical performance to Line in spite of the addition of synthetic data (NME AC : 1.59 vs 1.59, AUC: 84.14% vs 84.27%).

Ablation Study
To verify the robustness of ACE-Net against sparsity in training annotations, we reduce the number of landmarks used from 68 to only 36 (Fig. 7). We refer to our model trained with this reduced set of landmarks as ACE-Net sparse . All its hyper-parameters, loss functions and the use of synthetic training data remains the same as the original ACE-Net. For fair comparison, we create models GT36-line and GT36-interp similarly to GT68-line and GT68-interp with the set of 36 landmarks. Table 2 and Fig. 9 show their performance comparison. Although ACE-Net sparse is trained with only 36 landmarks, its performance is very close to ACE-Net which is trained with 68 landmarks (NME AC increase: 0.06, AUC reduction: 0.58%). It clearly outperforms GT36-line and GT36interp, both of which have a much larger performance drop compared with their GT68 version (NME AC increase: 0.94 and 0.23, AUC reduction: 9.43% and 2.37%).
We also observe that ACE-Net sparse outperforms Lmkline, GT68-line, GT68-interp, Line and Line * , despite them all trained with 68 landmarks. In particular, ACE-  Net sparse outperforms Lmk-line by a large margin (NME AC : 1.38 vs 2.20, AUC: 86.19% vs 77.95%). This comparison shows the ability of ACE-Net to make up for the reduced annotation density in the real training data. Contours predicted by Line appear more accurate, but they are really a smoothed version of line-contours and do not always follow local image cues, exhibiting similar error patterns as Lmk-line. In contrast, ACE-Net predicts facial contours whose shapes are more consistent with true facial contours, especially in places lacking training annotations such as inner lips.

Qualitative Results
The last two columns in Fig. 10 show the most common failure case of AC based models. Our model is trained under the assumption that contours appear as "ridges" in their corresponding heatmaps. When there is significant ambiguity, for example due to severe occlusion, predicted contour heatmaps are less "peaky" and get dropped during NMS+thresholding. However, in such cases fine-level alignment is usually out of the question anyway due to the lack of image cues for precise localization.

Conclusion
In this paper, we present ACE-Net, the first face alignment framework that learns to predict fine-level facial anchors and contours. Compared to contours generated blindly from landmarks or models trained with linecontours as ground truth, ACE-Net captures the fine-level details of facial contours with higher accuracy, especially in regions lacking annotations. Our approach allows the effective use of existing facial landmarks annotations as weak supervision through a novel "contourness" loss. ACE-Net also demonstrates the potential of using synthetic data as a complement to sparsely labelled real data. Specifically, treating the sparsely labelled real data as weak supervision, synthetic data are able to help bridge the density gap and improve fine-level prediction accuracy. We expect ACE-Net to make a significant impact on tasks dependent on accurate face alignment such as mesh fitting.

Appendix A. Efficient Computation of Contourness C
In this section we derive the close-form solution of the optimization of C(H) in Eq. 6 in the main paper: where H + = max(0, H), and We will omit σ from subscripts since it is a constant hyperparameter.
Since non-negative clipping max(0, ·) is already applied to the signal H in Eq. 6, we drop the same clipping for the template T in Eq. 5 which does not change the value of optimal C(H). Thus, the definition of T becomes Let ∆p = (i, j). By expanding · 2 in Eq. 6 we get Here the last term ∆p G(∆p)·T θ (∆p) 2 is a constant number given σ. Therefore, we drop it as it does not affect the optimization.
To efficiently compute contourness C(H) for every pixel p in H, we rewrite Eq. 17 with convolution: Note that T θ · G is a steerable filter (similar to G θ 2 in Table  III from [15]). According to [15], H + ⊗ (T θ · G) can be computed as where G 2a , G 2a and G 2a are convolutional filters: (Our filters and coefficients definition here are slightly different from Table III in [15], because θ in [15] is defined as the normal instead of the contour orientation.) Now, let R a , R b , R c be the convolution responses: then Eq. 19 becomes Thus we have Plugging Eq. 23 back to Eq. 18, we get the closed-form solution of C(H): Eq. 24 shows that the contourness map C(H) can be very efficiently computed simply with convolutions, and therefore can be integrated into loss functions. From Eq. 22, we also derive the closed-form solution of the contour orientation map O(H):    Table 3. Our example architecture is picked due to its simplicity (inference takes around 17ms per image on a 8GB GeForce GTX 1080 GPU). Since our model has intermediate outputs, we apply losses to intermediate outputs as well for better intermediate supervision. Because the intermediate outputs have very low resolution equal to 1/8 of the input, we just treat line-contours as ground truth and apply fully supervised loss. This intermediate loss is not required, especially when a network architecture without intermediate outputs (such as Stacked Hourglass) is chosen instead. ACE-Net does not rely on any specific network architecture.