{'fraction_non_alphanumeric': 0.028174123337363965, 'fraction_numerical': 0.00713422007255139, 'mean_word_length': 4.807818352059925, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Hierarchical Probabilistic Model for Facial Feature Detection

Facial feature detection from facial images has attracted great attention in the field of computer vision. It is a nontrivial task since the appearance and shape of the face tend to change under different conditions. In this paper, we propose a hierarchical probabilistic model that could infer the true locations of facial features given the image measurements even if the face is with significant facial expression and pose. The hierarchical model implicitly captures the lower level shape variations of facial components using the mixture model. Furthermore, in the higher level, it also learns the joint relationship among facial components, the facial expression, and the pose information through automatic structure learning and parameter estimation of the probabilistic model. Experimental results on benchmark databases demonstrate the effectiveness of the proposed hierarchical probabilistic model.


Introduction
Accurately analyzing the face is critical, since the face reflects the psychological and physical states of humans. For example, by analyzing the face, we can identify human facial expressions [8], detect the fatigue state of human, estimate the human head pose [19], and so on. As the bottom level representation, the locations of facial feature points such as eyebrow tips, eye corners, mouth corners provide a promising way to describe the face shape and capture the rigid and nonrigid deformation of the face.
Detection of these facial feature points is challenging, since facial appearance and shape tend to change significantly under different facial expressions and poses. For example, in Figure 1, the local appearance and shape of the mouth varies dramatically. In (a), the lip corners pull up. The mouth opens widely in (b), while closes tightly in (c). Therefore, a good facial feature detection method should be able to model the shape variations under different situations. Another important observation is that facial components do not change independently and there exists interac- tion among facial components. For example, in (b), eye lids go up and the mouth opens widely simultaneously. As a result, it is also important to model the relationship among facial components. Furthermore, since facial expression and pose are two major causes of the significant change of the face shape, it is necessary to take these information into account when modeling the face shape variations. Following these three observations, in this paper, we propose a hierarchical probabilistic model to capture the the variations of face shapes under significant facial expression and pose changes, and infer the true facial feature locations given the measurements. Specifically, we first model the local shape variations of each facial component using a mixture model. In addition, we model the joint interactions of facial components in the higher level. Unlike most of the previous methods, we explicitly add the facial expression and pose information to help construct and parameterize the model. During detection, the true facial feature locations are inferred through the bottom-up lower level shape constraints of facial components and the top-down constraint from the relationship among facial components.
The remainder of this paper is organized as follows: Section 2 reviews the related work. In section 3, we will describe the model, its learning and inference. Section 4 will show the experimental results on several benchmark databases. Section 5 will conclude the paper.

Related Work
As a promising way to describe the shape of the nonrigid face, facial feature localization has gained much more attention in computer vision community over the past years. Methods for facial feature detection can be classified into three categories: holistic methods, local methods and component based methods.
Holistic methods: The holistic methods build models to capture the appearance and shape of the whole facial image. Typical methods are the classic Active Shape Model (ASM) [2] and Active Appearance Model (AAM) [1]. ASM is a statistical model that uses the mean shape and eigenvectors generated from PCA to represent the deformations of objects' shape. AAM uses both the texture and shape information of the whole facial image to fit a linear generative model. However, it is sensitive to the lighting conditions and performs poorly on unseen identities [11].
Local methods: The local methods use local image features. They build a likelihood map for each landmark based on the response of a regression function or a classifier. Usually, these likelihood maps are combined with a global shape model as a constraint. In [3], Cristinacce and Cootes proposed the Constrained Local Model (CLM). It generates the likelihood map for each facial feature using the matching score of the local testing image to the templates. Then, the likelihood maps are combined with a prior distribution of the shape that is similar to ASM. More recently, in [25], Valstar et al. proposed the BoRMaN model that combines Boosted Regression with Markov Networks. It generates the likelihood map for each point based on the response of the support vector regressor, and then constraints the point locations by the shape distribution embedded in a pre-trained Markov Random Fields model. In [18], Martinez et al. proposed a Local Evidence Aggregation for Regression (LEAR) based facial feature detection method. It utilizes similar shape constraint as [25] with improved searching procedure that considers the aggregation of local evidence. In [22], instead of using the parametric model, the work of Saragih et al. represents the face shape prior model in a non-parametric manner with optimization strategy to fit facial images.
Component based methods: The component based methods focus on molding the shape variations of each facial component. The relationships among different facial components are further embedded in the model as a global constraint. In [12], a bi-stage model was proposed to characterize the face shape locally and globally. The local shape variation is modeled as a single Gaussian, and the nonlinear relationship among components is represented as a pretrained Gaussian process latent variable model. Similarly, [16] uses one PCA for local component shape and a global PCA to model locations of components. Tong et al. [24] proposed a model to capture the different states of facial components like mouth open and mouth closed. But, during testing, they need to dynamically and explicitly estimate the state of local components and switch between different states. In [7], for each facial component, Ding and Martinez emphasized distinguishing the image patch centered at the true facial component location from the contextual patches with the use of the subclass division method.
Among all the facial feature localization methods, only a few works explicitly model the face shape variations under expression and pose changes. In [4], instead of constructing one single set of regression forests, Dantone  In this paper, our major contribution is a hierarchical probabilistic model that could infer the true facial feature locations given image measurements. However, unlike some other component-based methods, we implicitly model the shape of facial components by learning the variations under different hidden states which is usually assigned or ignored by previous work. Furthermore, we explicitly learn the relationships among facial components, facial expression and poses, and use these relationships to improve facial feature detection accuracy.

Problem formulation
Given the initial locations of 26 facial feature points X m = [x m,1 , x m,2 , ..., x m,26 ] generated using individual facial feature detectors, our goal is to estimate the true facial feature locations X = [x 1 , x 2 , ..., x 26 ]. Figure 1 shows the 26 facial feature points. We can formulate this task as an optimization problem under the probabilistic framework: Therefore, if we can construct a model to represent the relationship between X m and X, we can find the true facial feature locations by maximizing the posterior probability of X given X m . Towards this goal, we propose the following hierarchical probabilistic model.

Hierarchical probabilistic model
As shown in Figure 2 (b), our hierarchical model consists of nodes from four layers. Nodes in the lowest layer indicate the measurements of facial feature locations for each facial component. Nodes in the second layer represent the true facial feature locations we want to infer. In the third layer, all nodes are discrete latent random variables indicating the states of each facial component. The top layer contains two discrete nodes representing the facial expression and the pose.
This model captures two levels of information, including local shape variations of each facial component and the joint relationship among facial components, facial expression and pose. The lower level information is embedded among the nodes belonging to the same facial component. The higher level information is represented by the joint relationship among nodes within the rectangle, which are learned through training data. In the following subsections, we will first illustrate the model that captures this two level information, and its properties and benefits. Furthermore, we discuss how to generate the measurements.

Modeling the shape variations for each facial component
In order to capture the face shape variations more accurately, we follow the component based method and model the face shape variations of each facial component including the eyebrow, eye, nose and mouth. However, for each component, the shape may change significantly due to varying facial expressions and poses. So, to represent the shape variations with different conditions, we use the mixture model shown in Figure 2 (a). Here, Z is a discrete variable that represents the hidden state of each facial component. X indicates the true facial feature locations for this component, and X m is the measurement. For one component shown in Figure 2 (a), we model the joint probability of X and X m as follows: P (X|z) describes the face shape pattern under a specific hidden state Z = z. We can use a Multivariate Gaussian Distribution to model this probability.
,where µ z is the mean vector and σ z is the covariance matrix. P (X m |X, z) captures the relationship between measurement and inferred true facial feature locations for a specific state z. It can be modeled as a linear Multivariate Gaussian Distribution. , where β 0,z + β z X is the mean of the Gaussian and Q z is the covariance matrix. P (Z) is the probability for different hidden states. Figure 3 shows some examples of the shape variations of mouth under different hidden states that are learned from data with expression variations. It can be seen that, different hidden states relate to different facial expressions. The combinations of all states describe the overall shape variations of mouth due to facial expression changes.

Modeling the relationship among facial components, facial expression and pose
The states of different facial components are highly related. For example, if people lower their eyebrows, it's difficult to open their eyes widely simultaneously. So, it's important to take advantage of relationships among facial components. Facial expression and pose are two major causes of the significant change of the face shape. As a result, by explicitly adding the label of facial expression and pose for all the training data, the model will be more accurate than one without this additional information.
The relationships among facial components, facial expression and pose are represented as the joint probability of all the nodes inside the dotted rectangle. It is important to notice that all the Z i nodes are hidden and the underlying relationship among Z i , E and P can not be assigned arbitrarily. Consequently, we need to automatically learn the model structure within the rectangle and estimate the parameters of the entire model. The dotted lines within the rectangle in Figure 2 (b) are shown as an special example for illustration purposes only.
With the combination of local shape models for each facial component, and the global relationship among facial components, facial expression and pose, the overall joint probability of X and X m shown in Figure 2 (b) is formulated as: where Z = [Z eb , Z e , Z n , Z m ].

Properties of the proposed model
There are a few important properties and benefits of the proposed hierarchical probabilistic model.
1. The local shape variation of each facial component is modeled implicitly with hidden states. As a result, it could more accurately capture the variations under different conditions. Furthermore, unlike [24] we do not need to assign any particular meaning for each state such as mouth open and mouth closed.
2. The relationship among facial components are represented by the joint distribution of all the hidden nodes Z i . This relationship is automatically discovered based on bottom-up information from the facial feature locations X i and the top-down information from facial expression and pose which are available during training. In this case, the hidden states relate to the major causes of the shape variations, but they also can represent the information within the training data.
3. Facial expression and pose information are the major causes of the face shape variation. In this model, they explicitly help the learning of both the local shape variations and the joint relationship among facial components. This important information is usually ignored by the previous work.

Facial feature measurements
An integral part of our model is the facial feature measurements (X m ). They serve as input to the proposed hierarchical model, help learn the model during training, and are used to infer the true facial feature positions during testing. The facial feature measurements X m = [x m,1 , x m,2 , ..., x m,26 ] are the initial locations of 26 facial feature points. To generate the measurements, we first detect eyes on images based on the Viola-Jones eye detector [26]. The detected eye positions provide normalization and initial locations of the facial features. For each feature point, multi-scale and multi-orientation Gabor-wavelets [5] are employed to generate the representation. The initial feature point positions are further refined by a fast Gabor wavelet jets matching method [9] that computes a displacement to move the initial facial feature positions to their final positions subject to certain shape constraints learned from the training data.

Model learning
Given the true feature location X, its measurement X m , its facial expression label E, and pose label P as training data, we could learn the model. We refer to model learning as learning the model structure and the model parameters. As shown in Figure 2(b), for structure learning, we only learn the global structure, denoted as M G that connects the nodes within the rectangular block, and we fix the local model M L . For parameter learning, we estimate the parameters of the whole model Θ = [Θ G , Θ L ]. To tackle this learning task, we applied the Structure EM algorithm [10] and modified it for our application.
The complete description of our learning algorithm is shown in algorithm 1. Given the training data, in the initialization step, the hidden states Z 0 = [Z eb , Z e , Z n , Z m ] for each data are first inferred by independent clustering of each local facial component, where the number of clusters is exhaustively searched on the validation data set. With the initial estimation of hidden states, initial model structure and parameters are estimated based on the complete data. Similar to structure EM algorithm [10], we iteratively search for better parameters and structure with three major steps, denoted as 1, 2, 3.
In step 1, standard parameter EM algorithm is applied to update the parameters. It keeps the structure unchanged and searches for the parameters that maximize the expected BIC score, as shown in Equation 6.
where E[.] denotes the expectation over the probabilities of the incomplete data under the current model as P (Z|E, P, X, X m : M n , Θ n,l ) , and Z = {Z eb , Z e , Z n , Z m } . N denotes the number of training data, and Dim(.) represents the model complexity in terms of the number of independent parameters. The parameters are updated iteratively to take advantage of the small computational cost of parameter EM compared to structure learning. In step 2, we search for the new structure that maximizes the expected BIC score given the current model and parameters as shown in Equation 8.
where Q BIC (.) has similar definition as in Equation 7. Because of the nice decomposable property of BIC score, we only need to search for the global model, M G , that maximizes the expected BIC score belongs to the nodes within the rectangular block. Here, we applied the structure learning method in [6] which guarantees a global optimality with respect to the expected score function based on branch and bound algorithm. It is important to note that the expected BIC score is calculated over P (Z|X, X m , E, P : M n , Θ n,l ) which is inferred with the whole model using the junction tree algorithm [15]. In step 3, the initial parameters for the next iteration for the new structure are estimated with Equation 9.
The updating stops when the number of iterations reaches the maximal value allowed, or the structure remains the same in two consecutive iterations.

Inference
Once the structure and parameters of the model have been estimated, we have learned the joint probability of the true facial feature locations X and the measurement X m . So, inference of X can be performed by maximizing the posterior probability: It is important to note that, during inference, we marginalize all the discrete latent states Z i , unknown facial expression E and pose P . Here we use the junction tree algorithm [15] to perform the inference.

Experimental Results
In this section, we test our facial feature detection method on several benchmark databases, and compare it to some existing works.

Databases
Facial images with expression variations: Facial expression could dramatically change the appearance and shape of the face. To test our model on images with facial expressions, we used the CK+ [13] [17] database and the MMI [20] database. In our experiments, for each database, we used sequences corresponding to 6 basic facial expressions, including anger, disgust, fear, happiness, sadness and surprised. For each sequence, we used the first frame, onset and apex frames. In total, CK+ contains 1339 images and MMI contains 584 images.
Facial images with pose variations: Pose is another cause for the significant changes of face shape. To verify our model under this condition, we test it on the FERET database [21] and the Annotated Facial Landmarks in the Wild database (AFLW) [14]. In our experiments, for FERET, we used 335 frontal, quarter left and right images and ignore images in other poses since not all facial features were observable in other poses. AFLW contains images collected from the website with large facial appearance and shape variations. In our experiments, we used 4226 images for which all the annotation of inner points are available. To build our hierarchical model, we discretized the pose based on the ground truth yaw (less than"−10 o ", between "−10 o " and "10 o ", larger than "10 o ") and pitch angles (less than "−15 o ", between "−15 o " and "0 o ", larger than "0 o ") provided by the database and ignored the roll angle since it can be excluded by in-plane rotation.
Facial images with expression and pose variations: Possible face shape variations could be due to both facial expressions and poses. In our experiments, we used the ISL multi-view facial expression database [23] to evaluate our method. The ISL multi-view database contains sequences of 8 subjects showing happy and surprised facial expressions under varying continuous face poses. We selected 460 images consisting of about every 5 frames from each sequence and some key frames with significant pose or facial expression changes. We discretized the continuous pose into three states, including the frontal, left and right. For each image in all databases, we manually labeled the facial feature locations as ground truth.

Implementation details
To evaluate the performance of the detection algorithm, we used the distance error metric: ,where D I (j) is the interocular distance measured at frame j,P i,j is the detected point i at frame j and P i,j is the manually labeled ground truth. For each database except ISL, we tested our method based on 10 fold cross-validation strategy. For ISL, to make up the limited training data, we used 46 fold cross-validation.

Algorithm evaluation 4.3.1 Facial feature detection on CK+ database
The learned hierarchical model for CK+ is shown in Figure 6 (a). It can be seen that, only the eyebrow and mouth directly link to the expression node. Eye and nose are independent of expression given the states of mouth. The learned local shape variations of the mouth have been shown in Figure 3, and the shape variations of eyebrow are shown in Figure 4. The learned states are highly related to different facial expressions. Figure 5 (a) explicitly illustrates the relationship between hidden states of mouth and facial expressions, which corresponds to the probability P (Z mouth |E) marked as the red bold line in Figure 6 (a), connecting E and Z mouth . The shape variation of the mouth under state 4 is highly related to the surprised facial expression while state 5 is caused by the happy facial expression. Another important benefit here is that shape variations with the surprised facial expression are modeled with the first 4 states which leads to a more accurate model than using one single state for each facial expression. The overall facial feature detection accuracy on the CK+ database is shown in Table 1. With the learned hierarchical model, performance is better than the manually constructed model without the hidden states shown in Figure 6 (b).   Figure 7 differs from hidden states learned for data from CK+ with expression variations shown in Figure 3. As a result, the global structure among hidden states, facial expression and poses of the model shown in Figure 6 (c) is different from the model shown in (a). The overall detection accuracies are shown in Table 2. Sample image results are shown in Figure 9.    database can generalize to MMI. But, the converse is not true as shown in Table 3.

Comparison with existing works
To compare our model with state-of-the-art works, we include the detection results using the RBM model [27], the BoRMaN model [25], the LEAR model [18], and the FPLL model [29]. We choose these models because they all utilize sophisticated face shape prior models constructed based on graphical models. For all the algorithms, we used the detection codes provided by their authors. For comparison, we used the shared points shown in Figure 8 Detection results are shown in Table 4 and Figure 8 (b). Our detection results are consistently better than the RBM model [27], the BoRMaN model [25], the LEAR model [18], and the FPLL model [29]. For the ISL database, in which facial images are with significant facial expressions and poses, BoRMaN, LEAR and FPLL can not generate reasonable detection results. For the AFLW database, we also compare our algorithm with the Supervised Descent method in [28], which proposes a robust method to solve a nonlinear least squares problem for face alignment. As shown in Figure 8 (b), while our algorithm consistently outperforms RBM, BoRMan and FPLL, it is slightly worse than the Supervised Descent method in a small distance range.

Conclusion
In this paper, we proposed a hierarchical probabilistic model that captures the face shape variations with varying facial expressions and poses and could infer the true facial feature locations given the measurements. There are two major benefits of this model. First, it implicitly models the local shape variation for each facial component. Second, it learns the joint relationship among facial components, the facial expression and the pose in the higher level by searching the optimal structure and parameterizations of the model. During testing, to infer true facial feature locations, the model combines the bottom-up information from local shape model and top-down constraints among facial components. Experimental results on benchmark databases demonstrate the effectiveness of the proposed hierarchical probabilistic model.