{'fraction_non_alphanumeric': 0.026310442909386427, 'fraction_numerical': 0.004774481917919545, 'mean_word_length': 4.804834905660377, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Smile recognition based on the fusion of the face texture feature and the mouth shape feature

Smile recognition has recently attracted significant attention due to the increasing accessibility in some digital products. Traditionally, the mouth feature has been extracted for smile recognition. However, it only extracts the features in the mouth region, ignoring the fact that face feature contribute to characterize expressions. In this paper, in order to overcome the drawbacks and make full use of the merits of both features, we propose a novel feature fusion approach for the smile recognition. In our approach, the mouth features are described by HOG descriptor and the face features are represented by Local Binary Pattern (LBP) histograms. Then, the two features are fused by discriminative canonical correlation analysis (DCCA). Robust Support Vector Machine (SVM) classifier is used for classification in our experiment and satisfactory results of experiments based on two databases are obtained finally.


Introduction
In recent years, with the rapid development of digital products and the growing demands from application, increasing attention has been paid to smile recognition from areas of lab research to actual application. For example, one application is to automatically detect smiles when taking photos, and this technique has been commercialized as a new function of digital cameras. As a result, improving the recognition performance of such digital systems has become an essential task.
Smile is regarded as an important expression, and for smile recognition, an essential task is to extract the features from face image efficiently, which is the principle problem of expression recognition. Over the last decade many methods have been proposed, which can be divided into two categories, image-based method and feature-based method [1]. The former is to extract features from face regions of an image. For example, using 2-D DCT [2], LBP [3] and the Gabor [4] in the extracting of the whole face features for expression recognition. The later is to extract features from components of a face, such as nose, mouth, eyes and eyebrows. For instance, in [5] only the eye shape changes are extracted.
However, it has also been noticed that both the image-based and the feature-based strategies have their own limitations. Since the feature-based method only takes the contribution of components of a face into account, ignoring the fact that the underlying face structure of particular face determinate the unique way for facial expression. Conversely, the image-based method takes the whole face into account but fail to lay stress on some regions with more clues.
To overcome the above limitations and preserve their merits of the two kinds of features, an effective way is to find a proper fusion of the applicable features. For this reason, we introduce a fusion approach for the two kinds of features based on the analysis of the correlation within them. More formally, we call the fusion method as discriminative canonical correlation analysis (DCCA) [6]. First, given that it is normally deemed that the mouth region for smile recognition plays a much more important role in comparison with the other regions on a face, we extract the features on the mouth region and HOG operator is introduced for representing the shape of mouth region. Second, block-weighted LBP operator is used to extract features on the whole region of human face because of its low computation cost and exceptional texture description ability. Finally, the two features are fused by DCCA. The experimental results based on two databases illustrate the superiority of the proposed method.
While regarding classifier design, SVM [7] is adopted here since it has been successfully applied to expression recognition.
The remainder of the paper is organized as follows: In section 2, the representations of the face and mouth features are discussed. Section 3 presents the fusion method, and in section 4 we expatiate upon the experiment based on person-independent. Finally, conclusions are given in section 5.

Feature representation
The face feature is represented by LBP histograms while the mouth feature is represented by HOG descriptor LBP operator, introduced by Ojala [8] in 1996, is capable of describing texture. In this paper, we use the extended version of LBP operator chiefly, denoted by R P LBP , . Let R denote the radius of a circle, the centre of which is the central pixel and the number of the pixels is specified by P in the circle. The formula is denoted as . c g , c g are gray value of the p-th and center pixels.
HOG descriptor can effectively express the shape features and have been used in many different problems in computer vision, such as person detection and face detection [9]. The algorithm for extracting HOG begins by counting occurrences of edge orientations in localized portions of an image. The image is divided into small spatial regions, called cells, and for each cell a histogram of edge orientations is computed. The histogram channels are evenly spread over 0 to 180 degrees or 0 to 360 degrees, depending on whether the gradient is unsigned or not.
However, the histogram counts alone usually could not give us satisfactory results. For better invariance to illumination, shadowing, etc., the histogram counts should be contrast-normalized. This can be realized by accumulating a measure of local histogram "energy" over somewhat larger spatial regions ("blocks") and using the results to normalize all the cells in the block. The combination of these histograms then represents the descriptor.

Feature fusion
Following the methods stated above, we obtain two kinds of features to represent smile expression. The fact that both features describe the same pattern inspires us that there should be correlation that can describe the relevant information contained in these two features. And we depend on the correlation to perform feature fusion. In the following, we present the feature fusion based on the DCCA in detail.

The basic idea of DCCA
CCA [10], which studies the correlation between two information channels, has been employed as a powerful tool for feature fusion in the realization of recognition system. However, one limitation of CCA is that it is unable to utilize supervised information. To this end, we introduce a new supervised learning method, namely discriminative CCA (DCCA), into the smile recognition system. DCCA aims to seek pairs of projection vector 1 2 ( , , , ) y y y yd = W w w w  for feature extraction such that the within-class correlation is maximized and the between-class correlation is minimized. More formally, DCCA can be formulated as the following optimization problem [11]: Where w C and b C denote the matrices of the within-class and between-class correlations respectively, ) 0 ( > η η is a tunable parameter that indicates the relative significance of the within-class correlation versus the between-class correlation Given two sets of mean-normalized samples as The purpose of DCCA is compute x w and y w such that the within-class correlation is maximized and the between-class correlation is minimized. The within-class correlation matrix w C can be defined as x y XDY (3) In contrast, the between-class correlation matrix b C can be defined as indicates that when DCCA maximizes the within-class correlation, at the same time, the between-class correlation is automatically minimized. Hence, the optimization problem of DCCA can be reformulated as follows:

The fusion algorithm
Once the basis vector pairs Where z is the fused feature for the subsequent recognition, and d satisfies the constraints d c ≤ and min( , ) d p q ≤ due to rank( ) c ≤ D . Now, we can summary the step of our feature fusion method based on DCCA as Algorithm 1: Algorithm 1. Feature fusion algorithm Input: the face feature described by LBP histograms and the mouth feature represented by the HOG descriptor Output: the fused feature z Step1: Setting the matrix X and Y .
Step2: According to Step1, compute the within-class correlation matrix w C with Eq.(3). Step3: Solving the eigenvalue problem of Eq.(5) and attaining x W and y W Step4: with Eq.(6) the fused feature z is our result.

Experiments
In this paper, robust SVM classifier is adopted for classification, and our experiment is to assess the performance of the proposed approach. We use the proposed fusion method to conduct the face texture and the mouth shape features. In order to illustrate the improvement of the proposed method, we also perform smile recognition with the face texture feature described by the block-weighted LBP histogram, the mouth shape feature represented by the HOG descriptor, the combined feature sets of the two kinds of features in series and the fused features based on CCA for comparison. And to sufficiently test the accuracy of our approach, we performed smile recognition on the JAFFE and CKAUCFE databases [12]. Table 1 indicates the recognition accuracy on the JAFFE and the CKAUCFE, respectively. The best results have been highlighted in boldface. As can be seen on this table, the recognition accuracy in our approach is better than other approaches.

Conclusions
In this paper, we propose a method that combines the face texture feature and the mouth shape feature to perform smile recognition. We select the block-weighted LBP histograms to describe the face texture feature and the HOG descriptor to represent the mouth shape feature. The discriminative canonical correlation analysis (DCCA) is employed to fuse the two kinds of features. The experimental results demonstrate the effectiveness and superiority of our approach. In the following, we expect that it also has great potential for other facial expression recognition task, which merits our future study.