{'fraction_non_alphanumeric': 0.030343258107339274, 'fraction_numerical': 0.007206523800493078, 'mean_word_length': 4.70501262170934, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Weighted Linear Discriminant Analysis based on Class Saliency Information

In this paper, we propose a new variant of Linear Discriminant Analysis to overcome underlying drawbacks of traditional LDA and other LDA variants targeting problems involving imbalanced classes. Traditional LDA sets assumptions related to Gaussian class distribution and neglects influence of outlier classes, that might hurt in performance. We exploit intuitions coming from a probabilistic interpretation of visual saliency estimation in order to define saliency of a class in multi-class setting. Such information is then used to redefine the between-class and within-class scatters in a more robust manner. Compared to traditional LDA and other weight-based LDA variants, the proposed method has shown certain improvements on facial image classification problems in publicly available datasets.


INTRODUCTION
Linear Discriminant Analysis (LDA), as a traditional statistical machine learning technique, has been employed for several classification tasks, such as human action recognition [1], [2] and person identification [3], due to its effectiveness in reducing dimensions and extracting discriminative features. In a classification task, LDA is used to define an optimal projection by means of Fisher criterion optimization. Despite the widespread application of traditional LDA, its performance is affected by several issues related to its underlying assumptions. Traditional LDA represents each class with the corresponding class mean and discriminates between classes based on the scatters of these class representations with respect to the total data mean. Such a class discrimination definition may cause large overlaps of neighboring classes [4], and receive a sub-optimal result, since an outlier class being far from the others dominates the solution [4]. Furthermore, in traditional LDA all classes equally contribute to the withinclass scatter definition [5] based on the assumption of the same Gaussian distribution for all classes. This assumption overemphasizes well-separated outlier classes, which should have lower contribution in the overall within-class scatter definition. A method that automatically determines optimized class representations for LDA-based projections was proposed in [6], [7]; however, it also suffers from the class imbalance problems discussed above. In order to overcome aforementioned drawbacks of traditional LDA, extensions imposing weighting strategies for the definition of the withinclass and between-class scatters have been proposed in [8], [9], [10], [11], [12]. In these methods, the weighting factors incorporated to the scatter matrices definitions are based on class statistics, e.g. class cardinality, and class representation is still assumed to be the class mean.
A novel extension of LDA that exploits intuitions from saliency [13] is proposed in this paper. A probabilistic criterion is formulated in order to express the samples around boundary within its original class following a probabilistic saliency estimation framework [14]. Such a definition is naturally expressed by graph notation, in which several types of graphs can be exploited. Both fully connected and k-NN graphs are considered. After defining the probability of each sample belonging to its corresponding class, this information is used to define new class representations, as well as new within-class and between-class scatters. Compared to traditional LDA and its weighted variants, the proposed Saliencybased weighted LDA (SwLDA) has shown enhanced performance on facial image classification problems.
The remainder of this paper is structured as follows. In Section 2, we briefly present related works. In Section 3, we rigorously derive the proposed SwLDA method on the basis of various weighted LDA methods and saliency estimation. Experimental results on publicly available facial image datasets are provided in Section 4, and Section 5 concludes this work.

RELATED WORK
In this section, first we briefly describe original LDA and two of its weighted variants, which have been proposed in order to overcome shortcomings of LDA related to class imbalance problems. Later, visual saliency estimation based on the recently proposed probabilistic interpretation [14] is presented.
In the following, we assume that each training sample is represented by a vector x i ∈ R D and is followed by a class label y i ∈ {1, . . . , C}. A set of training vectors x i , i = 1, . . . , N are used in order to define a linear projection from the input space R D to a discriminant subspace R d such that the representation of the i-th sample is given by z i = W T x i , where W ∈ R D×d is the projection matrix to be learned by optimizing class discrimination criteria.

Linear Discriminant Analysis
LDA defines the optimal data projection matrix W by maximizing the following criterion where S W , S B are within-class and between-class scatter matrices respectively, and defined as follows: In the above, α c i is an index denoting whether sample i belongs to class c, i.e.
The optimal projection matrix W is obtained through applying eigenvalue decomposition of the matrix S = S −1 W S B and keeping the eigenvectors corresponding to the largest (up to C − 1 in total) eigenvalues.

Weighted LDA Variants
Weighted versions of LDA aim at scaling the contribution of each class based on their influences on projection, by defining appropriate weights. In [11], between-class scatter matrix is redefined for enhancing robustness in multi-class problems, as follows: where p c , p j denote the prior probability of class c, class j, respectively. L cj expresses the dissimilarity between class c and class j, using a distance function in the Euclidean (or a Mahalanobis) space. In order to reduce the influence of outlier classes, an outlier-class-resistant weighted LDA method is proposed in this work [9] based on Loog's work [11]. They express the between-class scatter using (4) and a new withinclass scatter definition is proposed as follows: where Lic is a relevance-weight between class c and class i, reducing attention to outlier classes.
Another version of weighted LDA aiming at alleviating the influence of outlier class is proposed in [10]. They define the between-class scatter and within-class scatter as follows: (7) where n c , n j are the number of samples for class c and class j, in addition, w 1 (∆ cj ) and w 2 (∆ c: ) are defined as 1 ∆cj and 1 j =c ∆cj , respectively. ∆ cj is the Fisher's discriminant criterion in the discriminant space determined through applying LDA using the between-class scatter matrix S B and the total scatter matrix S T = S B + S W , i.e.: Using the above definition of ∆ cj , in the case where a class is well separated from all others, a smaller value of w(∆ cj ) will be used, reducing the influence of that class on the result. Once the new S w and S t (S t = S w + S b ) are obtained, the final projection matrix W can be determined by optimizing the following Fisher's discriminant criterion:

Visual Saliency Estimation
Visual saliency estimation has gained attention during the last decade, since it can be applied as a pre-processing step for higher level Computer Vision tasks. Recently, Aytekin et al. formulated the salient object segmentation problem based on probabilistic interpretation. Specifically, they defined a probability mass function P (x) encoding the probability that an image region (in the sense of pixel, super-pixel or patch) to depict a salient region. Estimation of P (x) is formulated as an optimization problem enforcing similar regions to have similar probabilities, while any prior information regarding saliency (defined based on the location of each region in the image lattice) can be exploited. This joint optimization is expressed as: where v i ≥ 0 denotes prior information for region i by nonnegative values and w ij expresses the similarity of regions i and j. The optimization problem in (11) can be expressed using a matrix notation as follows: where p is a vector having elements p i = P (x = x i ) corresponding to the probability of each region to be salient. W is the affinity matrix of a graph having as vertices for the region representations and D is the corresponding diagonal matrix having elements equal to D ii = j W ij . V is a diagonal matrix having elements [V] ii = v i . In visual saliency, the element V ii expresses the a priori knowledge that an image location belongs to background, that is introduced by the user.
As has been shown in [14], the optimization problem in (12) has a global optimum given by: p * pse = H −1 1. Interestingly, the above solution is equivalent to an one-class classification model, making a connection between salient object segmentation and one-class classification problems. In the following, we will use this connection in order to derive a new definition for class-representation and scatter matrices calculation in LDA.

SALIENCY-BASED WEIGHTED LINEAR DISCRIMINANT ANALYSIS
This section describes in detail the proposed weighted versions of LDA. We define the contribution of each sample to the corresponding class, and then new class representations and scatter matrices are proposed accordingly. We start by describing the proposed sample weights.

Sample Weights and Class Representation
Weighted LDA variants represent each class with the corresponding mean vector and define weights based on pair-wise class distances to address the outlier class problem. Such mutation yields a certain improvement over traditional LDA. Nevertheless, it neglects the influences of outlier samples within each class [12], which may affect the classification result greatly. This is due to the fact that all class samples equally contribute to the definition of the class representation and scatter matrix calculation. In our work, we determine the contribution of each sample based on its class saliency information. We define the class saliency information of a sample x i based on its probability to belong to its true class y i . In order to do so, we calculate the probability mass function P c (x) of each class c independently following the probabilistic saliency estimation (PSE) in [14]. That is, for each class c, we form the corresponding graph G C = {X c , W c }, where X c ∈ R D×Nc is a matrix formed by the samples belonging to class c and W c ∈ R Nc×Nc is the graph weight matrix expressing the similarity between the class samples. Any type of graph can be used to this end. In our experiments we have used fully connected and the k-NN graphs, using the heat kernel function: where the value of σ is set equal to the mean Euclidean distance between the class samples, which is the natural scaling factor for each class.
We define a priori saliency information as misclassificationbased probability for the class data to be set in the diagonal elements of the matrix V c . Misclassification-based probability assumes that a sample is less probable to have high saliency information if it is closer to another class, when compared to its true class. In this case, the elements of V c are set equal to: In this case, a sample which is close to another class is assigned to low saliency information, even if it may be close to the center of its class.
After having defined the matrices W c and V c , the probability of each sample x c,i to belong to class c is given by: Having obtained p c ∈ R Nc , c = 1, . . . , C, we define a new class representation as m c = X c p c .

Scatter Matrices Definition
By exploiting class-specific saliency information described above, we can define within-class scatter matrix in two different ways. The first one is to incorporate p c in S w as: where x c,j denotes j-th sample in class c, p c,j is saliency score for j-th sample in class c. The other one is inspired by relevance weighted LDA mentioned in section 2, as: Here Lic is a relevance-weight, where L ic is defined based on the Euclidean distance between pairwise mean vectors of class i and class c, as (18): Definitions of between-class scatter matrix in aforementioned LDA methods simply maximize either the variations between each class mean vector and the total mean vector, or the variations between class pairs. Here, we propose four types of between-class scatter matrices, which are not only based on the aforementioned definition of S b , but also capture the structure inside each class. The first definition is the same as (3): The second one uses saliency scores p c , when generating new class representations, as follows: where X c contains all samples in class c,μ c is the new class representation or weighted center of class c. The third definition extends (21) to exploit the relationships between pairs of new class representation for each class, as follows: The last definition, S b , intends to maximize discrimination between every sample in one class with other new class representations, meanwhile takes into account of each sample's saliency scores, as follows: where N c1 is the cardinality of class c 1 .

Discriminant Criterion
Using the above described scatter matrices, several optimization criteria can be formed as follows: where S and j ∈ {1, 2}. After obtaining projection matrix W by eigenvalue decomposition, we map corresponding class representations and test samples by the optimal W, and then nearest centroid classifier is applied for classification. It should be noted that when H or S t are singular, a regularized version is used.

EXPERIMENT RESULTS
In our experiments, we evaluate the performance of proposed SwLDA, traditional LDA and two weighted LDA approaches mentioned in section 2 on six public facial image datasets: BU, KANADE, JAFFE, ORL, YALE and AR. We evaluate the performance of the proposed SwLDA approaches, as illustrated in Table 1. The results of SwLDA ij illustrate classification accuracy obtained by using the matrices S (ij) t and S The result of traditional LDA is considered as baseline. The results comparison of baseline, Tang's work [9], Jarchi's work [10] and our work are presented in Table 2. We implement standardization on all datasets before training and split each dataset into 5 folds for cross-validation. When obtaining W c , we select k-NN graphs with k ∈ min(5, 0.1 * N c ) or fully connected graphs to evaluate its impact on the results. As shown, the best performances over datasets BU and KANADE are both achieved by using SwLDA 42 with fully connected graphs. SwLDA 41 is the most effective over dataset JAFFE. The maximal improvement is 10.57% on dataset BU using SwLDA 42 with fully connected graphs, compared to the result of traditional LDA. That over Tang's work [9] is 0.14% and over Jarchi's work [10] is 2.28%. SwLDA 12 and SwLDA 22 work better than SwLDA 32 and SwLDA 42 apparently on datasets JAFFE and ORL. Fully connected graphs works better than k-NN graphs does over YALE dataset for all cases. Graph connection does not affect the classification accuracy using SwLDA 11 , SwLDA 21 , SwLDA 41 , SwLDA 22 and SwLDA 32 over dataset AR.

CONCLUSION
In this paper, we propose weighted LDA variants based on a probabilistic definition of visual saliency estimation. We follow a class-specific saliency estimation process in order to determine the contribution of each sample in the optimization problems solved for discriminant subspace learning. Then, we employ our new approaches to six public datasets for evaluation and comparison with related LDA methods. Our new definitions target to reveal connections between each sample in every class, and further solve shortcomings in weighted LDA variants. Experimental results sufficiently demonstrate that the highest classification accuracy is always with one of our proposed approaches over these six facial image datasets.