{'fraction_non_alphanumeric': 0.033555062823355505, 'fraction_numerical': 0.01097560975609756, 'mean_word_length': 4.8624350086655115, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Rethinking the Learning Paradigm for Facial Expression Recognition

Due to the subjective crowdsourcing annotations and the inherent inter-class similarity of facial expressions, the real-world Facial Expression Recognition (FER) datasets usually exhibit ambiguous annotation. To simplify the learning paradigm, most previous methods convert ambiguous annotation results into precise one-hot annotations and train FER models in an end-to-end supervised manner. In this paper, we rethink the existing training paradigm and propose that it is better to use weakly supervised strategies to train FER models with original ambiguous annotation.


Introduction
Aiming at analyzing and understanding human emotion, Facial Expression Recognition (FER) is an active field of research in computer vision. In view of its practical importance in various intelligent systems, such as sociable robotics, psychological treatments, and automatic driving, numerous Facial Expression Recognition studies have been conducted.
In recent years, FER has achieved impressive performance on laboratory-controlled datasets, such as CK+ [28], JAFFE [31] and RaFD [21], which are collected under ideal conditions and annotated by experts. Recently, with the requirement of real-world FER applications, large-scale datasets collected in an unconstrained environments, such as FERPlus [4], RAF-DB [25] and AffectNet [35], continue to emerge. However, collecting large-scale datasets with fully precise annotations is a labor-intensive and challenging task. Consequently, the crowdsourcing approach is commonly utilized for annotation. Due to the subjective perception of annotators and the inherent inter-class similarity of facial expressions, the real-world FER datasets usually exhibit ambiguous annotation. To make it easier to use the crowdsourcing annotation results, most previous methods convert the annotation results into precise one-hot annotations by simply voting [4] or thresholding [25] and train FER models in an end-to-end supervised manner. However, these label conversion strategies do not always yield satisfactory results as shown in Figure 1. The performance of models supervised with such converted labels are inevitably impeded.
In this paper, we rethink the existing training paradigm and propose that it is better to use weakly supervised strategies to train FER models with original ambiguous annotation. Specifically, we model FER as a Partial Label Learning (PLL) problem, where each training instance is associated with a set of candidate labels among which exactly one is true, reducing the overhead of finding exact label from ambiguous candidates [29]. The major challenge of PLL lies in the label disambiguation. Existing PLL methods [27,33,43,53] state that examples close to each other in the feature space will tend to share identical label in the label space and their label disambiguation is based on this assumption. This also indicates the reliance of label disambiguation on a good feature representation. However, there is a contradiction here: the direct use of ambiguous annota-tions inevitably affects the quality of representation learning, which in turn hinders label disambiguation. Errors can accumulate severely, especially with the low-quality initial representations [49].
To mitigate the influence of ambiguous labels on feature learning, we propose to use unsupervised learning to obtain the initial feature representation. Specifically, we propose to use the recently emerged Masked Image Modeling (MIM) to pre-train facial expression representation. MIM is mostly built upon the Vision Transformer (ViT), which suggests that self-supervised visual representations can be done by masking input image parts while requiring the target model to recover the missing contents. Facial expressions are high-level semantic information obtained from the combination of action units in multiple regions of the human face. Vision Transformer can model the relationship between action units with the global self-attention mechanism. MIM pre-training forces the ViT to learn the local facial action units and global facial structures in various expressions [18]. Then the pre-trained ViT is finetuned with an ambiguous label set in the PLL paradigm. Moreover, unlike other tasks, facial expression data exhibit high interclass similarity, which makes label disambiguation more challenging. In this paper, we propose to treat the label disambiguation process as querying the confidence of the correlation between each category label and features. Specifically, inspired by DETR [5], we propose a decoder-based label disambiguation module. We leverage learnable label embeddings as queries to obtain confidence in the correlation between features and the corresponding label via the cross-attention module in Transformer decoder. Finally, we integrate the encoder of ViT backbone, and the Transformer decoder-based label disambiguation module to build a fully transformer-based facial expression recognizer in the partial label learning paradigm.
Our contributions are summarized as follows: • We rethink the existing training paradigm of FER and propose to use weakly supervised strategies to train FER models with original ambiguous annotation. To the best of our knowledge, it is the first work that addresses the annotation ambiguity in FER with Partial Label Learning (PLL) paradigm.
• We build a fully transformer-based facial expression recognizer, in which we explore the benefits of the facial expression representation based on MIM pre-train and the Transformer decoder-based label confidence queries for label disambiguation.
• Our method is extensively evaluated on large-scale real-world FER datasets. Experimental results show that our method consistently outperforms state-of-theart FER methods. This convincingly shows the great potential of the Partial Label Learning (PLL) paradigm for real-world FER.

Facial Expression Recognition
The purpose of Facial Expression Recognition (FER) [1,35,37,45] is to help computers understand human behavior and even interact with humans, which can be mainly divided into traditional-based and deep learning-based methods. For the former, it mainly focuses on hand-designed texture-based feature extraction for in-the-lab FER datasets, on which the recognition accuracy is very high.
Since then, most studies tried to tackle the in-the-wild FER task. These large-scale datasets are collected in an unconstrained environment and annotated by crowdsourcing. Due to the subjective perception of annotators and the inherent inter-class similarity of facial expressions, the realworld FER datasets usually exhibit ambiguous annotation. Previous methods [12,16,40,41,51] convert the annotation results into precise one-hot annotations by simply voting [4] or thresholding [25] and train FER models in an end-to-end supervised manner. However, these label conversion strategies do not always yield satisfactory results as shown in Figure 1. The performance of models supervised with such converted labels is inevitably impeded. In recent years, several attempts try to address the ambiguity problem in FER. SCN [45] and RUL [56] relabel noisy samples automatically by learning clean samples to suppress uncertainty from noise labels. DMUE [38] proposes a method of latent label distribution mining and an inter-sample uncertainty assessment, which aims to solve the problem of label ambiguity. However, they either require the noise rate to better filter out noisy samples or bring extra computation overhead and cannot generalize well to classification tasks with a large number of classes.
Recently, some researchers move from fully-supervised learning (SL) to semi-supervised learning (SSL). Ada-CM [23] proposes an SSL method by adaptive learning confidence margins to utilize all unlabeled samples during training time, which represents a new idea to solve FER tasks by making use of additional unlabeled data. In this paper, we rethink the existing training paradigm and propose that it is better to use weakly supervised strategies to train FER models with original ambiguous annotation. As Figure 1 shows, in-the-wild facial expressions often include compound emotions or even mixture ones. Thus, for these samples with uncertainty, it is natural to have multiple annotated candidates. Consequently, for the first time, a new Partial Label Learning (PLL) paradigm is proposed to address the annotation ambiguity in FER. Figure 2. The overview of our framework. In the left part, we use the pre-trained ViT encoder as our backbone for feature representation of the 2D image, In the right part, we input the obtained feature from the left part to the transformer decoder with learnable label embedding. After that, we revise (i−1) th confidence and update it to get i th confidence. Finally, the loss is computed between logits and ith confidence.

Partial Label Learning
Partial Label Learning (PLL) is an important category of weakly supervised learning methods in which each object has multiple candidate labels, but only one of them is the ground-truth label. It is mainly divided into average-based and identification-based methods. The most intuitive solution is average-based methods, which generally consider each label to be equally important during training time, and make predictions by averaging the output of all candidate labels [20,54]. Others [9,55] utilize parametric model to maximize the average scores of candidate labels minus that of non-candidate labels. The identification-based methods normally maximize the output of the most likely candidate labels in order to disambiguate the ground-truth label. Some typical methods mainly include maximum margin criterion [52], graph-based [42]. With the development of deep learning, more network structures [14,30,48] emerge, which rely on the output of the model itself to disambiguate the candidate label sets. Some researchers argue [26,32,58] that data points closer in feature space are more likely to be classified as the same label, which relied on a good feature representation. PICO [44] reconciles PLL-based label disambiguation and feature representation based on contrastive learning. However, it is required to elaborately design data augmentation due to its sensitivity to positive sample selection. In addition, the high inter-class similarity of facial expressions made it hard to obtain robust feature representations, which is a drawback of contrastive learning in FER.

Masked Image Modeling
Inspired by Masked Language Modeling (MLM), Masked Image Modeling (MIM) becomes a popular selfsupervised method. iGPT [7] predicts unknown pixels by operating on known pixel sequences. ViT [11] predicts masked patch prediction by employing a self-supervised approach. Moreover, iBOT [60] achieves excellent performance through siamese networks. However, these methods have the assumption of image semantic consistency. The ability of BEiT [2] to predict discrete tokens depends on the pre-trained model VQVAE [39] that it relies on. As an extension of the ViT work, the main aim of SimMIM [50] is to predict pixels. Unlike the previously mentioned approaches, MAE [19] proposes an encoder-decoder architecture that uses the decoder for the MIM task. Differently, Maskfeat [47] adopts hog descriptor as prediction target rather than pixels.

Proposed Method
In this section, we introduce a fully transformer-based facial expression recognizer with a new partial label learning paradigm, which mainly consists of a process of robust representation learning and label disambiguation as shown in Figure 2. Specifically, we explore the benefits of the facial expression representation based on MIM pre-train and the Transformer decoder-based label confidence queries for label disambiguation, which is detailed in the following sections.

Problem Formulation
Generally, given an in-the-wild FER dataset D(X , Y), each sample x i belongs to one of the K categories, we denote y i ∈ {y 1 , y 2 , ..., y k } as the deterministic category. This simplified form for labels is consistent with the supervised learning setting. However, as shown in Figure 1, most of the FER data is annotated by a way of crowdsourcing. Therefore, it is not ideal for most methods to simplify the FER task to supervised learning (SL) and use the one-hot label on all labeled data while ignoring the noise inherent in the label level. Figure 3. We adopt Masked Image Modeling (MIM) to pretrain, which predicts the hog descriptor (HOG) with the randomly masked input. After obtaining the pre-trained model, we use it to fine-tune our framework for the FER task.
Partial-label learning (PLL) is a multi-class classification problem. Similar to SL, PLL aims to predict a true label of the input using a mapping of the classification function. However, PLL can tolerate more uncertainty from the label space, and gradually eliminate it in a weakly supervised way during the training stage. We argue that modelling the FER task as partial label learning is more appropriate. PLL assumes that each input x i must have a true label y i in the corresponding candidate label set Y i as the Eq. (1): where S is the collection of all candidate label sets. In addition, different from SL setup, the definition of each sample Thus, the aim of partial label learning (PLL) is to recognize the (unseen)true label y i of each sample x i from Y i = {y 1 , y 2 , ..., y k } by means of train a classifier ζ : X → C k , which utilize to compute the probability of each class in the candidate label set. We construct a candidate label set Y i for the sample x i in the FER dataset, which Y i is a set of binary codes. e.g. y k = 1 if the crowdsourcing result contains this category, else y k = 0. Before training, we normalized each label set Y i to obtain a confidence vector C i = {c 1 , c 2 , ..., c k }, which represents the probability value of the sample x i in the different categories, it sums to 1. In training stage, we train a classifier ζ : X → C k to update the confidence vector c i . Ideally, the c i will concentrate most of the probability density on the true label y i of the sample x i . We use the categorical cross-entropy loss to constrain it as the Eq.(2) following: where i denotes the index of training sample x i and j denotes the index of confidence vector C i . And ζ i (x i ) is the soft-max layer of the model with the input x i . In the next section, we continue to describe the feature representation part of our proposed method.

PLL with Pre-trained Feature Representation
In this work, we use MIM to obtain a better initial feature representation and based on that, use PLL to do label disambiguation. An important assumption of PLL is that samples close to each other in sample space are more likely to share the identical label in label space, which indicates that feature robustness is important for label disambiguation. However, as shown in Figure 1, the uncertainty of the FER data leads to ambiguity at the label level, which hinders the feature representation. Based on these observations, as shown in Figure 3, we use MIM pre-training method to optimize feature representation. Furthermore, since the network is easily misled by the incorrect labels, guiding feature learning by labels alone is difficult for the FER task, which is one of the bottlenecks of current most fully-supervised learning methods. Therefore, we argue that using a self-supervised pre-training manner can help in obtaining a better feature representation by avoiding being misled by incorrect labels.
Inspired by MaskFeat [47], we adopt Masked Image Modeling (MIM) to pre-train facial expression representation by predicting hod descriptor. Specifically, in the encoder part, we adopt the same structure with ViT [11] to pre-train, which is only for visible, unmasked patches. This is because the MIM pre-training manner forces the ViT to learn the local facial action units and global facial structures in various expressions. In FER, focusing on the variations in the action units of the facial expression is more meaningful rather than the information embedded in the pixel level. Thus, in the decoder part, we adopt hog descriptor as prediction of decoder rather than pixel values. In addition, predicting pixels is easy to influence by redundant information (high-frequency details, lighting and contrast variations), which is less important in FER task. In contrast, hog descriptor [10] is dedicated to describe the distribution of edge directions within a local subregion. Finally, the output of decoder can reshape to a reconstructed hog prediction of the image. In the training stage, we only keep the encoder for feature extraction and discard the decoder part.

Label Disambiguation With Transformer Decoder
Given an image x, input pre-trained encoder to get visual token F HW ×d , where HW and d denote the number of visual tokens and the feature dimension of hidden state respectively. We initialize the query embedding Q k×d , where k denotes number of classes. In training stage, transformer decoder update Q k×d according to visual tokens, which reflects the correlation of category-related features. Inspired by DETR [6], we link the category-related features to the learnable query embedding Q k×d and predict the confidence sector C i of the sample by their correspondence. In detailed, we input the logits from the last layer of transformer decoder to sigmoid function to get confidence vector C i .

Label embedding regularization
It is worth noting that in addition to the ambiguity annotation these FER datasets also prevalently exhibit imbalance distribution. In the training stage, it makes the model more biased towards majority classes, resulting in an uneven distribution of the feature space, and degrading model performance by classifying minority classes as head classes. To mitigate its negative effects of it, inspired by [46], the ideal query embedding should be evenly distributed over a hypersphere. Thus, we design a uniform loss for query embedding to pull the distance between different classes on the hypersphere. Specifically, we design the target positions of K classes, before training, we first compute the optimal positions for different classes of query embedding, {t i } K i=1 , and optimize it as the Eq.(3).
where τ > 0 is a temperature hyperparameter, K denotes the number of classes in FER dataset.

Revision confidence
As we mentioned before, the robust representation in the feature space promotes the ability of PLL to disambiguate in the label space. For these samples with incorrect labels in the dataset, we first initialize the pseudo targets for candidate label set Y with a uniform distribution C i = 1 |Yi| Y i . Then, during the update, we continuously revise the confidence score C i for each batch based on the Eq.(4).
where ϕ = sigmoid(σ logits ) is the response of sigmoid layer, and σ logits is the output of the transformer decoder. After that, we compute the top2 metrics of the (C i−1 * ϕ) for updating confidence vector C i as the Eq. (5): where ∆v = |v 1 − v 2 |, the threshold ∈ [0,1), which shows the amount of uncertainty between top2 rank label in the candidate label set. Algorithm 1 is pseudo code of our method.

Algorithm 1 Pseudo-code of our method
Require: M odel f : Pre-trained model f ; Epoch T max : Number of total epochs; Iteration I max : Number of total iterations; query embedding: query Q ; DatasetD : partially labeled training setD =

Experiments
In this section, we first describe the FER datasets and experiment implementation details. To evaluate our proposed method, we compare our method with baseline and other related methods including fully-supervised learning: PSR [40], DDA [12], SCN [45], SCAN [16],DMUE [38], RUL [56], DACL [13], MA-Net [59], MVT [22], VTFF [34], EAC [57]; partial label learning: RC [15], LW [48], PICO [44]. [24] contains around 30K great-diverse facial images with single or compound expressions labeled by 40 trained human annotators in a crowdsourcing way. In our experiments, we use 12271 images for training and 3068 images for testing. FERPlus [3] is an extended dataset of the standard emotion FER dataset, which provides a set of new labels for each AffectNet [36] is one of the largest and most challenging FER datasets with manually labeled 440k images. According to previous work, we make two different divisions of AffectNet7 and AffectNet8. Compared to AffectNet7, there is one more expression of contempt with 3667 training images and 500 test images.

Baseline and Experiment setup
We adopt pre-trained ViT-B/16 [11] on Imagenet-1K without any modification as our baseline. For data preprocessing, we keep all the images with the size of 224 × 224 and use MobileFacenet [8] to obtain aligned face regions for RAF-DB and FERPlus. For AffectNet, alignment was obtained via the landmarks provided in the data. In the pretrain part, we use four Tesla V100 GPUs to pre-train the ViT for 600 epochs with batch size 192 and learning rate 2e-4. We set the mask ratio of MAE to 0.4. In finetune part, we use only two Tesla V100 GPUs. We utilize the AdamW optimizer, setting the batch size to 320 and the initial learning rate to 1e-4 with the weight decay of 5e-2. The pre-training and fine-tuning protocol that we use is based on [47]. In the PLL setup, for the FER dataset without crowdsourcing results, we construct the candidate set for it based on LW [48].

Comparison with SL methods
Compared with latest SL methods, we surpass EAC [57] by 2.06%, 1.79% on RAF-DB and AffectNet7 respectively, PSR [40] by 1.36% on FERPlus, and DMUE [38] by 1.28% on AffectNet8. These experimental results show the effectiveness of our proposed method in the FER task and demonstrate that converting ambiguous annotation results into one-hot labels using fully-supervised learning is not ideal. Compared to PLL, these CNN-based SL methods are misled by the incorrect labels in the label space, which hinders the performance of the model. Furthermore, the transformer-based SL method suffers from the same problems. Although MVT [22] uses ViT as the backbone for pre-training, it is not unsupervised pre-training and can still be misled by incorrect labels in the label space, thus affecting the quality of feature learning. The same problem exists with VTFF [34].

Comparison with PLL methods
We add several representative works of PLL methods, respectively RC [15], LW [48] and PICO [44]. The results in Table 1 are our implementation based on their opens-source codes. The general PLL method for label disambiguation, such as RC and LW relies on the robustness of the features, and from the results, for the FER task, RC and LW are misled by incorrect labels in the label space, which affects disambiguation due to the lack of robust facial expression feature representation. In addition, it is worth noting the result of PICO demonstrates that contrastive-based partial label learning does not apply to the FER task. As mentioned before, the contrastive-based approach is sensitive to data augmentation. Unlike objects, facial expressions have high inter-class similarities, which consist of five fixed facial action units. Whereas typical data augmentation usually works on the lighting and contrast variation of the image, which only leads to extra redundant information in the FER task, sometimes, even performance degradation.

Effectiveness of PLL paradigm.
To verify the effectiveness of PLL, we select DMUE [38] for a fair comparison, we only keep the PLL part with ResNet-18 as the backbone, and remove the other key modules, we report the results in Table 2. Compared to DMUE, we exceed DMUE across the board, improving the accuracy by 0.91% on RAF-DB, 1.16% on FERPlus, and 0.05% on Affectnet8. From the results, simplifying FER as a supervised learning task is not ideal, DMUE mines and describes the distribution of the label space corresponding to the samples. However, it is based on supervised learning, which easily suffers from incorrect labels in the label space. In addition, we adopt pre-trained ResNet-18 as the backbone, which is a less robust representation in FER compared to MAE. However, we still obtain competing results using PLL. For a more visual illustration, we have attached To verify the effectiveness of the MIM pre-train protocol and hog descriptor for the FER task. We use different pre-trained models to predict the pixel and hog descriptor, respectively. In addition, we also test them in the PLL paradigm, we report the related results in Table 3. From the results, on the one hand, the use of MIM pre-training avoids the negative effects of incorrect labels and obtains a better feature representation for FER compared to the SL pre-training protocol. On the other hand, MIM pre-training force the ViT to learn the local facial action units and global facial structures using self-attention mechanism in various expressions.
As for the prediction of the decoder, the result of HOG is better than that of Pixel. In FER, we focus more on the different action units of facial expressions rather than the redundant information (illumination, contrast variation) between pixels. Thus, hog descriptor is more ideal one to predict.

Evaluation of tranformer decoder and key modules
To evaluate the effectiveness of the transformer decoder and other key modules, we conduct an ablation study on RAF-DB and FERPlus as shown in Table 4. From the results, based on the PLL paradigm, the transformer decoder improve accuracy by 0.94% on RAF-DB and 0.55% on Af-fectNet7. The results show that it uses the cross-attention module to obtain the correlation of features and category labels via learnable label embedding, which in turn provides a more adequate foundation for the PLL disambiguation. As for the label uniform embedding module, L unif orm alleviates the influence of imbalanced FER data classes by distributing query embedding uniformly over the hypersphere. As for the revision confidence module, the revision confidence strategy can further facilitate the ability of PLL to disambiguate in the label space. This module enhances performance as an auxiliary module of disambiguation. For more details, we provide them in the appendix.

Conclusion
In this paper, we rethink the existing training paradigm and propose that it is better to use weakly supervised strategies to train FER models with original ambiguous annotation. To solve the subjective crowdsourcing annotation and the inherent inter-class similarity of facial expressions, we model FER as a Partial label learning (PLL) problem, which allows each training example to be labeled with an ambiguous candidate set. We use Masked Image Modeling (MIM) strategy to learn the feature representation of facial expressions in a self-supervised manner. Then a Transformer decoders module is proposed to query the probability of ambiguous label candidates being the ground truth. Extensive experiments show the effectiveness of our method.