{'fraction_non_alphanumeric': 0.035152799858682214, 'fraction_numerical': 0.016339869281045753, 'mean_word_length': 4.888195527821113, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Mid-level Representation Enhancement and Graph Embedded Uncertainty Suppressing for Facial Expression Recognition

Facial expression is an essential factor in conveying human emotional states and intentions. Although remarkable advancement has been made in facial expression recognition (FER) task, challenges due to large variations of expression patterns and unavoidable data uncertainties still remain. In this paper, we propose mid-level representation enhancement (MRE) and graph embedded uncertainty suppressing (GUS) addressing these issues. On one hand, MRE is introduced to avoid expression representation learning being dominated by a limited number of highly discriminative patterns. On the other hand, GUS is introduced to suppress the feature ambiguity in the representation space. The proposed method not only has stronger generalization capability to handle different variations of expression patterns but also more robustness to capture expression representations. Experimental evaluation on Aff-Wild2 have verified the effectiveness of the proposed method.


Introduction
Facial expression recognition (FER) is a highly focused field in computer vision for its potential use in sociable robots, medical treatment, driver fatigue surveillance, and many other human-computer interaction systems. Typically, the FER task is to classify an input facial image into one of the following six basic categories: anger (AN), disgust (DI), fear (FE), happiness (HA), sadness (SA), and surprise (SU).
In the field of facial expression recognition (FER), various FER systems have been explored to encode expression information from facial representations. Several studies have proposed well-designed auxiliary modules to enhance the foundation architecture of deep models [22,17,2,1,26]. Yao et al. [22] proposed HoloNet with three critical considerations in the network design. Zhou et al. [26] introduced an spatial-temporal facial graph to encode the information of facial expressions. Another area focuses on facial expression data for robust recognition [19,24,25,21,5]. In [24], the authors proposed an end-to-end trainable LTNet to discover the latent truths with the additional annotations from different datasets. Zhao et al. [25] presented a novel peak-piloted deep network (PPDN) that used the peak expression (easy sample) to supervise the non-peak expression (hard sample) of the same type and from the same subject.
Despite the promising results, prior approaches tend to learn a limited number of highly discriminative patterns and suffer from unavoidable expression data uncertainties. To address these issues, we propose to mix the mid-level representation with samples from other expressions in the learning process, thus suppressing the over-activation to some partial expression patterns and enriching the pattern combination to enhance classifier generalization. In addition, we introduce graph embedded uncertainty suppressing to reduce the feature ambiguity in the representation space.
The competition on Affective Behavior Analysis in-the-wild (ABAW) has been successfully held three times, in conjunction with IEEE FG 2020, ICCV 2021, and CVPR 2022. The goal of our work is to study facial expression recognition based on Aff-Wild2 [8,12,16,10,15,14,13,9,23,11,7] in the 4th competition. We adopted the proposed mid-level representation enhancement (MRE) and graph embedded uncertainty suppressing (GUS) for this FER task. Experiments on Aff-Wild2 show the effectiveness of the proposed method.

Overview
The overall architecture of our proposed method is illustrated in Fig. 1. It mainly consists of two parts, e.g., mid-level representation enhancement and graph embedded uncertainty suppressing. In order to further enhance the robustness of our framework, we adopt the model ensemble strategy by combining DMUE [4] with MRE and GUS.

Mid-level Representation Enhancement
The mid-level representation can support robust high-level representation to improve accuracy [3]. However, the learned mid-level representation tends to be dominated by a limited number of highly discriminative partial patterns, making false predictions when two expressions share significant similarities in specific facial parts that are regular in one category.
For facial image I i , based on the hierarchical structure of deep neural networks, we can take the expression representations of mid-level and high-level according to the layer depth, denoted as R m i and R h i , respectively. We mix the mid-level representation with a sample from other expressions to form an new variant. The main idea is to inject partial features as noise data to improve the generalization capability. Given a facial image I i and a randomly selected sample I j with a different expression, the mixing operation can be denoted as: where ½(·) is an indicator, N is the uniform sampled positions for where the noises are injected,R m i (p) indicates the value of the variant of R m i at position p. This strategy provides a way to allow gradients to suppress the overconfident partial patterns.
Following the procedure, the variantR m i is obtained, which helps prevent the network from over-focusing on a small number of discriminative partial patterns. To amplify this property, we introduce an expression classification module based onR m i as an additional branch via global average pooling (GAP) in the network. In this way, the mid-level expression representation can be enhanced by directly supervised with the expression label. Thus, the total loss for the FER task is defined as: where y i is the expression label for I, the first term denotes the original training loss for expression classification towards the high-level expression representation, the second term denotes the expression classification loss using the variant midlevel representation on the branch, and λ is a balanced parameter.

Graph Embedded Uncertainty Suppressing
For uncertainty suppressing, we use Graph Convolutional Networks (GCN) [6,20] to estabilish semantic relations between the repesentations and reduce the covariate shifting. The inputs are representation mapsF = {f 1 , ...,f N } extracted by backbone feature extractor from the original images. To begin with, we use cosine similarity coefficient to calculated the similarity between different representations as: then, we employ GCN as: where A = A + I is the sum of undirected graph A obtained by Eqn. (3) and the identity matrix, D is the diagonal matrix from A, which is F l and F l+1 are the corresponding input and output representations on the l th level, and W l are the trainable parameters on this level.

Mixed Loss Functions
In the training process, we have conducted multiple combinations of different loss functions, finally we use the combination of cross entropy loss L ce , focal loss L f l [18] and sparse regularization loss L f l [27]. Without loss of generality, the predicted outputs of our model isŶ = {ŷ 1 , ...,ŷ 6 } the ground truth label is Y = {y 1 , ...y 6 }, our final loss function can be formulated as: where ω 1 , ω 2 and ω 3 are balanced parameters.

Predictions Merging and Correcting
We employ a predictions merging strategy to further enhance the robustness of the whole framework. For the features extracted from different sub-networks, we use softmax function to calculate the confidential scores of each expression category, then the confidential scores of each class are summed up as: where weights ω r g , ω r m and ω r d are used to balance the imfluence of different sub-networks in the final predictions. During the training process we found GUS achieved better results than DMUE and MRE in all the expression categories except fear. As a consequence, we design two different weights settings, denoted as S1 and S2. S1: ω r g = 0.6, ω r m = 0.2 and ω r d = 0.2, convicing more on the predictions of GUS. S2: ω r g = 0.4, ω r m = 0.3 and ω r d = 0.3, relying more on the predictions of MRE, DMUE. Finally the argmax function is employed to predict the final outputs.
In addition, we design a predictions correcting strategy for the test set. We adopt a Imagenet pre-trained VGG-19 model to extract features from the test images, and calculate the cosine similarity between different features. The images with similarity higher than 0.93 are classified to the same subsets. During the inference stage, a voting process is carried out in each subset with more than 2 images: If more than two-thirds of the images are predicted as the same category, the predicted results of the remaining images are updated with this category.

Dataset
The Learning from Synthetic Data (LSD) Challenge is part of the 4th Workshop and Competition on Affective Behavior Analysis in-the-wild (ABAW). For this Challenge, some specific frames-images from the Aff-Wild2 database have been selected for expression manipulation. In total, it contains 277,251 synthetic images annotations in terms of the six basic facial expressions. In addition, 4,670 and 106,121 images from Aff-Wild2 are selected as the validation and test sets. The synthetic training data have been generated from subjects of the validation set, but not of the test set.

Settings
We only use the officially provided synthetic images to train the models and the validation images for finetuning. All the images are resized to 224 × 224 in these processes. We use random crop, Gaussian blur and random flip as the data augmentation strategies. We also adopt over-sampling strategy over the classes with less images to reduce data imbalance. For training the MRE branch, we use ResNet-50 as the backbone network. As for the DMUE [4] and GUS branches, we adopt ResNet-18 as the backbone network. All the backbones are pre-trained on ImageNet. We use SGD with a learning rate of 0.001 during the optimization in MRE and DMUE, and a learning rate of 0.0001 in GUS. Table 1 shows the results of the proposed method on the official validation dataset. By employing DMUE [4], MRE and GUS, we achieve 0.28, 0.14 and 0.32 increase on mean F1-score compared with the baseline model (Resnet-50), respectively. By applying the model ensemble strategy, we can achieve another 0.03 increase. Meanwhile, the predictions correcting stratagy contributes another 0.02 increase. Finally, our proposed framework scored 0.8482 on the validation dataset. Table 2 shows the results on the test set of 4th ABAW Learning From Synthetic Data (LSD) Challenge. The five submissions are obtained as follows: (1) Single GUS module; (2) Predictions merging with ensemble weights S1; (3) Predictions merging with ensemble weights S2; (4) Predictions merging with ensemble weights S1 and predictions correcting; (5) Predictions merging with ensemble weights S2 and predictions correcting. As shown in the table, the 4th submission owns the highest performance of 36.51 in F1-score, and we achieve 2nd place in the final challenge leaderboard.

Method
Class

Conclusion
In this paper, we have proposed the mid-level representation enhancement (MRE) and graph embedded uncertainty suppressing (GUS) for facial expression recognition task, aiming at addressing the problem of large variations of expression patterns and unavoidable data uncertainties. Experiments on Aff-Wild2 have verified the effectiveness of the proposed method.