{'fraction_non_alphanumeric': 0.03522443890274314, 'fraction_numerical': 0.019482543640897756, 'mean_word_length': 5.070955534531693, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Facial Expression Recognition based on Multi-head Cross Attention Network

Facial expression in-the-wild is essential for various interactive computing domains. In this paper, we proposed an extended version of DAN model to address the VA estimation and facial expression challenges introduced in ABAW 2022. Our method produced preliminary results of 0.44 of mean CCC value for the VA estimation task, and 0.33 of the average F1 score for the expression classification task.


Introduction
Recognition of facial expression is essential for various interactive computing domains, such as human-computer/machine interaction, human-robot interaction, and human-AI interaction. Previous studies on facial expression mainly utilized a set of human faces captured in a controlled setting, resulting in various limitations of the application in-the-wild. Recently, various works focusing on the affective behavior analysis in-the-wild have been introduced to realize the generation of trust, understanding and closeness between humans and machines in real life environments [17].
The 3rd competition on Affective Behavior Analysis in-the-wild (ABAW), held in conjunction with the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR) 2022, is a place where researchers can present their own contributions on the automatic analysis of human behavior and emotion recognition which is robust to video recording conditions, diversity of contexts, and timing of display [17]. The 3rd ABAW competition is based on the Aff-Wild2 database [2][3][4][5][6][7][8] which is an extension of the Aff-wild database [9] and consists of the following four tracks: 1) Valence-Arousal (VA) estimation, 2) Expression classification, 3) Action Unit (AU) detection, and 4) Multi-Task-Learning (MTL). In this paper, we describe our methods on VA estimation challenge and expression classification challenge and provide preliminary results. For the VA estimation challenge, totally 564 videos of around 2.8M frames that contain annotations in terms of valence and arousal (values ranged continousouly in [-1, +1]) were used. Similarly, for the expression classificaiton challenge, totally 548 videos of around 2.7M frames that contain annotations in terms of the 6 basic expressions (i.e., Anger, Disgust, Fear, Happy, Sad, Surprised), plus the neutral state, plus a category 'other' that denotes expressions/affective states other than the 6 basic ones were used. As evaluation metrics, mean Concordance Correlation Coefficient (CCC) of valence and arousal and the average F1 Score across all 8 categories were used for the VA estimation challenge and the expression classification challenge, respectively. In this paper, we propose an extended version of DAN model based on ResNet with attention mechanisms proposed by [10] to solve the challenges mentioned arXiv:2203.13235v1 [cs.CV] 24 Mar 2022

Data Pre-processing
The amount and the diversity of data is one of the most important key points for successful deep learning applications in terms of a model performance. However, as shown in Table 1, the Aff-wild2 dataset has a class imbalance problem, resulting in some emotional categories have far fewer images than others. To address this issue, we took two strategies, adding external databases and applying data augmentation. First, we used external facial expression databases, such as AffectNet [13], ExpW [14], and Ai-Hub vision dataset [15]. The sample images from each dataset can be found from Figure  2. As shown in the figure, Aff-wild2 and AffectNet share the same facial expression categories while ExpW and Ai-Hub datasets had only part of expression categories. The aforementioned databases were generally collected under in-the-wild setting. Note that Ai-hub dataset [15] is comprised of facial expressions taken by Korean actors in-the-wild. Among the classes included in the Ai-hub dataset, we only used a set of images with neutral, anger, fear, and surprise expressions. Second, we employed various data augmentation techniques; color jitter, random crop, horizontal flip, color jitter with random crop, random crop with flip to prevent over-fitting. Finally, we cropped the face region of each image using DeepFace face detector [16] algorithm and then resized every patch into 224 x 224 scale. Table 1 shows the statistics of dataset we used when training the model.

Model Architecture
The overall architecture of our method is illustrated in Figure 1. Our method is based on the DAN approach which consists of the following two modules: a feature extractor and an attention phase. First, the feature extractor module extracts the intermediate visual features from input images with a discriminative loss function, named affinity loss, to maximize the classes margin. For feature extraction, we utilized a ResNet-50 network pretrained on VGGFace2 dataset. Afterwards, a multihead attention network consisting of a combination of a spatial attention unit and a channel attention unit takes the features and outputs an attention map. Finally, attention fusion network merges attention maps to be learned in an orchestrated fashion [10]. Second, after applying the process mentioned above, the final feature information is fed to fully connected layer and batch normalization layer. At the final step of the model architecture, we adopted a softmax layer with focal loss for expression classification task, and a tanh function with CCC loss for VA estimation task. Finally, in this work, we applied a soft-voting based bagging approach (i.e., multiple variants models are trained and used for validation together).

Results
All the experiments were conducted using a GPU server with six NVIDIA RTX 3090 GPUs, 128 GB RAM, and an Intel i9-10940X CPU. We used Pytorch framework for the implementation/modification, training and evaluation of the model. Our preliminary results on the official validation set for the VA estimation task was 0.44, and 0.33 for the expression classification task.

Conclusion
In this paper, we proposed an extended version of DAN model to address the VA estimation and facial expression challenges introduced in ABAW 2022. Our method produced preliminary results of 0.44 of mean CCC value for the VA estimation task, and 0.33 of the average F1 score for the expression classification task. The details and results may be updated after submission of this paper to arxiv.