{'fraction_non_alphanumeric': 0.038539941393808126, 'fraction_numerical': 0.013823416995795642, 'mean_word_length': 4.97601827179292, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
ABAW: Learning from Synthetic Data&Multi-Task Learning Challenges

This paper describes the fourth Affective Behavior Analysis in-the-wild (ABAW) Competition, held in conjunction with European Conference on Computer Vision (ECCV), 2022. The 4th ABAW Competition is a continuation of the Competitions held at IEEE CVPR 2022, ICCV 2021, IEEE FG 2020 and IEEE CVPR 2017 Conferences, and aims at automatically analyzing affect. In the previous runs of this Competition, the Challenges targeted Valence-Arousal Estimation, Expression Classification and Action Unit Detection. This year the Competition encompasses two different Challenges: i) a Multi-Task-Learning one in which the goal is to learn at the same time (i.e., in a multi-task learning setting) all the three above mentioned tasks; and ii) a Learning from Synthetic Data one in which the goal is to learn to recognise the basic expressions from artificially generated data and generalise to real data. The Aff-Wild2 database is a large scale in-the-wild database and the first one that contains annotations for valence and arousal, expressions and action units. This database is the basis for the above Challenges. In more detail: i) s-Aff-Wild2 -- a static version of Aff-Wild2 database -- has been constructed and utilized for the purposes of the Multi-Task-Learning Challenge; and ii) some specific frames-images from the Aff-Wild2 database have been used in an expression manipulation manner for creating the synthetic dataset, which is the basis for the Learning from Synthetic Data Challenge. In this paper, at first we present the two Challenges, along with the utilized corpora, then we outline the evaluation metrics and finally present the baseline systems per Challenge, as well as their derived results. More information regarding the Competition can be found in the competition's website: https://ibug.doc.ic.ac.uk/resources/eccv-2023-4th-abaw/.


Introduction
Automatic facial behavior analysis has a long history of studies in the intersection of computer vision, physiology and psychology and has applications spread across a variety of fields, such as medicine, health, or driver fatigue, monitoring, e-learning, marketing, entertainment, lie detection and law. However it is only recently, with the collection of large-scale datasets and powerful machine learning methods such as deep neural networks, that automatic facial behavior analysis started to thrive. When it comes to automatically recognising affect inthe-wild (i.e., in uncontrolled conditions and unconstrained environments), there exist three iconic tasks, which are: i) recognition of basic expressions (anger, disgust, fear, happiness, sadness, surprise and the neutral state); ii) estimation of continuous affect (valence -how positive/negative a person is-and arousal -how active/passive a person is-); iii) detection of facial action units (coding of facial motion with respect to activation of facial muscles, e.g. upper/inner eyebrows, nose wrinkles).
Ekman [11] defined the six basic emotions, i.e., Anger, Disgust, Fear, Happiness, Sadness, Surprise and the Neutral State, based on a cross-culture study [11], which indicated that humans perceive certain basic emotions in the same way regardless of culture. Nevertheless, advanced research on neuroscience and psychology argued that the model of six basic emotions are culture-specific and not universal. Additionally, the affect model based on basic emotions is limited in the ability to represent the complexity and subtlety of our daily affective displays. Despite these findings, the categorical model that describes emotions in terms of discrete basic emotions is still the most popular perspective for Expression Recognition, due to its pioneering investigations along with the direct and intuitive definition of facial expressions.
The dimensional model of affect, that is appropriate to represent not only extreme, but also subtle emotions appearing in everyday human-computer interactions, has also attracted significant attention over the last years. According to the dimensional approach [12,65,55], affective behavior is described by a number of latent continuous dimensions. The most commonly used dimensions include valence (indicating how positive or negative an emotional state is) and arousal (measuring the power of emotion activation).
The fourth ABAW Competition contains two Challenges (i) the Multi-Task-Learning (MTL) one in which the goal is to create a system that learns at the same time (i.e., in a multi-task learning setting) to estimate valence and arousal, classify eight expressions (6 basic expressions plus the neutral state plus a category 'other' which denotes expressions/affective states other than the 6 basic ones) and detect twelve action units; ii) the Learning from Synthetic Data (LSD) one in which the goal is to create a system that learns to recognise the six basic expressions (anger, disgust, fear, happiness, sadness, surprise) from artificially generated data (i.e., synthetic data) and generalise its knowledge to real-world (i.e., real) data.
Both Challenges' corpora are based on the Aff-Wild2 database [27,40,31,37,39] [35, 36,33,32,34,70,30], which is the first comprehensive in-the-wild benchmark for all the three above-mentioned affect recognition tasks; the Aff-Wild2 database is an extensions of the Aff-Wild database [34, 70,30], with more videos and annotations for all behavior tasks. The MTL Challenge utilises a a static version of the Aff-Wild2 database, named s-Aff-Wild2. The LSD Challenge utilizes a synthetic dataset which has been constructed after manipulating the displayed expressions in some frames of the Aff-Wild2 database.
The remainder of this paper is organised as follows. The Competition corpora is introduced in Section 2, the Competition evaluation metrics are mentioned and described in Section 3, the developed baselines in each Challenge are explained and their obtained results are presented in Section 4, before concluding in Section 5.

Competition Corpora
The fourth Affective Behavior Analysis in-the-wild (ABAW) Competition relies on the Aff-Wild2 database, which is the first ever database annotated in terms of the tasks of: valence-arousal estimation, action unit detection and expression recognition. These three tasks constitute the basis of the two Challenges.
In the following, we provide a short overview of each Challenge's dataset along with a description of the pre-processing steps that we carried out for cropping and/or aligning the images of Aff-Wild2. These images have been utilized in our baseline experiments.

Multi-Task Learning Challenge
A static version of the Aff-Wild2 database has been generated by selecting some specific frames of the database; this Challenge's corpora is named s-Aff-Wild2. In total, 221,928 images are used that contain annotations in terms of: i) valence and arousal; ii) 6 basic expressions (anger, disgust, fear, happiness, sadness, surprise), plus the neutral state, plus the 'other' category (which denotes expressions/affective states other than the 6 basic ones); 12 action units. Figure 1 shows the 2D Valence-Arousal histogram of annotations of s-Aff-Wild2. Table 1 shows the distribution of the 8 expression annotations of s-Aff-Wild2. Table 2 shows the name of the 12 action units that have been annotated, the action that they correspond to and the distribution of their annotations in s-Aff-Wild2.
The s-Aff-Wild2 database is split into training, validation and test sets. At first the training and validation sets, along with their corresponding annotations, are being made public to the participants, so that they can develop their own methodologies and test them. At a later stage, the test set without annotations is given to the participants.
The participants are given two versions of s-Aff-Wild2: the cropped and cropped-aligned ones. At first, all images/frames of s-Aff-Wild2 are passed through the RetinaFace detector [8] so as to extract, for each image/frame, face bounding boxes and 5 facial landmarks. The images/frames are then cropped according the bounding box locations. All cropped-aligned images have the same dimensions 112 × 112 × 3. These cropped images/frames constitute the cropped version of s-Aff-Wild2 that is given to the participants. The 5 facial landmarks (two eyes, nose and two mouth corners) have then been used to perform similarity transformation. The resulting cropped-aligned images/frames constitute the cropped-aligned version of s-Aff-Wild2 that is given to the participants. The cropped-aligned version has been utilized in our baseline experiments, described in Section 4.
Let us note that for the purposes of this Challenge, all participants are allowed to use the provided s-Aff-Wild2 database and/or any publicly available or private database; the participants are not allowed to use the audiovisual (A/V) Aff-Wild2 database (images and annotations). Any methodological solution will be accepted for this Challenge.

Learning from Synthetic Data Challenge
Some specific cropped images/frames of the Aff-Wild2 database have been selected; these images/frames, which show a face with an arbitrary expression/affective state, have been used -in a facial expression manipulation manner [28,29,38]-so  as to synthesize basic facial expressions of the same person. Therefore a synthetic facial dataset has been generated and used for the purposes of this Challenge. In total, 277,251 images that contain annotations in terms of the 6 basic expressions (anger, disgust, fear, happiness, sadness, surprise) have been generated. These images constitute the training set of this Challenge. Table 3 shows the distribution of the 6 basic expression annotations of these generated images. The validation and test sets of this Challenge are real images of the Aff-Wild2 database. Let us note that the synthetic data have been generated from subjects of the validation set, but not of the test set. At first the training (synthetic data) and validation (real data) sets, along with their corresponding annotations, are being made public to the participants, so that they can develop their own methodologies and test them. At a later stage, the test set (real data) without annotations is given to the participants. Let us note that for the purposes of this Challenge, all participants are allowed to use any -publicly or not-available pre-trained model (as long as it has not been pre-trained on Aff-Wild2). The pre-trained model can be pre-trained on any task (eg VA estimation, Expression Classification, AU detection, Face Recognition). However when the teams are refining the model and developing the methodology they must only use the provided synthetic data. No real data should be used in model training/methodology development.

Evaluation Metrics for each Challenge
Next, we present the metrics that will be used for assessing the performance of the developed methodologies of the participating teams in each Challenge.

Multi-Task Learning Challenge
The performance measure is the sum of: the average between the Concordance Correlation Coefficient (CCC) of valence and arousal; the average F1 Score of the 8 expression categories (i.e., macro F1 Score); the average F1 Score of the 12 action units (i.e., macro F1 Score). CCC takes values in the range [−1, 1]; high values are desired. CCC is defined as follows: where s x and s y are the variances of all video valence/arousal annotations and predicted values, respectively,x andȳ are their corresponding mean values and s xy is the corresponding covariance value. The F 1 score is a weighted average of the recall (i.e., the ability of the classifier to find all the positive samples) and precision (i.e., the ability of the classifier not to label as positive a sample that is negative). The F 1 score takes values in the range [0, 1]; high values are desired. The F 1 score is defined as: Therefore, the evaluation criterion for the Multi-Task-Learning Challenge is:

Learning from Synthetic Data Challenge
The performance measure is the average F1 Score of the 6 basic expression categories (i.e., macro F1 Score):

Baseline Networks and Performance
All baseline systems rely exclusively on existing open-source machine learning toolkits to ensure the reproducibility of the results. All systems have been implemented in TensorFlow; training time was around five hours on a Titan X GPU, with a learning rate of 10 −4 and with a batch size of 128.
In this Section, we first describe the baseline systems developed for each Challenge and then report their achieved performance.

Multi-Task Learning Challenge
The baseline network is a VGG16 network with with fixed convolutional weights (only the 3 fully connected layers were trained), pre-trained on the VGGFACE dataset. The output layer consists of 22 units: 2 linear units that give the valence and arousal predictions; 8 units equipped with softmax activation function that give the expression predictions; 12 units equipped with sigmoid activation function that give the action unit predictions.
Let us mention here that no data augmentation techniques [53] have been utilized when training this baseline network with the cropped-aligned version of s-Aff-Wild2 database. We just normalised all images' pixel intensity values in the range [−1, 1]. Table 4 illustrates the performance of the baseline model on the validation set of s-Aff-Wild2. Table 4. Multi-Task Learning Challenge: Performance of baseline model on the validation set; evaluation criterion is the sum of each task's independent performance metric.

Learning from Synthetic Data Challenge
The baseline network is a ResNet with 50 layers, pre-trained on ImageNet (ResNet50); its output layer consists of 6 units and is equipped with softmax activation function that gives the basic expression predictions.
Let us mention here that no data augmentation techniques have been utilized when training this baseline network with the synthetic images. We just normalised all images' pixel intensity values in the range [−1, 1]. Table 5 illustrates the performance of the baseline model on the validation and test sets, which consist of only real data of the Aff-Wild2 database. Table 5. Learning from Synthetic Data Challenge: Performance of baseline model on the validation and test sets, which consist of only real data of the Aff-Wild2 database; evaluation criterion is the average F1 Score of the 6 basic expression categories. The performance on the validation set is indicated inside the parenthesis.

Conclusion
In this paper we have presented the fourth Affective Behavior Analysis in-thewild Competition (ABAW) 2022 held in conjunction with ECCV 2022. This Competition is a continuation of the first, second and third ABAW Competitions held in conjunction with IEEE FG 2020, ICCV 2021 and IEEE CVPR 2022, respectively. This Competition comprises two Challenges: i) the Multi-Task-Learning (MTL) Challenge in which the goal is to create a system that learns at the same time (i.e., in a multi-task learning setting) to estimate valence and arousal, classify eight expressions (6 basic expressions plus the neutral state plus a category 'other' which denotes expressions/affective states other than the 6 basic ones) and detect twelve action units; ii) the Learning from Synthetic Data (LSD) Challenge in which the goal is to create a system that learns to recognise the six basic expressions (anger, disgust, fear, happiness, sadness, surprise) from artificially generated data (i.e., synthetic data) and generalise its knowledge to real-world (i.e., real) data. Each Challenge's corpora is derived from the Aff-Wild2 database.