{'fraction_non_alphanumeric': 0.03449517263950254, 'fraction_numerical': 0.014269350351824578, 'mean_word_length': 4.930900621118012, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
QMagFace: Simple and Accurate Quality-Aware Face Recognition

Face recognition systems have to deal with large variabilities (such as different poses, illuminations, and expressions) that might lead to incorrect matching decisions. These variabilities can be measured in terms of face image quality which is defined over the utility of a sample for recognition. Previous works on face recognition either do not employ this valuable information or make use of non-inherently fit quality estimates. In this work, we propose a simple and effective face recognition solution (QMagFace) that combines a quality-aware comparison score with a recognition model based on a magnitude-aware angular margin loss. The proposed approach includes model-specific face image qualities in the comparison process to enhance the recognition performance under unconstrained circumstances. Exploiting the linearity between the qualities and their comparison scores induced by the utilized loss, our quality-aware comparison function is simple and highly generalizable. The experiments conducted on several face recognition databases and benchmarks demonstrate that the introduced quality-awareness leads to consistent improvements in the recognition performance. Moreover, the proposed QMagFace approach performs especially well under challenging circumstances, such as cross-pose, cross-age, or cross-quality. Consequently, it leads to state-of-the-art performances on several face recognition benchmarks, such as 98.50% on AgeDB, 83.95% on XQLFQ, and 98.74% on CFP-FP. The code for QMagFace is publicly available


Introduction
Face recognition systems are spreading worldwide and are increasingly involved in unconstrained environments [29]. In these environments, these systems have to deal with large variabilities, such as challenging illuminations, poses, and expressions, that might result in incorrect matching decisions [20] [19]. The face image quality of a sample is defined as its utility for recognition [16] [34] [5] [42] and measures the impact of these variabilities on the face recognition performance. Previous works either do not employ face image quality information during comparison [27] [44] [9] [18] or include quality estimates in comparison process 1 https://github.com/pterhoer/QMagFace that are not inherently suitable for such a task [40] [26]. While the first case results in a loss of valuable information for the comparison, in the second case, the limiting factor lies in the utilized quality estimates.
In this work, we propose QMagFace, a solution that combines a quality-aware comparison function with a face recognition model trained with a magnitude-aware angular margin (MagFace) loss. Incorporating model-specific face image qualities in the comparison process aims at enabling an improved face recognition performance even under challenging circumstances. Exploiting the linear relationship between the qualities and their comparison scores that is induced by the MagFace loss, our quality-aware comparison function is simple but effective.
In Figure 1, the effect of the proposed quality-aware scoring function is visualized. Using the standard comparison score, the genuine (same person) and imposter (different person) distributions are strongly overlapping (see top plot). Even if their respective quality distributions are very similar (right plot), combining both information with the proposed quality-aware scoring function increases the separability leading to more reliable comparison scores for matching. Especially for lower comparisons and quality scores, the proposed approach adapts the scores more strongly to increase the accuracy. For higher qualities and comparison scores, the proposed solution does not alter the score since high comparison scores imply face pairs of high quality. Consequently, QMagFace is especially effective when dealing with challenging circumstances such as crosspose or cross-age.
The experiments were conducted on four face recognition databases and six benchmarks. The results demonstrate (a) a constant improvement in the face recognition performance compared to standard comparison scores over a wide range of false match rates, (b) the suitability of the used linear function for quality weighting, and (c) a high generalizability of the proposed approach despite changes in backbone architecture, training databases, and evaluation benchmarks. Moreover, the QMagFace consistently reaches high performances in video-based recognition tasks and achieves state-of-the-art results on three of the four image-based face recognition benchmarks. Especially under challenging circumstances, such as cross-pose, cross-age, or cross-quality, QMagFace achieved high performances.  [48] consists of supervised learning algorithms based on human or artificially constructed quality labels. However, humans may not know the best characteristics for face recognition systems and artificially labelled quality values, derived from comparison scores, rely on error-prone labelling mechanisms.

Related
The third generation of FIQA approaches [42] completely avoids the use of quality labels by utilizing the face recognition networks themselves. In 2020, Terhörst et al. [42] proposed stochastic embedding robustness for FIQA (SER-FIQ). This concept measures the robustness of a face representation against dropout variations and uses this measure to determine the quality of a face. It avoids the need for training and takes into account the decision patterns of the deployed face recognition model. In 2021, Meng et al. [31] proposed a class of loss functions that include magnitude-aware angular margins encoding the quality into the face representation. Training with this loss results in a face recognition model that produces embeddings whose magnitudes can measure the FIQ of their faces.
While the first two generations of FIQA methods aim to assess the utility of an image for face recognition in general, the third generation aims at determining modelspecific quality values. Therefore, we assert that these methods, SER-FIQ [42] and MagFace [31], have the highest potential for improving the recognition performance. SER-FIQ can be applied to arbitrary face recognition networks but produce a complex relation between FIQ and comparison scores. On the other hand, MagFace produces a linear relationship between the qualities and the comparison scores, as we will show in Section 5.4, and thus, it is more suitable for a generalisable enhancement of the recognition performance.

Face Recognition
In recent years, face recognition is strongly driven by advances in deep representation learning. Early works rely on metric-learning based losses [31], such as contrastive loss [8], triplet loss [37], or penalty angular-margin losses [45] [9]. However, due to the combinatorial explosion in the number of face triplets needed for training, the research focus shifted to classification-based approaches. These approaches are typically based on softmax and aim at classifying on a closed-set of identities during training and utilize the previous layer as a feature extractor for unseen faces. Combining the softmax activation with cross-entropy loss, most face recognition losses L are of the form Here, the training set contains N samples and Θ yi refers to the angle between between last layer weight-vector and the normalized feature vector x i (with x i 2 = r). For training without margins m 1 = 1, m 2 = m 3 = 0, this refers to a simplified softmax loss. In SphereFace [27], a multiplicative angular margin is deployed with m 1 = α and m 2 = m 3 = 0. For keeping the cosine margin penalty m 2 = α (m 1 = 1 and m 3 = 0), this refers to CosFace [44] and for penalizing an angular margin m 2 = α (m 1 = 1 and m 3 = 0), this refers to the ArcFace [9] loss. However, these losses select a fixed margin α assuming that the samples are equally distributed in the embedding space around the class centers, which is not true when dealing with largely intraclass variations. To solve this problem, solutions based on variable margins are proposed. In [6], Boutros et al. proposed ElasticFace in which random margins are drawn from a Gaussian distribution N in each training iteration. This allows ElasticFace to extract and retract the margins individually for each class (e.g. m 1 = 1 and m 2 ∼ N ). Similarly, CurricularFace [18] addresses easy samples in the early training stage and hard ones in the later stage adapting m 2 while keeping m 1 = 1 and m 3 = 0. In MagFace [31], a magnitude-aware angular margin m(r) (with m 1 = 1 and m 3 = 0) with a regularization g(r) is proposed that aims at including the utility (face image quality) of a sample in the margin. While the regularization g(r) rewards samples with large magnitudes r, m(r) is a simple linear function that aims at concentrating high-quality samples in a small region around the class centers. This results in more suitable margins that are based on the utility of the samples and are encoded in the magnitude of the embeddings.

Quality-Aware Face Recognition
The qualities of face images are often used in video-tovideo recognition tasks where a set of images from one person is matched to a set of images from another [28] [33] [48] [36] [49]. There, the quality of each image is used for a weighted aggregation of information. For recognition tasks based on single images, only a few works included the face image quality to enhance the face recognition performance. In EQFace [26], Liu et al. attached a quality-prediction network on a face recognition model to include the qualities in the training process. However, this method is limited by computationally-expensive training that is not end-to-end.
In [40], Shi and Jain proposed probabilistic face embeddings (PFE). Representing face images as Gaussian distributions in the embedding space, the variance of each feature is interpreted as its uncertainty and thus, as its quality. For comparison, they make use of a mutual-likelihood score to include the quality in the comparison score. However, performance is determined by the quality estimate that is limited by (a) the used uncertainty estimation module which is trained separately from the face recognition network and (b) the assumption that each feature can be independently represented as a Gaussian Process.
In contrast to previous works, we make use of modelspecific quality estimates that were linearly included in an end-to-end fashion. Consequently, this allows our proposed solution to work simplistically and more efficiently.

Methodology
The main contribution of this work, QMagFace, combines a quality-aware comparison function with a face recognition model trained with MagFace loss [31]. Including the model-specific face image quality in the compari-son process aims to consistently improve the face recognition performance, especially under challenging conditions such as cross-pose, cross-age, or cross-quality. Moreover, the proposed quality-aware scoring function can be robustly trained on any face recognition network based on the Mag-Face loss as it will be shown in Section 5.4.

Quality-Aware Comparison Scoring
In face biometrics, a comparison score reflects the identity similarity of two face images. This reflection of similarity is more accurate when the compared samples are of high quality [16]. The biometric quality of a face image is defined as its utility for recognition [16] [34] [5] [42]. In [41], it was shown that model-specific quality assessment reflects challenging situations for the face recognition model, such as comparisons with strong variations in pose or age. Similar to SER-FIQ [42], the qualities of MagFace networks [31] utilize the deployed face recognition system and thus, strongly reflect the decision patterns and model biases. Consequently, we propose a simple, but effective, comparison function that includes these modelspecific quality values to enhance the accuracy and robustness of the face recognition system.
Given a face recognition model M trained with Mag-Face loss and two face images I 1 and I 2 , their embeddings are given by e 1 = M(I 1 ) and e 2 = M(I 2 ) together with the corresponding face image qualities q 1 = e 1 2 and q 2 = e 2 2 encoded through the vector length. The standard comparison score s = cos(e 1 , e 2 ) is defined over cosine similarity of these templates and thus, represents the angular similarity between both templates. However, comparisons with low-quality images affect the comparison scores and thus, needs to be adjusted. The proposed quality-aware comparison scoreŝ is defined aŝ with a quality-weighting function This comparison function consists of only two trainable parameters (α and β) and thus, can be robustly trained.
Since the biometric sample quality is included linearly in the MagFace loss (through m(r)), we assume a linear relationship between the face image qualities and their comparison scores. In Section 5.4, we will demonstrate the suitability of a linear function for the quality weighting. We further assume that the score-adjustment is dependent on the lowest quality of the comparison and that only comparisons with at least one low-quality sample needs to be adjusted. Please note that the qualities q 1 and q 2 are not easily exchangeable with other FIQ methods since these need to be model-specific with regard to the FR model and require a linear relationship between the quality estimates and its comparison scores.
A high similarity score s can only be achieved through the comparison of two high-quality samples. In this case, the similarity is well reflected in the comparison score s and thus, no quality-based score adjustments (ω = 0) is needed. A lower comparison score might result from the degradation of a pair with at least one low-quality sample. In this case, the similarity is altered by the sample quality and our proposed function adjusts the score based on the quality. Consequently, if two comparisons result in similar comparison scores s 1 ≈ s 2 (with ω(s) < o) but have different minimum qualities q min 1 q min 2 , the score with the higher quality undergoes a stronger adjustment and thus, results in a lower quality-aware comparison score (ŝ 1 <ŝ 2 ).

Training the Quality-Weighting Function
For training, the comparison scores of the training set S = S G + S I are separated into genuine and imposter comparisons with the corresponding minimal qualities Q min of the sample pairs. The training process is divided into three steps and aims at learning the quality weighting function ω(s) as shown in Algorithm 1.
In the first step, we define the optimal quality weight w opt (t) for a given threshold t. This is given through where Θ(·) describes the Heaviside function and is a quality-aware scoring function given by ω and s that is scaled to the range of [0, 1], similar to the range of comparison scores, with a sigmoid function. This optimization aims at minimizing the FNMR at a threshold t through including face image quality information.
In the second step, a relevant threshold range T needs to be defined that represents the target FMR range. In this work, we choose the range from FMR max = 10 −2 to FMR min = 10 −5 to cover a wide variety of potential applications and due to the amount of training data available (in the order of 10 5 images). The relevant threshold range T = [t(FMR max ), t(FMR min )] is determined by finding the threshold t that corresponds to the required FMR on the quality-aware scores on the training data. This can be determined by The third step aims to learn the quality weighting function ω(s). Since the quality is included linearly in the Mag-Face loss, a linear relationship between the importance of the quality q(s) and its comparison score s is assumed. Therefore, we model the quality weights through a simple linear function ω(s) = β * s − α. The parameters α and β can be learned by solving the following optimization min α,β t∈T resulting in the optimal parameterŝ Algorithm 1: Quality-weighting function
LFW [17] is a face verification benchmark containing 13k images of over 5k identities. In the benchmark experiments, we followed the standard protocol [17] using the 6k predefined comparison pairs. Moreover, we conducted the experiments on three more challenging benchmarks representing the issues of cross-age (AgeDB [32]), cross-pose (CFP-FP [39]), and cross-quality (XQLFW [25]). AgeDB [32] is unconstrained face recognition benchmark for ageinvariant face verification. It contains over 16k images of over 5k identities. In the experiments, we follow the protocol of AgeDB-30 since it is the most reported and challenging one for AgeDB consisting of age gaps of over 30 years. CFP-FP [39] is a face recognition benchmark that addresses the issue of comparing frontal to profile face images. In our experiments, we followed the evaluation protocol of [39] containing 3500 genuine pairs and 3500 imposter pairs. XQLFW [25] is a benchmark that addresses the problem of cross-quality comparisons in face recognition. The protocol defines 6k face image pairs based on the LFW database. However, for each pair, one face image is of much lower quality than the other face. The IARPA Janus Benchmark-B (IJB-B) [46] contains around 21k images and 55k frames from over 7k videos of 1,845 identities. In the experiment, we follow the standard evaluation protocol using around 10k genuine and 8M impostor comparisons. The IARPA Janus Benchmark-C (IJB-C) [30] extends on the IJB-B by adding more identities. In total, it consists of 31k images with over 117k frames of over 11k videos from 3531 identities. The verification protocol considers over 19k genuine and 16M imposter comparisons. In contrast to IJB-B, IJB-C focuses more on occlusion and diversity of subject occupation to improve the representation of the global population.
Besides reporting the verification accuracy based on benchmarks, we make use of four face recognition datasets to cover a much wider range of possible decision thresholds and thus, to cover more potential applications. Morph [22] consists of 55k face images from over 13k subjects. The images are frontal and of high quality. LFW [17] contains 13k images of over 5k identities and the ColorFeret database [35] consists of 14k high-resolution face images from over 1k different individuals. The data possess a variety of poses (from frontal to profile) and facial expressions under well-controlled conditions. The Adience dataset [12] consists of 26k images from over 2k different subjects. The images of the Adience dataset possess a wide range in terms of image quality. In the supplementary material, we included a more detailed discussion, such as on the licenses.

Evaluation Metrics
Following the international standard for biometric verification evaluation [21], we report the face verification error in terms of false non-match rate (FNMR) at fixed false match rate (FMR). Moreover, we report the equal error rate (EER) and the area under curve (AUC) of the receiver operating characteristic (ROC) curve. The EER equals the FMR at the threshold where FMR = FNMR and is well known In our experiments, we report the face verification performance over a wide range of FMRs to cover a variety of potential applications. On the benchmarks, we follow the mentioned protocols and report the verification accuracy to be comparable with previous works.

Face Recognition Models
In the experiments, the proposed QMagFace approach is built on three pre-trained models 2 based on MagFace loss released by the authors [31]. These were trained on the MS1MV2 database [15] and are based on iResNet-18, iResNet-50, and iResNet-100 backbones [10]. In the following, we use the name of the loss function and the model trained with it interchangeably to keep this work easily comprehensible.
The parameters α and β needed for the proposed QMag-Face approach are trained on the Adience dataset [12] due to the large quality variance in its samples and to create a generalizable approach by using the estimated parameters on this dataset. However, the training process is robust and thus, the choice of the training database only affect the performance minimally as we will show in Section 5.4. Due to the simplicity of the proposed approach, the learned parameters are shown in Table 1. To extract a face embedding from a given face image, the image is aligned, scaled, and cropped as described in [31]. Then, the preprocessed image is passed to the face recognition models to extract the feature embedding.

Investigations
The proposed approach is analysed in three steps. First, we report the performance of QMagFace on six face recognition benchmarks against ten recent state-of-the-art methods in image-and video-based recognition tasks to provide a comprehensive comparison with state-of-the-art. Second, we investigate the face recognition performance of QMagFace over a wide FMR range to show its suitability for a wide variety of applications and to demonstrate that the quality-aware comparison score constantly enhances the recognition performance. Third, we analyse the optimal quality weight over a wide threshold range to demonstrate the robustness of the training process and the generalizability of the proposed approach.

Performance on Single-Image Benchmarks
To demonstrate that the proposed QMagFace approach achieves state-of-the-art performance in image-to-image face recognition tasks, the proposed method is compared against ten recent face recognition models on four benchmarks. For PFE 3 [40], ArcFace 4 [9], and the MagFace variants [31], we used the implementations released by the authors. The remaining benchmark results are taken from [6] [26]. In Table 2, the face recognition performances of these are shown. On the LFW benchmark, the proposed QMagFace approach based on the iResNet-100 backbone achieved a performance of 99.83%, which is close to the state-of-the-art performance of 99.85%. On more challenging and less-saturated benchmarks, the proposed approach achieves state-of-the-art performance. This includes 98.50% on cross-age face recognition (AgeDB), 98.74% on cross-pose face recognition (CFP-FP), and 83.95% on cross-quality face recognition (XQLFW). Since the FIQ captures these challenging conditions and the quality values represent the utility of the images for our specific network, the proposed quality-aware comparison score can specifically address the circumstance and their effect on the network. Consequently, it performs highly accurate in the cross-age, cross-pose, and cross-quality scenarios and achieves state-of-the-art performances.

Performance on Video-Based Benchmarks
In Table 3, the video-based face recognition performance is analysed based on IJB-B and IJB-C. The FNMR is investigated over a wide range of FMRs. The performances of most state-of-the-art approaches are taken from the respective works. Since some original works did not investigate the performance on IJB-B/C, the remaining performances are taken from [31] and [6]. For creating an embedding with the corresponding quality-value for a video, the unitsized embeddings and qualities per frame are aggregated by a quality-weighted sum. Introducing quality-awareness to the MagFace-100 model generally reduces the recognition error. Especially for FMRs up to 10 −3 , QMagFace achieves state-of-the-art performance. Despite the effectiveness of QMagFace for image-to-image face recognition tasks, the quality of a video-embedding does currently not well represent its utility for recognition. While the qual- Table 2. Image-to-image face recognition performance on four benchmarks reported in terms of benchmark accuracy (%). The highest performance is marked bold. The proposed approach, QMagFace-100, achieves state-of-the-art face recognition performance, especially in cross-age (AgeDB), cross-pose (CFP-FP), and cross-quality (XQLFW) scenarios.

Model
AgeDB CFP-FP LFW XQLFW SphereFace [27] 98.17 86.84 99.67 -CosFace [44] 98 ity corresponding to an embedding for a single frame has a high correlation with the true utility of this frame, the same does not apply for the (weighted-sum) aggregated embedding and thus for the quality of the video. This needs to be addressed by future work.

Full Performance Analysis
In Table 2, it was already shown that proposed qualityaware face recognition approach leads to stable improvements in the recognition performance. In this section, we will demonstrate these improvements for a wide FMR range. Table 4 shows the recognition performance of Mag-Face and QMagFace variants. To cover a wide range of potential applications, the performance is analysed over a wide range of decision thresholds (ranging from FMR of 10 −1 to 10 −5 ) for three databases. The analysis involved over 300k/160k/1.1M comparisons on the Color-Feret/LFW/Morph database. For three backbones (iResNet-18/50/100), the performance of MagFace and the proposed QMagFace approach is compared.
For QMagFace-18, 19 out of 21 scenarios showed an improved recognition performance while for QMagFace-50, 20 out of 21 scenarios showed a performance enhancement. The three cases with a decreased performance took place on ColorFeret, which involves many challenging frontal to profile face comparisons. Since the recognition performance of Table 3. Video-based face recognition performance. The performance [%] is reported in terms of FNMR at different FMRs. The best is marked bold. The asterisk (*) denotes a method that is optimized for video-based recognition. In nearly all cases, the quality-awareness increases the performance of MagFace. For FMRs up to 10 −3 , the proposed quality-aware solution performs best despite that aggregated quality does not reflect well the combined embeddings per video. MagFace-18 and MagFace-50 is much lower than the models based on iResNet-100, the performance in estimating the model-specific face image quality correctly is lower as well. Therefore, the QMagFace approaches that make use of these quality estimates become less accurate when the quality estimate is failed by a large degree. For QMagFace-100, the performance, and thus the quality estimation, is higher. Consequently, the proposed QMagFace-100 approach leads to strong performance improvements in all investigated cases.

Robustness Analysis
Lastly, we demonstrate (a) the suitability of choosing a linear quality-weighting function and (b) the generalizability of the QMagFace solution.
In Figure 2, the correlation between the optimal quality weight and different decision thresholds are shown for different MagFace models and databases. For each of the four databases, the optimal quality weight ω opt is computed with Eq. 5 for several thresholds t ∈ T . These weights show the optimal solutions for a given database that can be achieved by using the quality-aware score functionŝ(s, q 1 , q 2 ) from Eq. 3. Moreover, a linear function is fitted through these points and shown in the same color.
For all three models (Figure 2a, 2b, 2c), two observations are made. First, the optimal quality weights ω opt follow a linear function with respect to the decision threshold t. This is observed for all MagFace models and on each database, proving the suitability of our linear quality weight function ω(s) from Eq. 4. Second, for each model, the optimal quality weight functions are similar. The only exception is the quality weight function for MagFace-100 that is optimized on LFW. In this case, the database turns out to be too easy to train the model effectively since even for an FMR of 10 −5 the decision threshold is below 0.5. However, in all the other investigated cases, the relation between the optimal quality weights and the decision thresholds is similarly independent of the analysed database. Utilising these databases for training, QMagFace will lead to similar matching decisions demonstrating the robustness of the QMagFace training process. Moreover, it indicates a high generalizability since the learned function on one database is very similar to the optimal functions of the others.

Limitations and Ethical Considerations
Despite the high generalizability and the effectiveness of QMagFace for unconstrained face recognition, the approach has to deal with two limiting factors. First, the additional quality information is most beneficial for images of lower quality and thus, the performance improvements of QMag-Face decrease for very low FMRs, such as 10 −7 . Second, for more effective video-based recognition, a more suitable quality aggregation is needed. For single images, the quality of an embedding well reflects its utility for recognition. However, this does not apply when fusing the frames of a video with the corresponding qualities. To more efficiently exploit the quality information in video-based recognition with QMagFace, future works need to focus on more advanced quality-based fusion techniques for video frames.
While the proposed quality-awareness approach might strongly improve unconstrained face recognition for the sake of higher security or convenience, we want to point out the importance of unbiased quality estimates for fair face recognition. The use of biased quality estimates might lead  to unfair, and thus discriminatory, matching decisions depending on demographic and non-demographic factors of their users [43].

Conclusion
For recognition in unconstrained environments, FR systems have to deal with challenging situations, such as different illuminations, poses, and expressions. Previous works either focused on learning margin-based approaches while not considering FIQ information or included noninherently fit quality estimates. In this work, we proposed QMagFace, a simple and robust quality-aware FR approach. It integrates model-specific FIQ information in the comparison process to allow a more accurate performance under challenging situations, such as cross-pose or cross-age. The experiments were conducted on ten FR databases and benchmarks. The results demonstrated that including the quality-awareness consistently increases the FR performance. Moreover, it was shown that QMag-Face reaches competitive recognition results with stateof-the-art solutions. For challenging circumstances, such as cross-pose, cross-age, or cross-quality, QMagFace constantly beat state-of-the-art approaches. Additional experiments indicated a high generalizability of the proposed approach demonstrated the suitability of a linear function for the quality weighting.