{'fraction_non_alphanumeric': 0.03305101437857002, 'fraction_numerical': 0.012290722867835335, 'mean_word_length': 4.8859262694180385, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Domain Balancing: Face Recognition on Long-Tailed Domains

Long-tailed problem has been an important topic in face recognition task. However, existing methods only concentrate on the long-tailed distribution of classes. Differently, we devote to the long-tailed domain distribution problem, which refers to the fact that a small number of domains frequently appear while other domains far less existing. The key challenge of the problem is that domain labels are too complicated (related to race, age, pose, illumination, etc.) and inaccessible in real applications. In this paper, we propose a novel Domain Balancing (DB) mechanism to handle this problem. Specifically, we first propose a Domain Frequency Indicator (DFI) to judge whether a sample is from head domains or tail domains. Secondly, we formulate a light-weighted Residual Balancing Mapping (RBM) block to balance the domain distribution by adjusting the network according to DFI. Finally, we propose a Domain Balancing Margin (DBM) in the loss function to further optimize the feature space of the tail domains to improve generalization. Extensive analysis and experiments on several face recognition benchmarks demonstrate that the proposed method effectively enhances the generalization capacities and achieves superior performance.


Introduction
Feature descriptor is of crucial importance to the performance of face recognition, where the training and testing images are drawn from different identities and the distance metric is directly acted on the features to determine whether they belong to the same identity or not. Recent years have witnessed remarkable progresses in face recognition, with a variety of approaches proposed in the literatures and applied in real applications [18,32,4,7,6,42]. Although yielding excellent success, face recognition often suffers from poor Figure 1. The long-tailed domain distribution demarcated by the mixed attributions (e.g., race and age) in the MS-Celeb-1M [8] and CASIA-Webface [36]. Number of classes per domain falls drastically, and only few domains have abundant classes. (Baidu API [1] is used to estimate the race and age) generalization, i.e., the learned features only work well on the domain the same as the training set and perform poorly on the unseen domains. This is one of the most critical issues for face recognition in the wild, partially due to the non-negligible domain shift from the training set to the deployment environment.
Real-world visual data inherently follows a long-tailed distribution, where only a limited number of classes appear frequently, and most of the others remain relatively rare. In this paper, we aim to investigate the long-tailed domain distribution and balance it to improve the generalization capacity of deep models. However, different from the long-tailed problem in classes, domain labels are inaccessible in most of applications. Specifically, domain is an abstract attribute related to many aspects, e.g., age (baby, child, young man, aged, etc), race (caucasian, indian, asian, african, etc.), expression (happy, angry, surprise, etc.), pose (front, profile, etc.), etc. As a result, the domain information is hard to label or even describe. Without the domain label, it is difficult to judge whether a sample belongs to the head domains or the tail domains, making existing methods inapplicable. Figure 1 illustrates a possible partition by the mixed attributions (e.g., race and age).
We formally study this long-tailed domain distribution Figure 2. The face features are trivially grouped together according to different attributions, visualized by t-SNE [20] problem arising in real-world data settings. Empirically, the feature learning process will be significantly dominated by those few head domains while ignoring many other tail domains, which increases the recognition difficulty in the tail domains. Such undesirable bias property poses a significant challenge for face recognition systems, which are not restricted to any specific domain. Therefore, it is necessary to enhance the face recognition performance regardless of domains. An intuitive method to handle the long-tailed problem is over-sampling and under-sampling samples on the tail and the head, respectively [10,27,41]. However, it does not work well on domain balancing since the the ground-truth domain distribution is inaccessible. To overcome this drawback, we propose a Domain Balancing (DB) mechanism to balance the long-tailed domain distribution. Firstly, since the ground truth domain distributions are inaccessible without domain labels, for each sample, we should predict where the belonged domain locates on the distribution. To this end, we point out that the domain frequency can be instructed by the inter-class compactness. In the holistic feature space, the classes with similar attributes tend to group, forming a specific domain as shown in Figure 2. Besides, in the feature space, the compactness is not everywhere equal. Take the domains in Figure 1 as an example, we find the compact regions tend to belong to the head domains (e.g., caucasian male), and the sparse regions tend to belong to the tail domains (e.g., children, african female). The detailed analysis will be shown in Section 3.1. Motivated by these observations, we propose to utilize the inter-class compactness which is the local distances within a local region as the Domain Frequency Indicator (DFI). Secondly, considering the samples belong to the same do-main share some appearance consistency, we design a novel module called Residual Balancing Mapping (RBM) block, which can adaptively change the network based on DFI to find the best network to adjust each domain. The block consists of two components: a domain enhancement branch and a soft gate. The domain enhancement branch aims to adjust the network to each domain through enhancement residual and the soft gate attaches a harmonizing parameter to the residual to control the amount of residual according to the domain frequency. Thirdly, in the loss function, we propose a Domain Balancing Margin (DBM) to adaptively modify the margin according to the DFI for each class, so that the loss produced by the tail domain classes can be relatively up-weighted. The framework is shown in Figure 3.
The major contributions can be summarized as follows: • We highlight the challenging long-tailed domain problem, where we must balance the domain distribution without any domain annotation.
• We propose a Domain Balancing (DB) mechanism to solve the long-tailed domain distribution problem. The DB can automatically evaluate the domain frequency of each class with a Domain Frequency Indicator (DFI) and adapt the network and loss function with Residual Balancing Mapping (RBM) and Domain Balancing Margin (DBM), respectively.
• We evaluate our method on several large-scale face datasets. Experimental results show that the proposed Domain Balancing can efficiently mitigate the longtailed domain distribution problem and outperforms the state-of-the-art approaches.

Related Works
Softmax based Face Recognition. Deep convolutional neural networks (CNNs) [3] have achieved impressive success in face recognition. The current prevailing softmax loss considers the training process as a N-way classification problem. Sun et al. [28] propose the DeepID for face verification. In the training process, for each sample, the extracted feature is taken to calculate the dot products with all the class-specific weights. Wen et al. [35] propose a new center loss penalizing the distances between the features and their corresponding class centers. Wang et al. [31] study the effect of normalization during training and show that optimizing cosine similarity (cosinebased softmax loss) instead of inner-product improves the performance. Recently, a variety of margin based softmax losses [18,32,4] have achieved the state-of-the-art performances. SphereFace [18] adds an extra angular margin to attain shaper decision boundary of the original softmax loss. It concentrates the features in a sphere mainfold. CosFace [32] shares a similar idea which encourages the intra-compactness in the cosine manifold. Another effort ArcFace [4] uses an additive angular margin, leading to similar effect. However, these efforts only consider the intra-compactness. RegularFace [38] proposes an exclusive regularization to focus on the inter-separability. These methods mainly devote to enlarge the inter-differences and reduce the intra-variations. Despite their excellent performance on face recognition, they rely more on the large and balanced datasets and often suffer performance degradation when facing with the long-tailed data.
Long − tailed Learning Long-tailed distribution of data has been well studied in [37,19]. Most existing methods define the long-tailed distribution in term of the size of each class. A widespread method is to resample and rebalance training data, either by under-sampling examples from the head data [10], or over-sampling samples from the rare data more frequently [27,41]. The former generally loses critical information in the head sets, whereas the latter generates redundancy and may easily encounter the problem of over-fitting to the rare classes. Some recent strategies include hard negative mining [5,15], metric learning [12,23] and meta learning [9,34]. The range loss [37] proposes an extra range constraint jointly with the softmax loss. It reduces the k greatest intra-class ranges and enlarges the shortest inter-class distance within one batch. The focal loss [15] employs an online version of hard negative mining. Liu et al. [19] investigate the long-tailed problem in the open set. Its so-called dynamic meta-embedding uses an associated memory to enhance the representation. Adaptiveface [16] analyzes the difference between rich and poor classes and proposes the adaptive margin softmax to dynamically modify the margins for different classes. Although the longtailed problem has been well studied, they are mainly based on the category frequency distribution. None of previous works consider the similar problem in domain. One possi- ble reason may be due to the ambiguous domain partition as discussed above. In fact, the domains may not even have explicit semantics, i.e., they are actually data-driven.
In contrast, our method focuses on the long-tailed domain, which is more in line with the real-world application. The proposed method balances the contribution of domains on the basis of their frequency distribution, so that it can improve the poor generalization well.

Domain Balancing
We propose to balance the samples from different domains without any domain annotation. Domain Balancing (DB) mechanism has three components: Domain Frequency Indicator (DFI) to evaluate the domain frequency, the Residual Balancing Mapping (RBM) to adjust the network and the Domain Balancing Margin (DBM) to adjust the loss functions according to domain distribution.

Domain Frequency Indicator
To handle the long-tailed domain distribution problem, we first need to know whether a sample is from a head domain or from a tail domain. We introduce a Domain Frequency Indicator (DFI) based on the inter-class compactness. Inter-class compactness function of a given class is formulated as: where w is the prototype of one class in the classification layer and k is the k-th nearest class, where the distance of two classes i, j is formulated as cos(w i , w j ). The high frequency domain, i.e., head domain, usually corresponds to a large IC(w), and vice versa. Then we define the Domain Frequency Indicator as: which is inversely proportional to the inter-class compactness IC(w) and ε is a constant value. Ideally, if the classes are uniformly distributed, each class will have the same DFI. Otherwise, the classes with larger DFI are more likely to come from a tail domain and should be relatively upweighted. As shown in Figure 4, the identities with larger DFI values usually come from Africa, children or the aged, which are highly related with the tail domains.

Residual Balancing Module
In real-world application, face recognition accuracy depends heavily on the quality of the top-level feature x. The goal in this section is to design a light-weight solution to adjust the network to extract domain specific features according to the domain distribution. Our Residual Balancing Module (RBM) combines the top-level image feature and a residual feature, using DFI to harmonize the magnitude. Even though big training data facilitates the feature discriminative power, the head domains dominate the learning process and the model lacks adequate supervised updates from the tail classes. We hope to learn a harmonizing module through a mapping function M re (.) to adjust the features for samples of different domain frequency to mitigate the domain imbalance problem. We formulate M re as a sum of the original feature x and residual acquired by a feature enhancement module R(x) weighted by f (x). We denote the resulting feature as x re and the RRM can be formulated as: where x is the top-level feature, f (x) is a soft gate depending on the DFI. When DFI is large, the input feature probably belongs to a tail class, and a large enhancement is assigned to the residual. Otherwise, the enhancement is trivial. The magnitude of residual is thus inversely proportional to the domain frequency. The combination of the soft gate and and the residual can be regarded as a harmonizing mechanism that adopts domain distribution information to control the magnitude to be passed to the next layer. We now describe the implementation of the two components: The first component is the residual R(x), which is implemented by a light-weighted full-connected layer. It consists of two full-connected layers and a batch norm layer shown in Figure 5. The second component is the soft gate coefficient DFI, which is learned from the feature x and supervised by the DFI. For simplicity, the linear regression is employed by the L2 loss: where DFI(x) is defined in Eq. 2 from the last iteration. f (x) is a mapping function devoting to associate the representation x and DFI.

Domain Balancing Margin
We propose a domain-aware loss by Domain Balancing Margin (DBM) to adaptively strengthen the classes in the tail domains. Specifically, we formulate the DBM loss by embedding the DFI into CosFace as: where β y i = DFI y i and m is a fixed parameter as defined in CosFace. Figure 6 visualizes the phenomenon through a trinary classification. The main difference between DBM and CosFace is that our margin is dynamic and feature compactness related. For the CosFace, the decision boundary assigns the same margin without considering the feature compactness. It cannot efficiently compact the feature space of the tailed-domain class C 2 since the sparse interclass distribution makes the decision boundary easy to satisfy. The termination of optimization is so early, leading to poor generalization. In contrast, our DBM drives adaptive decision boundary in terms of the inter-compactness, where margin 2 (tailed-domain margin) should be much larger than margin 1 (head-domain margin). Consequently, both the inter-separability and the intra-compactness can be guaranteed.
We combine the mentioned L dbm and L rrm by a parameter λ . The final loss function can be formulated as: 4. Experiments

Datasets
Training Set. We employ CASIA-Webface [36] and MS-Celeb-1M [8] as our training sets. CASIA-WebFace is collected from the web. The face images are collected from various professions and suffer from large variations in illumination, age and pose. MS-Celeb-1M is one of the largest real-world face datasets containing 98,685 celebrities and 10 million images. Considering the amount of noise, we use a refined version called MS1MV2 [4] where a lot of manual annotations are employed to guarantee the quality of the dataset.
Testing Set. During testing, we firstly explore databases (RFW [33], AFW [2]) with obvious domain bias to check the improvement. RFW is a popular benchmark for racial bias testing, which contains four subsets, Caucasian, Asian, India and African. Moreover, we collect a new dataset from CACD [2], called Age Face in-the-Wild (AFW). We construct three testing subsets, Young (14-30 years old), Middle-aged (31-60 years old) and Aged (60-90 years old). Each subset contains 3,000 positive pairs and 3,000 negative pairs respectively. Besides, we further report the performance on several widely used benchmarks including LFW [13], CALFW [40], CPLFW [39] and AgeDB [21]. LFW contains color face images from 5,749 different persons in the web. We verify the performance on 6,000 image pairs following the standard protocol of unrestricted with labeled outside data. CALFW is collected with obvious age gap to add aging process intra-variance on the Internet. Similarly, CPLFW is collected in terms of pose difference. AgeDB contains face images from 3 to 101 years old. We use the most challenging subset AgeDB-30 in the following experiments. We also extensively evaluate our proposed method on large-scale face dataset, MegaFace [14]. MegaFace is one of the most challenging benchmark for large scale face identification and verification. The gallery set in MegaFace includes 1M samples from 690K individuals and the probe set contains more than 100K images of 530 different individuals from FaceScrub [22]. Table 1 shows the detailed information of the involved datasets. 2,000 160,000 MegaFace [14] 530 (P) 1M(G)

Experimental Settings
For data prepossessing, the face images are resized to 112 × 112 by employing five facial points, and each pixel in RGB images is normalized by subtracting 127.5 and dividing by 128. For all the training data, only horizontal flipping is used for data augmentation. For the embedding neural network, we employ the widely used CNNs architectures, ResNet18 and ResNet50 [11]. They both contain four residual blocks and finally produce a 512-dimension feature.
In all the experiments, the CNNs models are trained with stochastic gradient descents (SGD). We set the weight decay of 0.0005 and the momentum of 0.9. The initial learning rate starts from 0.1 and is divided by 10 at the 5, 8, 11 epochs. The training process is finished at 15-th epoch. We set ε = 5.5 and λ = 0.01 in all the experiments. The experiments are implemented by PyTorch [25] on NVIDIA Tesla V100 (32G). We train the CNNs models from scratch and only keep the feature embedding part without the final fully connected layer (512-D) during testing.  We use cosine distance to calculate the similarity. For the performance evaluation, we follow the standard protocol of unrestricted with labeled outside data [13] to report the performance on LFW, CALFW, CPLFW, AgeDB, RFW and AFW. Considering the well solved on LFW, we further use the more challenging LFW BLUFR protocol to evaluate the proposed method. On MegaFace, there are two challenges. We use large protocol in Challenge 1 to evaluate the performance of our approach. For the fair comparison, we also clean the noisy images in Face Scrub and MegaFace by the noisy list [4].
To the compared approaches, we compare the proposed method with the baseline Softmax loss and the recently popular state-of-the-arts, including SphereFace [18], CosFace [32] and ArcFace [4].

Ablation Study
In this section, we investigate the effectiveness of each balancing module in the proposed method.
Effectiveness of the RBM. Recall that the RBM module consists of two main components: the residual enhancement and the soft gate. The soft gate produces a harmonizing coefficient to automatically control the magnitude of the residual attached to the top feature. When the soft gate is closed, i.e, f (x) = 1 is constant for all samples, the RBM module degenerates to a conventional residual that loses the ability of distinguishing the head and tail domains. From Table 2, we observe that the combination of the residual en-hancement and the soft gate brings large improvements on all the datasets. The average performance of LFW, CALFW, CPLFW, AgeDB has been improved from 91.35 to 92.12. It is because RBM actually harmonizes the potential feature bias among different domains.
Effectiveness of the Soft Gate. The soft gate produces the coefficient DFI to control the magnitude of residual added on the original feature. In this experiment we analyze the effectiveness of the soft gate. As displayed in Table 2, the performance drops significantly without the soft gate. The average accuracy decreases 0.87%. These results suggest that the improvement attained by the RBM block is not mainly due to the additional parameters, but its internal domain balancing mechanism.
Effectiveness of the DBM. We further validate the effectiveness of DBM that whether it can improve the poor generalization caused by the long-tailed domain distribution. From the first row of each sub-boxes in Table 2, we can find that DBM boosts the performance on all the datasets. The average performance is stably improved compared to the baselines, presenting its contribution to mitigate the potential imbalance. Particularly, DBM achieves about 0.48% average improvement over RBM, which indicates that balancing the contribution from different domains through loss function can better address the problem.

Exploratory Experiments
We first investigate how our method improves the performance on the different domains with different domain frequency. We train Resnet50 on CASIA-Webface by Cos-Face and our method. Figure 7 shows the performances on different domains on two datasets. Firstly, for the Cosface, the accuracy of Caucasian on RFW is significantly higher than other races, and Asian gains the worse performance. Besides, on AFW, the Young subset acquires the highest accuracy while the performance on the aged persons degrades heavily. The performance decay confirms our thought that the popular methods is susceptible to the long-tailed domain distribution. Secondly, our method consistently improves the performance on almost all the domains. Particularly, the accuracy increases more obviously on the tail domains, such as the Asian on RFW and [60, 90) aged persons on AFW, which indicates that the proposed method can alleviate the potential imbalance cross domains.
The nearest neighbor parameter K in Eq. 1 plays an important role in DFI. In this part we conduct an experiment to analyze the effect of K. We use CASIA-WebFace and ResNet18 to train the model with our method and evaluate the performance on the LFW, CALFW and CPLFW as presented in Table 3. We can conclude that the model without DFI suffers from the poor performances on all these three benchmarks. The model attains the worst result on all the datasets when K = 0, where the model degenerates into the original form without balancing representation and margin supplied by RBM and DBM. The model obtains the highest accuracy at K = 100. However, when K keeps increasing, the performances decrease to some extent because a too large K covers a too large region with sufficient samples and weakens the difference between head and tail domain.

Results on LFW and LFW BLUFR
LFW is the most widely used benchmark for unconstrained face recognition. We use the common larget dataset MSIMV2 to train a ResNet50. Table 4 displays the the comparsion of all the methods on LFW testset. The proposed method improves the performance from 99.62% to 99.78%. Further, we evaluate our method on the more challenge LFW BLUFR protocol. The results are reported in Table 5. Despite the limited improvement, our approach still achieves the best results compared to the state-of-thearts. Table 6 shows the performances on CALFW, CPLFW and AgeDB, respectively. We also use MSIMV2 to train the ResNet50. The results show the similar treads that emerged on the previous test sets. Particularly, the margin-based methods attain better results than the simple softmax loss for face recognition. Our proposed method, containing efficient domain balancing mechanism, outperforms all the other methods on these three datasets. Specifically, our method achieves 95.54% average accuracy, about 0.4% average improvement over ArcFace.

Results on MegaFace
We also evaluate our method on the large Megaface testset. Table 7 displays the identification and verification performances. In particular, the proposed method surpasses the best approach ArcFace by an obvious margin (about 0.82% at Rank-1 identification rate and 0.68% verification rate). The reason behind may be that the proposed balancing strategy can efficiently mitigate the potential impact of the longtailed domain distribution, which is ubiquitous in the realworld application.

Conclusion
In this paper, we investigate a novel long-tailed domain problem in the real-world face recognition, which refers to few common domains and many more rare do- Table 7. Face identification and verification on MegaFace Chal-lenge1. "Rank 1" refers to the rank-1 face identification accuracy, and "Ver" refers to the face verification TAR at 10 −6 FAR. DBM assigns an adaptive margin to balance the contribution from different domains. Extensive analyses and experiments on several face recognition benchmarks demonstrate that the proposed method can effectively enhance the discrimination and achieve superior accuracy.