{'fraction_non_alphanumeric': 0.027888446215139442, 'fraction_numerical': 0.010151700888752682, 'mean_word_length': 4.966857142857143, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Understanding Unequal Gender Classification Accuracy from Face Images

Recent work shows unequal performance of commercial face classification services in the gender classification task across intersectional groups defined by skin type and gender. Accuracy on dark-skinned females is significantly worse than on any other group. In this paper, we conduct several analyses to try to uncover the reason for this gap. The main finding, perhaps surprisingly, is that skin type is not the driver. This conclusion is reached via stability experiments that vary an image's skin type via color-theoretic methods, namely luminance mode-shift and optimal transport. A second suspect, hair length, is also shown not to be the driver via experiments on face images cropped to exclude the hair. Finally, using contrastive post-hoc explanation techniques for neural networks, we bring forth evidence suggesting that differences in lip, eye and cheek structure across ethnicity lead to the differences. Further, lip and eye makeup are seen as strong predictors for a female face, which is a troubling propagation of a gender stereotype.


Introduction
The problem of unequal accuracy rates across groups has recently been highlighted in gender classification from face images. A study by NIST shows that automated gender classification algorithms are more accurate for males than females [24]. Going further, Buolamwini and Gebru created a dataset of parliament members from three European and three African countries -the Pilot Parliaments Benchmark (PPB), balanced across two attributes: gender and Fitzpatrick skin type [14], and evaluated the accuracy of three commercial facial gender classifiers [4]. All three achieved much lower accuracy on dark-skinned females (Fitzpatrick skin types IV-VI) than light-skinned females, dark-skinned males, and light-skinned males. (Note that gender classification is a distinct task from race classification [15].) The discrepancy is conjectured to be largely due to imbalanced training datasets and test benchmarks. Commonly used training datasets such as CelebA [21] and IMDb face [29] are made up of celebrities and biased towards light-skinned people. Test benchmarks such as Labeled Faces in the Wild [17] and Adience [11] are also biased [4], so high overall accuracies achieved on these test datasets obfuscate the inequality issue. The IJB-A dataset purports to be geographically diverse [19], but a close examination reveals that only 8 percent of the faces are of African descent, whereas more than 50 percent of the faces are of European descent. The PPB dataset is the first of its kind to be balanced by gender and balanced between African and European descent [4].
These works, however, do not investigate the underlying causes of the unequal misclassification rates in gender classification. In particular, since the partition in [4] is phenotypic into different skin type categories but the darkskinned people are predominantly of African descent, it may be that other features, such as hairstyle, facial structure, cosmetics or clothing are the reason for disparity, rather than skin type alone [5]. A study of unequal gender classification accuracy, conducted using images with different parts of the face masked out, points to the nose region as important, but does little to disentangle the various aspects of identity [27]. Buolamwini points to several shortcomings of that study and calls for "further scholarship that attends to the impact of phenotypic characteristic on gender classification that extends beyond skin type" [5].
Heeding this call, we rigorously analyze gender classi- fiers and test the extent to which various features influence the classification outcome by skin type and gender. The contributions of this paper are as follows.
• Using principles from color theory and the framework of optimal transport, we test stability to skin type by varying the skin type of a face keeping all other features fixed, and statistically show that the effect of skin tone on classification outcome is minimal. Thus, the unequal accuracy observed in [4] likely arises not specifically because of the skin type, but other correlated features of identity [1].
• Motivated by a visual observation that most misclassified dark-skinned females have short hair, we test the significance of hair patterns in gender classification. We find that ignoring hair information retains high overall accuracy as well as differential performance, suggesting that information about the hair is also unimportant.
• Finally, we use recently proposed ideas on contrastive explanation [10] to show that neural networks used for gender classification latch on to various facial features like lips, cheeks, and eyes with cosmetics as sufficient explanations for the female gender -suggesting that discrepancies in these features are the root of the inequality observed in intersectionally-defined groups.

Pilot Parliaments Benchmark dataset
The PPB dataset is the first benchmark dataset that is balanced across gender and Fitzpatrick skin type; the methodology of its collection is detailed in [4]. The creators intentionally chose countries with majority populations at opposite ends of the skin type scale to make the lighter/darker dichotomy more distinct. The images are uniform in (high) resolution quality, pose, illumination and expression, reducing the possibility of attributing differences in performance to variations in these quantities, all of which are known to be significant technical challenges [32]. We use an approximation of the PPB dataset for the experiments in this paper. This dataset contains images of parliament members from the six countries identified in [4] and were manually labeled by us into the categories dark-skinned and light-skinned. 1 Our approximation to the PPB dataset, which we call PPB*, is very similar to PPB and satisfies the relevant characteristics for the study we perform. Table 1 compares the decomposition of the original PPB dataset and our PPB* approximation according to skin type and gender.

Classification models
We employ several classifiers in our experiments. The first classifier is the IBM Watson gender classifier service available in August 2018, which achieves 99% accuracy on several test benchmarks, as well as 99% accuracy on the light male, light female and dark male groups of the PPB* dataset. We access the gender classifier using the API, which takes in an input image of variable size and returns (in the event that a face is detected) a score s ∈ [0, 1] that the image is of a male person. Values s ≤ 0.5 are classified female and values s > 0.5 are classified male. Accuracies on the PPB* dataset are presented in Table 2. The accessibility of scores from the IBM Watson API as well as its high level of performance make this a good classifier for carrying out stability experiments. (Scores are not available from the other two commercial classifiers studied in [4].) The second and third classifier are needed for studying face-only cropped images (i.e. cropped so that no hair is present) because the commercial classifier API does not offer the flexibility to restrict modeling to smaller cropped areas. As a second classifier, we use IBM Watson's "deepface-features" API to extract a 256-dimensional representation for every face-cropped image. We train a downstream support vector machine (SVM) classifier with radial basis function (RBF) kernel using 20000 images from the CelebA dataset for training and 5000 validation images to choose the RBF kernel parameter. The third classifier is the same as the second in terms of the SVM and its training, but has a more modern feature extractor: a convolutional neural network (CNN) trained on the recently created VGGFace2 dataset [7]. In particular, we use the ResNet-50 network to extract a 128-dimensional   representation for every face-cropped image. 2 The fourth classifier is needed because unfortunately, apart from the scores, we only have black-box access to the Watson API. The details of the model architecture are not available and we cannot inspect any intermediate layers. For interpretability experiments, we create a customized classifier: a simple CNN described in Figure 1. We train this neural network on CelebA [21] (162, 768 training images). Before input to the neural network, all images are face-cropped with padding, eye-aligned and resized to 128 × 128. Accuracies on the PPB* dataset for the customized model are presented in Table 2. While they are not as good as the Watson classifier, especially on females, they do achieve close to state-of-the-art accuracy on males, and serve as a model that can be further interrogated.

Experiments
The first set of experiments tests the stability of gender classification algorithms to variation in skin type. Next, we test the influence of hair length, by seeing whether the unequal performance persists when we remove all information about this attribute from faces. Finally, we seek sufficient explanations on faces for the classification decisions of female and male respectively.

Stability experiment: Does skin type
alone influence gender classification?
In the first set of experiments, we systematically isolate the skin type and test the gender classification outcome 2 https://github.com/ox-vgg/vgg_face2 for significant changes as a function of varying skin type. Isolating a latent facial attribute, and thus changing it, is in general known to be a challenging computer vision task. Likelihood based generative models [18] and conditional generative adversarial networks (GANs) [8,28] have made recent progress in varying attributes like hair color and facial expressions. However, these tools themselves are trained on biased celebrity datasets. Moreover, these approaches are not effective in varying one attribute in isolation, leaving other attributes unchanged. We empirically show the existence of an approximately low-dimensional structure in color space that describes the group of human skin types. Leveraging this structure, we provide simple but mathematically grounded rules to change the skin type of a face.

A low-dimensional skin type group in YCrCb space
Recall that image pixels can be represented in the 3dimensional vector space [0, 255] 3 . Multiple bases for the color space such as the standard RGB [12], HSV [26] and YCrCb [16], have been used to create skin detection rules. More recently, hybrid rules have been proposed that work under complex lighting conditions [25,22]. We use the following skin detection rule based on the YCrCb space [16], where Y stands for luminance and Cr, Cb stand for chrominance values.
We employ this rule for its simplicity and fairly good performance in skin detection under the favorable lighting conditions of the PPB* dataset. We also plot histograms of the YCrCb values of skin type pixels detected for each face image and observe that the Cr and Cb values fall into an even narrower range than described in (1). As the illustration in Figure 2 depicts, the chrominance values do not appear different for individuals with light or dark skin type. More rigorously, Figure 3 plots the histogram of Cr and Cb values across all 1204 images in PPB*; we observe that the chrominance values are stable. Practically all the variation in skin type is captured by the Y channel alone. 3

Methods to change the skin type
Based on the low-dimensional structure described in the previous subsection, we describe two rules that we em- Procedure 1 (Luminance mode-shift) We shift the skin type luminance mode of an image in the following sequence of steps: 3. Shift the luminance values, i.e. I Y = I Y + δ.

Clip luminance values to [0, 255].
Procedure 1 is attractive for its simplicity and quick computation (O(1) time), but the results of skin type change according to luminance mode shift are not always visually attractive, as demonstrated in Figure 4. Perhaps the luminance mode of skin type pixels is not sufficiently descriptive, and we would rather consider a transform between skin type histograms. Motivated by this, we next consider a skin type operation based on optimal transport, which has recently shown to be effective in color transfer in RGB space [13].
Procedure 2 (Optimal transport [13]) This procedure takes as input a target skin type distribution over Y values. We denote the skin type distribution of a grayscale image by µ(I) and the target skin type distribution by µ Y . Then, the optimally transported image is defined as follows: Figure 5 shows that the results of optimally transported skin type are visually more realistic. However, the computational cost of using this operation is more; the optimal transport operation has complexity O((w × h) 3 ). 4

Results
We consider the following ensemble of skin-type changes on the PPB* dataset: The set of darkened/lightened mode-shifted images represents all luminance-mode-shifts with negative/positive δ. Owing to the computational expense of optimal transport, we pick ten images on varying ends of the skin type spectrum. Figure 6 shows the distribution of affected differences in prediction on lightening the set of dark females in the     PPB dataset, either using mode-shift (Figure 6a) or optimal transport (Figure 6b). 5 We observe that most images' scores do not change meaningfully after lightening/darkening. In the case of dark females, 86.6% of the images' scores do not change by more than 0.1 on lightening using mode-shift. 76.6% of the images' scores do not change by more than 0.1 on lightening using optimal transport. In the case of light females, 96.3% of the images' scores do not change by more than 0.1 on darkening using mode-shift. 92.1% of the images' scores do not change by more than 0.1 on darkening using optimal transport. We conducted one-sample t-tests to test the null hypothesis that the mean of differences in scores is equal to 0. The results in terms of the 95% confidence intervals are presented in Table 3. Figures 8 and 9 shed insight into the relative difference in predictions, which also matters -in particular, we may care about the fraction of images whose average classification decision would change after lightening/darkening. 5 The quality of the experiment itself is better with the optimal transport method as the lightened images are more realistic, but owing to computational complexity of optimal transport, we also have fewer lightened samples to average over. On the other hand, the mode-shift operation generates images that are not as realistic, but the experiment itself is statistically more robust as we can quickly generate many lightened samples. Thus, observing similar conclusions for the two methods strengthens our result.    In the scatterplots of original score vs. score after change in skin type (Figures 8b, 8c, 9b and 9c), we highlight the points that fall in the red-shaded region as representing dark females that are correctly classified only after lightening, or light females that are incorrectly classified only after darkening. Very few images fall into these categories: 5 and 9 dark females (out of 212) are correctly classified only after lightening using mode-shift and optimal transport respectively. The effect of darkening is even less pronounced for light females -after mode-shift and optimal transport, 0 and 2 females (out of 296) respectively become incorrectly classified. Looking at the distribution on original scores of dark and light females (Figures 6 and 7), we see that almost all light females are classified as female with extremely high score, and almost all dark females are classified as either female or male with extremely high score. The dark females that are classified as male with extremely high score, say above 0.9, do not change significantly in score or classification decision on lightening.
All of these results, together, lead us to conclude that the skin type by itself has a minimal effect on classification decisions.

The potential influence of hair length
If the skin type is not an underrepresented facial feature that matters, then what is? We visually observed that most of the dark females that were misclassified as male by the Watson classifier had short hair. We manually labeled all the dark and light females in the PPB* dataset as either short or long-haired, and considered the intersectional performance of the Watson classifier on females across skin type and hair length. 6 The results are presented in Table 4. We notice a meaningful split in performance across hair length, especially for dark females: the classification accuracy on dark females with short hair is only 75%, as opposed to a much higher 92% on dark females with long hair. While the difference is less pronounced for light females, the accuracy on short-haired light females is also lower at 90%. We also observed a relatively higher proportion of short-haired dark females in the PPB* dataset: 7 72% of the dark females were observed to be short-haired, as opposed to 46% of the light females. Looking at purely misclassified dark females, 29 out of 33 were short-haired! A hypothesis to explain the unequal performance displayed above is that the neural networks latch on to hair length as a significant predictor for gender. It is wellknown in human visual perception that certain hairstyles (including male facial hair) are convincingly attributed to respective (binary) genders [3]. Such simplistic explana- 6 We did not consider a similar split across males, because all males in the PPB* dataset are short-haired. 7 We are not making this claim for the general population of lightskinned and dark-skinned females, only the ones in the PPB* dataset. tions can lead not only to racial biases but also confirmations of gender stereotypes.
We do not replicate the stability experiment for hair patterns because it is challenging to develop methodology to change a face's hairstyle while keeping all other attributes unchanged. 8 We can do a different sort of experiment, however -we can consider the performance of state-ofthe-art gender classifiers that completely ignore information about the hair. This is achieved by using cropped facial images as input obtained via standard face detectors in computer vision [34]. By definition, these input images will contain only facial information, and not information about hair patterns ( Figure 10 contains some example images).
We achieve intersectional accuracies on the PPB* dataset reported in Table 5 and Table 6 using SVMs with Watson deep face features and ResNet-50 deep face features with different kernel parameters on only-face cropped images, . We observe that the accuracies are greater than 90% for males and relatively high for light females (80%), but only 66% for dark females with the Watson features. Similarly, but at a higher overall accuracy level, the accuracies are greater than 95% for males and light-skinned females, but only around 80% for darkskinned females. Thus, the unequal performance persists

Sufficient facial features
What do gender classifiers look at, if not skin type and hair length? One reason why this sort of question is so challenging to answer is the high dimensionality of facial images, but another reason is the complexity of ML models. State-of-the-art gender classifiers typically use neural networks, whose interpretability is challenging and currently an active area of machine learning research [23]. One class of interpretability methods explains a neural network's decision on each image [2,31,33,20]. The challenge is that superfluous features that do not actually contribute (or could even negatively contribute) to the classification decision can often be highlighted. We seek minimal sufficient explanations for a classification decision in an image. In other words, what are the minimal features in an image that, by themselves, would be classified as female/male? We use the recently proposed contrastive explanations method [10] to answer this question, particularly looking at finding pertinent positives.

Procedure 3
The contrastive explanations method takes as input an image X, which is classified in category k, and a (possibly neural network) model f that maps images to logits on the classification decision. It selects a "pertinent positive explanation" ∆(X) as a solution to the following  optimization problem: where f κ (X, ∆) := max{max k =k f (∆) k − f (∆) k , −κ}, and the parameters (c, β, κ) are regularization hyperparameters.
Effectively, the contrastive explanations method selects as simple an explanation as possible (where simplicity is measured by the elastic net regularizer [36]), subject to the classification decision of the original image being preserved. This method has been validated on the MNIST hand-written digits dataset as well as an MRI dataset [10],  We set parameters κ = 10 and β = 0.1 and searched over the space of hyperparameters for c. We applied the contrastive explanations method to all images in the PPB* dataset and used the customized neural network classifier described in Section 2.2 as the function f . Examples of contrastive explanations, effectively values of ∆(X) for correctly predicted females and males in the PPB* dataset are presented in Figure 11. We can see in Figure 11a that the lips, eyes and cheeks show up very prominently as a sufficient explanation for a female classification -in particular, female lips look pink/red in color, and cheekbones are more prominent. This could be a result of celebrities, who wear more prominent makeup in photographs than the average human population, in training datasets. In Figure 11b, we see that the nose and forehead area are highlighted as sufficient explanations for a male classification. These are clearly consistent patterns noticed across correctly classified females and males. The average sufficient explanation masks for females and males are presented in Figure 12.
Work in visual perception [3,6] has shown that humans can adeptly classify the gender of a face using certain facial features in isolation. It is interesting to see that ML models are also able to do this. It is particularly interesting that a few of the dark females in Figure 11a have underrepresented skin type and hair pattern, and are yet classified correctly, probably largely due to their lip and cheek patterns.
However, the questions of racial and gender bias are still relevant for facial features in isolation. It has been statistically established that the facial structure of Europeanand African-descent females, including cheekbones and lips, is different [35]. So we cannot expect the performance of ML algorithms trained primarily on European-descent females to generalize to African-descent females, even if such simple facial features form a sufficient explanation. And lip makeup alone constitutes a simplistic explanation of a female face, furthering a gender stereotype.

Discussion and Future Work
We rigorously tested the influence of various features on the gender classification task. First, we showed that gender classification is relatively stable to variations in skin type and thus the skin type by itself has a minimal effect on the classification decision. Second, we observed unequal performance on females with varying hair length and tested the performance of a classifier that ignores hair information. We saw that the unequal performance across gender and skin tone persists, suggesting that facial features other than skin type and hair pattern are behind the phenomenon. Finally, using the contrastive explanations method, we identified red/pink lips, cheeks and eyes; and nose and forehead as sufficient facial features for a classification decision to be female or male respectively.
We began this research with the aim of developing invariant or equivariant face classifiers that would ignore skin type completely and thereby have equal accuracy across groups. Such an approach would preclude the need for a high level of diversification in training datasets. However, our mathematically-oriented analysis using the low-dimensional skin type group revealed that high-performing gender classifiers are already invariant to skin type. Moreover, we showed that the classifiers are sensitive to a host of facial features that are not easily considered in isolation. To solve the problem of unequal performance, we require diverse training datasets that represent humanity across many dimensions of identity, starting but not ending with ethnicity.
Many questions remain as to how exactly to go about diversifying training data as even ethnicity does not fully encapsulate an individual's identity. The contrastive explanations for female images consist of stereotypical attributes like lip makeup, and thus gender stereotypes commonly used by humans are confirmed in machine learning algorithms. This is a parallel issue to the issue of bias in skin type and correlated attributes: while females and males are balanced in training data, they are stereotypical females and males from the celebrity population. Informally speaking, we would expect the appearance of the general population of females and males alike to be quite different. We suggest that a good training dataset should diversify not only across ethnicity, but also across profession, cultural norms, and economic status, to capture a truly global population. Collecting such a dataset while controlling for image quality is a difficult, but necessary task.
As a parallel effort, it would also be interesting to examine the potential of decoupled classification on demographic groups, which along with task transfer learning has been shown to mitigate biases in classification of other facial attributes across race and gender [30].
Finally, the perspectives presented here are limited to the problem of binary gender classification from visual data, itself a flawed problem especially when considering various non-binary gendered individuals. The community needs to move beyond the binary gender construct in future work.