{'fraction_non_alphanumeric': 0.0377191036165964, 'fraction_numerical': 0.016418903927224317, 'mean_word_length': 4.575457694210787, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Learning fair prediction models with an imputed sensitive variable: Empirical studies

As AI has a wide range of inﬂuence on human social life, issues of transparency and ethics of AI are emerg-ing. In particular, it is widely known that due to the existence of historical bias in data against ethics or regulatory frameworks for fairness, trained AI models based on such biased data could also impose bias or unfairness against a certain sensitive group (e.g., non-white, women). Demographic disparities due to AI, which refer to socially unacceptable bias that an AI model favors certain groups (e.g., white, men) over other groups (e.g., black, women), have been observed frequently in many applications of AI and many studies have been done recently to develop AI algorithms which remove or alleviate such demographic disparities in trained AI models. In this paper, we consider a problem of using the information in the sensitive variable for fair prediction when using the sensitive variable as a part of input variables is prohibitive by laws or regulations to avoid unfairness. As a way of reﬂecting the information in the sensitive variable to prediction, we consider a two-stage procedure. First, the sensitive variable is fully included in the learning phase to have a prediction model depending on the sensitive variable, and then an imputed sensitive variable is used in the prediction phase. The aim of this paper is to evaluate this procedure by analyzing several benchmark datasets. We illustrate that using an imputed sensitive variable is helpful to improve prediction accuracies without hampering the degree of fairness much.


Introduction
Recently, artificial intelligence (AI) is being used as decision-making tools in various domains such as credit scoring, criminal risk assessment, education of college admissions (Angwin et al., 2016). As AI has a wide range of influences on human social life, issues of transparency and ethics of AI are emerging. However, it is widely known that due to the existence of historical bias in data against ethics or regulatory frameworks for fairness, trained AI models based on such biased data could also impose bias or unfairness against a certain sensitive group (e.g., non-white, women) (Kleinberg et al., 2018;Mehrabi et al., 2021). Therefore, designing an AI algorithm which is accurate and fair simultaneously has become a crucial research topic.
Demographic disparities due to AI, which refer to socially unacceptable bias that an AI model favors certain groups (e.g., white, men) over other groups (e.g., black, women), have been observed frequently in many applications of AI such as COMPAS recidivism risk assessment (Angwin et al., 2016), Amazon's prime free same-day delivery (Ingold and Soper, 2016), credit score evaluation (Dua and Graff, 2019) to name just a few. Many studies have been done recently to develop AI algorithms which remove or alleviate such demographic disparities in trained AI models so that they will treat sensitive groups as equally as possible. In general, these methods try to search AI models which are not only accurate but also similar between sensitive groups in a certain sense. For an example of similarity, it is required that accuracies of an AI model for each sensitive group are similar (Zafar et al., 2019).
In this paper, we consider a problem of using the sensitive variable for fair prediction. In most real applications, the sensitive variable itself has important information for prediction and using the sensitive variable as a part of input variables is usually helpful to improve prediction accuracies. Moreover, some fairness AI algorithms inevitably produce prediction models which depend on the sensitive variable as well as input variables. An example is the algorithm of Jiang et al. (2020) for the strong demographic parity, which transfers the score function of each sensitive group such that the distributions of the scores of each sensitive group are all equal. For this algorithm, the transformation of the score function should be differ for each sensitive group and hence the sensitive variable should be known in the prediction phase. In many cases, however, using the sensitive variable as a part of input variables is prohibitive by laws or regulations to avoid unfairness. In such cases, fairness AI algorithms yielding prediction models depending on the sensitive variable cannot be used.
A simple solution to reflect the information of the sensitive variable into prediction when using the sensitive variable explicitly in prediction is prohibitive to use an imputed sensitive variable in the prediction phase. That is, the sensitive variable is fully included in the learning phase to have a prediction model depending on the sensitive variable and then an imputed sensitive variable is used in the prediction phase. The aim of this paper is to evaluate this procedure by analyzing several benchmark datasets. We illustrate that using an imputed sensitive variable is helpful to improve prediction accuracies without hampering the degree of fairness much. That is, prediction models with an imputed sensitive variable are superior compared to prediction models not using the sensitive variable at all.
The paper is organized as follows. Various fairness algorithms are reviewed in Section 2. The proposed procedure to include the information of the sensitive variables by using an imputed sensitive variable into the prediction phase is explained in Section 3. Results of numerical studies are presented in Section 4 and concluding remarks follow in Section 5.

Review of fair AI algorithms
be a set of training data of size n consisting of triplets of input vector x i , sensitive variable z i and class label y i , which are assumed to be independent copies of a random vector (X, Z, Y) defined on X × Z × Y, where X ⊂ R p . We consider a binary classification problem, which means Y = {−1, 1}, and for notational simplicity, we let Z = {0, 1}, where Z = 0 refers to the unprivileged group and Z = 1 refers to the privileged group. Whenever the probability is mentioned, we mean it by either the probability of (X, Z, Y) or its empirical counterpart unless there is any confusion.
In this paper, we focus on between-group fairness (BGF) which requires that certain statistics of predictive values in each sensitive group should be similar. Even if we do not consider other concepts of fairness such as individual fairness (Dwork et al., 2012) and counter-factural fairness (Kusner et al., 2017) , our proposed method can be applied to such fairness concepts after minor modifications.
We consider AI algorithms which yield a real-valued function f : X → R so called a score function which assigns positive labeled instances higher scores than negative labeled instances. An example of the score function is the conditional class probability Pr(Y = 1|x = x). In most human-related decision makings, real-valued score functions are popularly used (e.g. scores for credit scoring).
Let F be a given set of score functions, in which we search an optimal score function in a certain  (Barocas and Selbst, 2016) 1{C f (X) = 1} ∅ Equal opportunity (Hardt et al., 2016) 1{C f (X) = 1} {Y = 1} Disparate mistreatment w.r.t. Error rate (Zafar et al., 2019) 1{C f (X) Y} ∅ Mean score parity (Coston et al., 2019) f (X) ∅ sense (e.g. minimizing the cross-entropy for classification problems). Examples of F are linear functions, reproducing kernel Hilbert space and deep neural networks to name a few. For a given f ∈ F , the corresponding classifier C f is defined as C f (x) = 1( f (x) > 0).

Definition of between-group fairness
For a given score function f and a sensitive group Z = z, we consider the group performance function of f given as for events E and E that might depend on f (X) and Y.
The group performance function q z in (2.1), which is considered by Celis et al. (2019), includes various performance functions used in fairness AI. We summarize representative group performance functions having the form of (2.1) in Table 1.
For given group performance functions q z (·), z ∈ {0, 1}, we say that f satisfies the BGF constraint with respect to q z if q 0 ( f ) = q 1 ( f ). A relaxed version of the BGF constraint so called the -BGF constraint, is frequently considered, which requires |q 0 ( f ) − q 1 ( f )| < for a given > 0. Typically, AI algorithms search an optimal function f among those satisfying the -BGF constraint with respect to given group performance functions q z (·), z ∈ {0, 1}.

Learning algorithms for fair artificial intelligence (AI)
Several learning algorithms have been proposed to find an accurate model f satisfying a given BGF constraint, which are categorized into the following three groups. In this subsection, we review some methods for each group.
Pre-processing methods: Pre-processing methods remove bias in training data or find a fair representation with respect to a sensitive variable before the learning phase and learn AI models based on de-biased data or fair representation (Calmon et al., 2017;Creager et al., 2019;Dixon et al., 2018;Feldman et al., 2015;Kamiran and Calders, 2012;Quadrianto et al., 2019;Webster et al., 2018;Xu et al., 2018;Zemel et al., 2013). Kamiran and Calders (2012) suggested pre-processing methods to eliminate bias in training data by use of label changing, reweighing and sampling. Based on the idea that transformed data should not be able to predict the sensitive variable, Feldman et al. (2015) proposed a transformation of input variables for eliminating the disparate impact. To find a fair representation, Calmon et al. (2017) and Zemel et al.(2013) proposed a data transformation mapping for preserving accuracy and alleviating discrimination simultaneously. Pre-processing methods for fair learning on text data were studied by Dixon et al. (2018) and Webster et al. (2018).
In-processing methods: In-processing methods generally train an AI model by minimizing a given cost function (e.g. the cross-entropy, the sum of squared residuals, the empirical AUC etc.) subject to a -BGF constraint. Most group performance functions q z (·) are not differentiable, and thus various surrogated group performance functions and corresponding -BGF constraints have been proposed (Bechavod and Ligett, 2017;Celis et al., 2019;Cho et al., 2020;Donini et al., 2018;Goh et al., 2016;Kamishima et al., 2012;Menon and Williamson, 2018;Narasimhan, 2018;Vogel et al., 2020;Zafar et al., 2017Zafar et al., , 2019. Kamishima et al. (2012) used a fairness regularizer which is an approximation of the mutual information between the sensitive variable and the target variable. Zafar et al. (2017Zafar et al. ( , 2019 proposed covariance-type fairness constraints as tractable proxies targeting the disparate impact and the equality of the false positive or negative rate, and Donini et al. (2018) used a linear surrogated group performance function for the equalized odds. On the other hand, Menon and Williamson (2018) and Celis et al. (2019) derived an optimal classifier for a constrained fair classification as a form of an instance-dependent threshold. Also, for fair score functions, Vogel et al. (2020) proposed fairness constraints based on ROC curves of each sensitive group.
Post-processing methods: Post-processing methods first learn an AI model without any BGF constraint and then transform the decision boundary or score function of the trained AI model for each sensitive group to satisfy given BGF criteria (Chzhen et al., 2019;Corbett-Davies et al., 2017;Fish et al., 2016;Hardt et al., 2016;Jiang et al., 2020;Pleiss et al., 2017;Wei et al., 2021). Chzhen et al. (2019) and Hardt et al. (2016) suggested finding sensitive group dependent thresholds to get a fair classifier with respect to equal opportunity. Jiang et al. (2020) and Wei et al. (2021) developed an algorithm to transform the original score function to achieve a BGF constraint.

Fair prediction models with an imputed sensitive variable
We consider two situations according to whether the sensitive variable can be used in the prediction. The first situation (Situation 1) is that the sensitive variable z is allowed to be used in the prediction phase and thus we assume that z is one of the entries of the input vector. That is, there exists j ∈ [p] such that x j = z. In contrast, the second situation (Situation 0) is that the sensitive variable cannot be used in the prediction phase and thus the sensitive variable does not belong to the entries of the input vector. As we mentioned in Introduction, there are many cases where the sensitive variable is not allowed to be a part of the input vector by regulations or laws to ensure fairness although using z in prediction would help improving prediction accuracy.
In this paper, we propose a method to use the information in the sensitive variable under Situation 0. The idea of the proposed method is simple and intuitive. At the learning phase, a prediction model is learned with the input vector including the sensitive variable. Then, at the prediction phase, we impute the sensitive variable based on the other input variables and make a prediction with the imputed sensitive variable.
To be more specific, let x be the input vector not including z and let x z = (x , z) is the input vector including z. In the learning phase, we learn a prediction model f : X × Z → R with the training data (x z,1 , y 1 ), . . . , (x z,n , y n ). In addition, we learn a prediction model g : X → Z which predicts z based on x. Then, in the prediction phase, for a given input vector x, we make a prediction by f (xẑ), wherê z = g(x). The proposed procedure is summarized in Algorithm 1.
There are at least two advantages of the proposed method compared to the standard method for Situation 0 that learns a prediction model f 0 : X → R in the learning phase. First of all, using the information of z in the prediction phase would be usually helpful for improving prediction accuracy. This advantage would keep being materialized even when an imputed z is used, which will be confirmed by numerical studies on Section 4.
The second advantage, which is the main motivation of our proposed method, lies in that there are several useful fair AI algorithms which yield prediction models which are functions on X × Z. An example is the Wasserstein fair algorithm (Jiang et al., 2020) whose details are given in Section 4.1. It would not be easy to modify this algorithm for not using z in the prediction phase. In contrast, we can use such algorithms without much modification by use of an imputed sensitive variable. By analyzing several real datasets, we show that using an imputed sensitive variable instead of the sensitive variable itself does not degrade the performance of prediction models much in view of prediction accuracy as well as fairness.
Algorithm 1 Learning fair prediction models with an imputed sensitive variable [1] Learning phase: Learn a prediction model f (x z ) based on the training data D [2] Imputation phase: Learn a prediction model g : X → {0, 1} which predicts the sensitive variable z by x.
[3] Prediction phase: For a given new input x, we predict y based on f (xẑ), whereẑ = g(x).

Numerical studies
In this section, we investigate the performance of the proposed procedure and compare it with the predictions models without using an imputed sensitive variable (either not using the sensitive variable under S0 or using the sensitive variable fully in the prediction phase under S1).

Considered fair AI algorithms
For fairness AI algorithms, we consider the three algorithms : (1) fair classifier with the disparity impact (DI) constraint (Zafar et al., 2017), (2) fair classifier with the prejudice index (PI) constraint (Kamishima et al., 2012) and (3) Wasserstein fair classifier of (Jiang et al., 2020). The first two algorithms are in-processing methods and the third one is a post-processing method.
Fair classifier with the DI constraint: The DI constraint requires that a prediction model f satisfies where E n is the expectation with respect to the empirical distribution. By use of the Lagrangian multiplier, we could learn a prediction model by minimizing the penalized empirical risk given as is a given empirical risk. In this paper, we consider the cross-entropy for the empirical risk. The DI constraint (4.1) is hard to be optimized since the indicator function 1(·) is neither continuous nor convex. A typical remedy is to replace the indicator function by a convex surrogate function. One of the popularly used convex surrogate functions is the hinge function defined as φ hinge (w) = (1 − w) + (Donini et al., 2018). Using the hinge surrogate loss, we learn the prediction model by minimizing the surrogate penalized empirical risk over F . Fair classifier with the PI constraint: PI measures statistical dependence between the class label and sensitive variable. We think that a given classifier is more fair if its PI is smaller. For data (x i , z i ) n i=1 = D, we define PI as the prejudice remover regularizer given as σ is sigmoid function. We learn a fair classifier by minimizing C n ( f ) + λ n PI( f ) over F . Wasserstein fair classifier: This method is a post-processing method. Let S z (x) be the estimated conditional class probability Pr(Y = 1|X = x, Z = z). Let P n,z be the empirical distribution of {S z (x i ) : z i = z}. Then, the barycenter distributionP is defined as where W 1 is the L 1 Wasserstein distance and P(Ω) is set of distributions on Ω = [0, 1]. Once the barycenterP is obtained, the fair classifier is constructed by use of the quantile matching between P n,z andP. For detailed procedures, we refer (Jiang et al., 2020).

Analyzed data sets
To evaluate the proposed procedure, we analyze three benchmark real world datasets : Adult income dataset, Bank marketing dataset and Law school dataset.
• Adult dataset is composed of 45,222 individuals' features (e.g. age, education, race) with a binary label which indicates whether individual's income is larger (positive class) than 50K USD. We take the variable 'sex' as a sensitive variable.
• Bank dataset is composed of 41,188 clients' features (e.g. job, education) with a binary label which indicates whether has a client subscribed a term deposit. We take the variable 'age' as a sensitive variable which is 1 when client's age is between 25 and 60 years.
• Law dataset is composed of 26,551 law school applicants' features (e.g. lsat score, family income) with a binary label which indicates whether applicant is accepted to law school. We take the variable 'race' as a sensitive variable which is 1 when applicant is white.

Performance of the proposed method
We use three classification algorithms for g : logistic regression, boosting, deep neural network. Also we use logistic regression and deep neural network for classifier f . The structure of DNN for g is two hidden layers with the number of nodes for each hidden layers is equal to the number of input nodes. The structure of DNN for f is two hidden layers with the numbers of nodes for each hidden layers Whenever the regularization parameter is selected (e.g. Lagrangian multipliers for DI and PI), we search it so that the corresponding estimated classifier achieves a certain level of fairness which is set in advance and compare the accuracy of the classifier on test data. We select the one with the highest train accuracy among classifiers corresponded to regularization parameters that make the DI of the train data less than 0.05(the case of PI is 0.005). We repeat 5 times to split train data and test data with ratio 7:3 and average the performances.
For the Wasserstein fair classifier which is a post-processing method, we evaluate the strong demographic parity (SDP) measure as well as the DI.The SDP for a given belief function S (x) that is an estimate of Pr(Y = 1|X = x) is defined as where P n is the empirical distribution.
Results of our numerical studies are presented in Tables 2 to 5. The remarks are summarized as follows.
• Table 2 summarizes the results without fairness constraints. It is observed that using the sensitive variable in prediction is not helpful. However, as we see in Tables 3 and 4, using the sensitive variable in prediction is helpful for fair learning algorithms. It is interesting to figure out why the role of the sensitive variable in prediction is different for standard and fair learning algorithms, which we leave as a future work.
• From Tables 3 and 4, we can see that the fair classifiers with an imputed sensitive variable are superior to the fair classifiers without using the sensitive variable (S 0 ). Moreover, the fair classifiers with an imputed sensitive variable is not much worse than the fair classifier using the sensitive variable fully (S 1 ).   • For the Wasserstein fair classifier whose results are presented in Table 5, it is somehow surprising that the fair classifier with an imputed sensitive variable is superior to that using the sensitive variable itself. A possible answer would be that an imputed sensitive variable could regularize the estimated classifier and so that could avoid overfitting.
• In general, the performances of fair prediction models with an surrogated sensitive variable do not strongly depend on the choice of an imputational algorithm. Table 6 summarizes the accuracies of the three imputation algorithms. Boosting seems to be the best but the performances of the corresponding fair prediction models are similar to the other imputation algorithms. These observations suggest that the accuracy of an imputed sensitive variable is not important for the performance of the corresponding fair prediction model unless the accuracy is too bad.

Concluding remarks
In this paper, we have illustrated that using an imputed sensitive variable is helpful when using the sensitive variable itself in the prediction phase is not allowed. Also, the accuracy of imputing the sensitive variable does not affect the overall performance of fair classifiers. Any reasonable supervised learning algorithms would be enough to obtain an imputed sensitive variables.
In this paper, we proposed a two-step procedure where we learn the fair classifier and the prediction model for surrogated sensitive variables separately. A better procedure would do this two jobs at the same time. That is, we are to learn the fair classifier and imputed sensitive variable simultaneously. This would be a promising direction for future works.