{'fraction_non_alphanumeric': 0.030508789880011163, 'fraction_numerical': 0.01565745822094069, 'mean_word_length': 4.611343075852471, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 16, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
On the Choice of Fairness: Finding Representative Fairness Metrics for a Given Context

It is of critical importance to be aware of the historical discrimination embedded in the data and to consider a fairness measure to reduce bias throughout the predictive modeling pipeline. Various notions of fairness have been defined, though choosing an appropriate metric is cumbersome. Trade-offs and impossibility theorems make such selection even more complicated and controversial. In practice, users (perhaps regular data scientists) should understand each of the measures and (if possible) manually explore the combinatorial space of different measures before they can decide which combination is preferred based on the context, the use case, and regulations. To alleviate the burden of selecting fairness notions for consideration, we propose a framework that automatically discovers the correlations and trade-offs between different pairs of measures for a given context. Our framework dramatically reduces the exploration space by finding a small subset of measures that represent others and highlighting the trade-offs between them. This allows users to view unfairness from various perspectives that might otherwise be ignored due to the sheer size of the exploration space. We showcase the validity of the proposal using comprehensive experiments on real-world benchmark data sets.


Introduction
Machine learning (ML) has become one the most applicable and influential tools to support critical decision makings such as college admission, job hiring, loan decisions, criminal risk assessment, etc. (Makhlouf, Zhioua, and Palamidessi 2021). Widespread applications of ML-based predictive modeling have induced growing concerns regarding social inequities and unfairness in decision-making processes. With fairness being critical in practicing responsible machine learning, fairness-aware learning has been the primary goal in many recent machine learning developments.
fairness-aware learning can be achieved by intervention at pre-processing, in-processing(algorithms), or postprocessing strategies (Friedler et al. 2019). Pre-processing strategies involve the fairness measure in the data preparation step to mitigate the potential bias in the input data and produce fair outcomes (Kamiran and Calders 2012;Feldman et al. 2015;Calmon et al. 2017). In-process approaches (Agarwal et al. 2018;Celis et al. 2019;Zafar et al. Copyright © 2022, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. 2015) incorporate fairness in the design of the algorithm to generate a fair outcome. Post-process methods (Hardt, Price, and Srebro 2016;Kamiran, Calders, and Pechenizkiy 2010), manipulate the model outcometo mitigate unfairness.
Fairness is an abstract term with many definitions. The literature on Fairness in ML encompasses more than 21 fairness metrics (Narayanan 2018; Verma and Rubin 2018) that could be utilized to mitigate the bias in a given model. Yet, one could face a situation of not being able to choose among various notions of fairness. For one, choosing the proper fairness notion could depend on the context. More importantly, even if chosen properly, there is no evidence to support that ensuring the selected fairness notion enables to describe the overall unfairness and is adequate to address the challenge in a given problem. On the other hand, improving algorithms to satisfy a given notion of fairness, does not guarantee the lack of unfairness in another notion. In fact, the potential intractability of various notions has not been studied in the literature and is the main focus of this paper.
For example, the choice of fairness may depend on the users' knowledge about the type of disparities influencing the outcome of interest and its corresponding independent variables in a specific application domain. However, formalizing and evaluating the appropriate notion is often inaccessible to practitioners due to the limited awareness of the applicability of fairness notions within a given problem.
While recent literature aims to answer questions such as how to measure fairness and mitigate the algorithmic bias, little is known about the sufficiency of different fairness notions and how one should choose among them. In this paper, we aim to elaborate on the sufficiency of different fairness metrics in a given problem by considering their potential interactions and overlaps. We develop an automated tool to help users decide on the choice of fairness.
Choosing appropriate fairness metrics for auditing and bias mitigation within a specific context is part of the responsibilities of practitioners (who are applying existing approaches) or researchers (who are developing methodologies). However, this is a troublesome task and it requires a rigorous exploration of the combinatorial space of fairness notions. The Impossibility theorem (Kleinberg, Mullainathan, and Raghavan 2016) and the trade-offs between fairness notions exacerbate the difficulty of the task, increase the chance of impulsiveness, and may introduce additional bias as a result of a wrong choice.
In order to help data scientists to specify the fairness metrics to consider, in this paper we propose an automatic framework for identifying a small subset of fairness metrics that are representative of other metrics. In summary, we make the following contributions: • We propose the problem of using the correlations between different fairness metrics, for a given context (specified by training data and a model type), to find a small subset of metrics that represent other fairness metrics. To the best of our knowledge, we are the first to propose this problem. • We design a sampling-based Monte-Carlo method to estimate the correlations between the fairness metrics. • We develop an efficient approach for sampling models with different fairness metric values, which enables estimating the correlation between the fairness metrics. • We adapt and extend the existing work in order to specify the small subset of representative metrics, using the correlations between fairness metrics. • In order to evaluate our findings, we conduct comprehensive experiments, using real-world benchmark datasets, multiple types of classifiers, and a comprehensive set of fairness metrics. Our experiment results, verify the effectiveness of our approach for choosing fairness metrics.
In the following, we first provide a brief explanation of the fairness metrics and then propose our technical solution.

Fairness Model
Consider a training dataset D consisting of n data points denoted by vectors of (X, S, Y ). Let X be the set of nonsensitive attributes of dimension d, S be the set of sensitive attributes, and Y be the response variable. In a classification setting Y = 1, . . . , K with K being the total number of distinct classes. Let h be the classifier function where h : (X, S) −→ Y . Let F denotes the set of m (group) fairness metrics, F = {f 1 , f 2 , . . . , f m }. Since fairness metrics are defined for each sensitive group, let f s j be the fairness metric j defined for sensitive group s.
Let Y andŶ denote the true and predicted labels in a given test problem, respectively. Most of the existing fairness notions are defined based on the joint distribution of the S, Y andŶ variables, and fall into one of three wellknown categories of Independence(f ⊥ S), Separation(f ⊥ S|Y ), Sufficiency(Y ⊥ S|f ) (Barocas, Hardt, and Narayanan 2017). Various fairness metrics have been proposed in the literature, each based on one of the aforementioned ideas. Following the literature on defining the fairness metrics and for the ease of explanation let us consider a binary classification h and a binary sensitive attribute S 1 . For the purpose 1 Please note that techniques provided in this paper are agnostic to the choice of fairness metrics and machine learning tasks. Extensions to higher numbers and carnality of the sensitive attributes and class labels are straightforward as we shall explain in the discussion section

Fairness Notion Label Formulation
Equalized Odds f1 T P1+F P1 Table 1: Fairness notions of our analysis, we assume S = 1 and S = 0 to represent the Privileged and Unprivileged sensitive groups, respectively. The fairness metrics can be derived by expanding the confusion matrix on the outcome of h split according to each sensitive attribute value (Kim, Chen, and Talwalkar 2020). Let T P (True Positive), F N (False Negative), F P (False Positive), and T N (True Negative) be the elements of a confusion matrix. Given the binary sensitive attribute S = {0, 1}, a split of the confusion matrix on Privileged group is denoted by T P 1 , F N 1 , F P 1 , and T N 1 , and total observations of Privileged group is denoted by N 1 . Thus, for instance, Statistical parity would be equivalent to T P1+F P1 which measures positive prediction outcome (Ŷ = 1) among different sensitive groups without considering their true Y label. Similarly, Equalized odds can be expressed as Note that when K > 2, the fairness metrics can be defined upon multiple confusion matrices split according to a combination of class labels.
Having briefly discussed the fairness metrics next we propose our framework for identifying a small subset of representative metrics for a given context.

Identifying Representative Fairness Metrics
Fairness is an abstract metric with many definitions from different perspectives and in different contexts. This variety of definitions, coupled with the Impossibility Theorems and the trade-off between definitions make it overwhelmingly complicated for ordinary users and data scientists to select a subset of those to consider. After providing the preliminaries and the set of well-known fairness metrics, in this section, we focus on identifying a subset of representative fairness metrics to facilitate the choice of fairness for a given context. In particular, we use the correlation between the fairness metric as the measure to identify their similari-ties. given a universe F of fairness definitions of interest, we would like to find a subset R F , with a significantly smaller size that represents all metrics in F. That is, ∀f i ∈ F, there exists a fairness metric f j ∈ R F such that the correlation between f i and f j is "high".
To this end, we first need to be able to calculate the correlations between metrics of fairness for a given context. Estimating these correlations is one of the major challenges we shall resolve in this section. While the general ideas proposed in this section are not limited to a specific ML task, without loss of generality, we will focus on classification for developing our techniques. First, we note that given a classifier, one can audit it and compute its (un)fairness with regard to different metrics. It, however, does not provide the relation between the metrics. In other words, it is not clear how trying to resolve unfairness on one metric will impact the other metrics. Next, using the existing systems for achieving fairness does not seem to provide a promising approach: (i) existing fairML learning approaches (including the benchmark methods in AIF360 (Bellamy et al. 2018)) are designed for a subset of fairness metric; (ii) the ones, including (Zhang et al. 2021), that claim to cover different metrics are inefficient when multiple metrics are considered simultaneously -due to the exponentially large search space they consider. In summary, the FairML approaches are designed to, build fair models, given a set of metrics, in oppose to finding the relation between different fairness metrics.
Therefore, in the following, we design a Monte-Carlo (Robert 2004;Hickernell et al. 2013) method for estimating the underlying correlation and trade-offs between fairness metrics. Monte-Carlo methods turn out to be both efficient and accurate for such approximations.
After identifying the correlations, we use them to find the set R F of the representative metrics.

Estimating Between Fairness Correlations
Monte-Carlo methods use repeated sampling and the central limit theorem for solving deterministic problems (Durrett 2010). At a high level, the Monte-Carlo methods work as follows: first, they generate a large enough set of random samples; then they use these inputs to estimate aggregate results. We use Monte-Carlo methods to estimate correlations between the fairness metrics. The major challenge towards developing the Monte-Carlo method is being able to generate a large pool of samples. Every sample is a classifier that provides different values for every fairness metric. We use a sampling oracle that upon calling it, returns the fairness values for a sampled classifier. We shall provide the details of the oracle in the next subsection.
Correlation is a measure of linear association between two variables f i and f j . When both variables are random it is referred to as Coefficient of Correlation r. The correlation model most widely employed to calculate r is the normal correlation model. The normal correlation model for the case of two variables is based on the bivariate normal distribution, (Neter et al. 1996). Having enough repeated sampling (N > 30), we can assume the variables f i and f j follows the Normal distribution (central limit theorem) with means µ i and µ j , and standard deviations of σ i and σ j , respec-tively. In a bivariate normal model, the parameter ρ ij provides information about the degree of the linear relationship between the two variables f i and f j , which is calculated as ρ ij = σij σiσj . The correlation coefficient takes values between -1 and 1. Note that if two variables are independent σ ij = 0 and subsequently ρ ij = 0. When ρ ij > 0, f 1 tends to be large when f 2 is large, or small when f 2 is small. In contrast, when ρ ij < 0 (i.e. two variables are negatively correlated), f 1 tends to be large when f 2 is small, or vice versa. ρ ij = 1 indicates a perfect direct linear relation and -1 denotes a perfect inverse relation. Since σ i is unknown, a point estimator of ρ is required. The estimator is often called thePearson Correlation Coefficient.
In order to estimate the correlations, we use the sampling oracle to sample N classifiers h k , ∀k = 1, . . . , N , and to calculate fairness values for each. Let f i,k be fairness metric i of classifier h k , thus the Pearson Correlation Coefficient is defined as follow ∀i, j = 1, . . . , m: In order to reduce the variance of our estimation, we repeat the estimation L times (we use the rule of thumb number L = 30 in our experiments), where every iteration returns the correlation estimation r ij , ∀i, j = 1, . . . , m. Then, the correlation r * ij is computed as the average of estimations in each round. That is, Using the central limit theorem, ρ ij follows the Normal distribution N ρ ij , σij √ L . Given a confidence level α, the confidence error e identifies the range Using the Z-table, while using the sample variance s 2 ij to estimate σ 2 ij , the confidence error is computed as

Developing the Sampling Oracle
Having discussed the estimation of the correlations between the fairness metrics, next we discuss the development details of the sampling oracle. Upon calling the oracle, it should draw an iid sample classifier and evaluate it for different fairness metrics. Considering the set of fairness metrics of interest F = {f 1 · · · f m }, the output of the sampling oracle for the i th sample can be viewed as a vector of values {f 1i · · · f mi }, where f ji is the fairness of sampled classifier for metric f j . Calling the oracle by the correlation estimator N times forms a table of N samples where each row are the fairness values for a sampled classifier ( Figure 1). There are two requirements that are important in the development of the sampling oracle. First, since our objective is to find the correlations between the fairness metrics, we would like the samples to provide different values for the fairness metrics. In other words, the samples should provide Besides, since the correlation estimator calls the oracle many times before it computes the correlations, we want the oracle to be efficient. Our strategies to satisfy the design requirement for the sampling oracle are based on two simple observations. First, we note that the performance of a model for a protected group depends on the ratio of samples from that protected group and the distribution of their label values. To better explain this, let us consider a binary classifier h and two groups g 1 and g 2 . Clearly, if all samples in the training data belong to g 1 then the model is only trained for g 1 , totally ignoring the other group. As the ratio of samples from g 2 increases in the training data, the model trains better for this group. Specifically, since the training error -the average error across training samples -is minimized during the training process, the ratio of g 1 to g 2 in the training data directly impacts the performance of the model for each of the groups. Besides, the other factor that impacts the prediction of the model for a protected group is the ratio of positive to negative samples. Therefore, as an indirect method to control variation over the space of fairness values, we consider variation over the ratios of the samples from each protected group and label values in the training data. The next observation is that when sampling to estimate the correlations between the fairness metrics, we are not concerned with the accuracy of the model. This gives the opportunity to use subsampling to train the model on small subsets of the training data in order to gain efficiency. Using these observations, we propose a Bootstrap resampling approach to generate different subsamples from the dataset for the training purpose.
Bootstrapping is a data-driven statistical inference (standard error and bias estimates, confidence intervals, and hypothesis tests) methodology that could be categorized under the broader class of resampling techniques (Efron and Tibshirani 1994;Hesterberg 2011). The core idea of Bootstrapping is similar to random sampling with replacement without further assumptions. Let K be the number of drawn Bootstraps samples. Given the training dataset D, we aim to construct smaller representative subsets of D we use to train the sampled model.
Consider a binary classification problem (Y ∈ {0, 1}) with two protected groups (g 1 : Group1, g 2 : Group2) to describe our sampling procedure. In order to (indirectly) control the fairness values, we bootstrap different samples ratios from each of the protected groups and label values as shown in Figure 2. Let w = {w 1 , w 2 , w 3 , w 4 } be the ratios for each of the cells of the table. To generate each sample, we need to draw the vector w uniformly from the space of possible values for w. Given that w represents the ratios from each cell, To make sure values in w are drawn uniformly at random, we first generate four random numbers, each drawn uniformly at random from the range [0, 1]. Then, we normalize the weights as w i = w i / 4 i=1 w i . Having sampled the ratios, we bootstrap w i * K samples from the samples of D that belong to cell i of the table to form the bootstrapped dataset B j . Next, the oracle uses the dataset B j to train the sampled classifier h j . Having trained the classifier h j , it next evaluates the model to compute the values f kj , for each fairness metric f k ∈ F, and return the vector {f k1 , · · · , f km }. The set of samples collected from the sampling oracle then form the table of fairness values shown in Figure 1, which is used to estimate the correlations between the fairness metrics. The Pseudocode of our proposed sampling approach is provided in the Appendix.

Finding the Representative Fairness Metrics using Correlations
To discover the R F representative subset of fairness metrics that are highly correlated, we utilize the correlation estimation from our proposed Monte-Carlo sampling approach described in the previous sections.
Consider a complete graph of m vertices (denoting each fairness metric f i ∈ F), where the weight of an edge (f i , f j ) between the nodes f i and f j is equal to their correlations r ij . The goal is to identify the subsets of vertices such that the within subset positive correlations and between subsets negative correlations are maximized. This problem is proven to be NP-complete. The well-known approximation algorithm is proposed in (Bansal, Blum, and Chawla 2004) provides a constant approximation ratio for this problem.
Considering the complete graph of correlations where the edge weights are in {+, −}, the algorithm first selects one of the nodes as the pivot, uniformly at random. Next, all the nodes that are connected to the pivot with a + edge are connected to the cluster of the pivot. Next, the algorithm removes the already clustered node from the graph and repeats the same process by selecting the next pivot until all points are clustered.
In order to adapt this algorithm for finding the representative fairness metrics, we consider a threshold τ . Then, after selecting the pivot (a fairness metric f i ∈ F), we connect each fairness metric f j ∈ F to the cluster of f i , if r ij ≥ τ .
Moreover, in order to find a small subset R F , we repeat the algorithm multiple times and return the smallest number of subsets. For every cluster of metrics, the pivot f i is added to the set of representative metrics R F .

Experiments Datasets
Our empirical results are based on the benchmark datasets in fair-ML literature 2 toolkit (Bellamy et al. 2018): COMPAS 3 : published by ProPublica (Angwin et al. 2016), this dataset contains information of juvenile felonies such as marriage status, race, age, prior convictions, etc. We normalized data so that it has zero mean and unit variance. We consider race as the sensitive attribute and filtered dataset to black and white defendants. The dataset contains 5,875 records, after filtering. We use two-year violent recidivism record as the true label of recidivism: Y = 1 if the recidivism is greater than zero and Y = 0 otherwise. We consider race as the sensitive attribute.
Adult 4 : contains 45,222 individuals income extracted from the 1994 census data with attributes such as age, occupation, education, race, sex, marital-status, native-country, hours-per-week etc. We use income (a binary attribute with values ≥ $50k and ≤ $50k) as the true label Y . The attribute sex is considered as the sensitive attribute.
German Credit Data 5 : includes 1000 individuals credit records containing attributes such as marital status, sex, credit history, employment, and housing status. We consider both sex and age as the sensitive attributes, and credit rating (0 for bad customers and 1 for good customers) as the true label, Y , for each individual.
Bank marketing 6 : Published by (Moro, Cortez, and Rita 2014) the data is related to direct marketing campaigns number of phone calls of a Portuguese banking institution. The classification goal is to predict if a client will subscribe to a term deposit (variable Y ). The Dataset contains 41188 and 20 attributes that are collected from May 2008 to November 2010. We consider age as the sensitive attribute.

Performance Evaluation
In order to estimate between-fairness correlations using our proposed Monte-Carlo method, we use 1000 sampled models for each round and repeat the estimation process 30 times. Our proposed approaches are evaluated using a set of common classifiers; Logistic Regression (Logit), Random Forest (RF), K-nearest Neighbor (KNN) , Support Vector Machines (SVM) with linear kernel, and Neural Networks (NN) with one dense layer. Our findings are transferable to other classifiers. Correlation estimation quality: We begin our experiments by evaluating the performance of our correlation estimation method. Recall that we designed Monte-carlo method. In every iteration, the algorithm uses the sampling oracle and samples classifiers to evaluate correlations between pairs of fairness notions, for which we use 1K samples. To see the impact of number of estimation iteration in the estimation variance and confidence error, we varied the number of iterations from 2 to 30 times. Since the number of estimation pairs are quadratic to |F|, we decided to (arbitrarily) pick a par of notions and provide the results for it. To be consistent across the experiments, we fixed the pair f 7 and f 12 for all data sets/models/settings. We confirm that the results and findings for other pairs of notions are consistent to what presented. 7 Figure 3 provides the results for f 7 and f 12 for COMPAS (a), Adult (b), Credit (c), Bank (d) datasets. Looking at the figure, one can confirm the stable estimation and small confidence error bars which demonstrate the high accuracy of our estimation. Also, as the number of iterations increase the estimation variance and confidence error significantly decreases. Impact of data/model on correlation values: In this paper, we proposed to identify representative fairness metrics for a given context (data and model). The underlying assumption behind this proposal is that correlations are data and model dependent. Having evaluated our correlation estimation quality, in this experiment we verify that the data/model dependent assumptions for the correlations are valid. To do so, we first fix the data set to see if the correlations are model dependent ( Figure 4) and then fix the model to see if the correlations are model dependent ( Figure 5). First, we confirm that the results for other data sets/models are consistent with what presented here. Looking at Figure 4, it is clear that correlations are model dependent. In particular, generally speaking, the complex boundary of the non-linear models (e.g. NN) reduce the correlation between fairness metrics, compared to linear models (e.g. Logit). Similarly, Figure 5 verifies that correlations are data dependent. This is because different data sets represent different underlying distributions with different properties impacting the fairness values. Number of representative metrics: Next, we evaluate the impact of the parameter parameter τ , used for identifying the representative metrics, in the number of representatives R F . Figure 6 present the results for various values of the threshold τ for each ML model for COMPAS, Adult, Credit, Bank datasets. The thresholds values are selected as τ ∈ {0.1, 0.2, · · · , 0.9}. We can observe that as τ increases the number of subsets increases. For non-linear classifiers such as NN, the number of subsets is relatively larger due to the sensitivity of the non-linear decision boundaries to the subset of samples in the training set. In such situation, the fairness metrics would be less correlated. In general, fairness metrics of linear decision boundaries are more correlated. Although similar overall pattern can be observed from one dataset to another, the number of subset of representatives are different. The results indicates that the proposed approach for estimation of correlation is model dependent.
In our next experiments, Figure 7, represents the number of representative subsets of fairness metrics, R F , for different datasets fixing the ML model. We demonstrate that given a model as τ increases the size of R F increases. The nonlinear models as expected, require more subsets The results indicates that the proposed approach for estimation of correlation is data dependent.
To provide an example of representative subsets of fair-    f1  f2  f3  f4  f5  f6  f7  f8  f9  f10  f11  f12  f13  f14  f15  f16   f1  f2  f3  f4  f5  f6  f7  f8  f9  f10  f11  f12  f13  f14  f15 f1  f2  f3  f4  f5  f6  f7  f8  f9  f10  f11  f12  f13  f14  f15  f16   f1  f2  f3  f4  f5  f6  f7  f8  f9  f10  f11  f12  f13  f14  f15 Figure 8, we illustrate a graph with nodes representing the fairness metrics, orange nodes indicating the representative metrics of each subset, and edges showing the subsets of metrics that are highly correlated. We used τ = 0.5 for this plot. The graph confirms that the number of subsets are smaller using Logit model than that using NN, as previously discussed. Similarly, Figure 9 shows the representative metrics for Credit dataset using Logit and NN models. Discovery Ratio, Predictive Parity, Equality of Opportunity, and Average Odd Difference are selected as representative metrics for Credit dataset when we use Logit classifier.

Related work
Algorithmic fairness have been studied extensively in recent years (Corbett-Davies et al. 2017;Kleinberg et al. 2018). Various fairness metrics have been defined in the literature to address the inequalities of the algorithmic decision making from different perspectives. (Barocas, Hardt, and Narayanan 2017) and (Verma and Rubin 2018) define different fairness notions in details. Majority of works focus on the fairness consideration in different stages of predictive modeling including pre-processing (Feldman et al. 2015;Kamiran and Calders 2012;Calmon et al. 2017), in-processing (Calders and Verwer 2010;Zafar et al. 2015;Asudeh et al. 2019), and post-processing (Pleiss et al. 2017;Feldman et al. 2015;Stoyanovich, Yang, and Jagadish 2018;Hardt, Price, and Srebro 2016) to mitigate bias of the outcome. Furthermore, the proposed interventions are tied to a specific notion of fairness; statistical parity (Calders and Verwer 2010), equality of opportunity (Hardt, Price, and Srebro 2016), disparate impact (Feldman et al. 2015), etc. A few recent work, discuss the challenge of choosing the appropriate fairness metric for bias mitigation considerations. (Makhlouf, Zhioua, and Palamidessi 2021) surveys notions of fairness and discusses the subjectivity of different notions for a set of realworld scenarios. The challenges about a growing pool of fairness metrics for unfairness mitigation and some aspects of the relationships between fairness metrics are highlighted in (Castelnovo et al. 2021) with respect to the distinctions individual vs. group and observational vs. causality-based. As a result, the authors highly promotes quantitative research for fairness metric assessment for bias mitigation. Building on previous works (Kleinberg, Mullainathan, and Raghavan 2016;Chouldechova 2017), (Garg, Villasenor, and Foggo 2020) provides a comparative using mathematical representations to discuss the trade-off between some of the common notions of fairness.

Final Remarks
The abundance, trade-offs, and details of fairness metrics is a major challenge towards responsible practices of machine learning for the ordinary data scientists. To alleviate the overwhelming task of selecting a subset of fairness measures to consider for a context (a data set and a model type), we proposed a framework that, given a set of fairness notions of interest, estimates the correlations between them and identifies a subset of notions that represent others. Our comprehensive experiments on benchmark data sets and different classification models verify the validity of out proposal and effectiveness of our approach.  (b) show similar results for the correlation estimates, using Credit dataset with S=age referred to as "Credit-age".
Figures 11 and 12 demonstrate a comprehensive comparison on the number of representative subsets between different models using all of datasets (including credit-age). As we can observer and discussed before, the number of representative subsets are model-dependent. Figure 13 illustrate the correlation estimations for Creditage dataset. Note that in (c) the correlation estimates for certain pairs are missing, when we use NN. The reason for missing correlations is the NA values for f 9 which indicate that F N 1 is zero for this case. Similarly for f 14 , since F N 1 is zero it yiels NA for FNR ratio which is f 14 . In addition, f 8 is always zero for any sampled model (FOR disparity rate is zero). Similarly for f 13 the disparity is zero.     Figure 13: Using Credit-age data set to illustrate that Correlations are model dependent.   f1  f2  f3  f4  f5  f6  f7  f8  f9  f10  f11  f12  f13  f14  f15  f16   f1  f2  f3  f4  f5  f6  f7  f8  f9  f10  f11  f12  f13  f14  f15