{'fraction_non_alphanumeric': 0.030959670608108107, 'fraction_numerical': 0.010055954391891891, 'mean_word_length': 5.183939937979435, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Fairness via Explanation Quality: Evaluating Disparities in the Quality of Post hoc Explanations

As post hoc explanation methods are increasingly being leveraged to explain complex models in high-stakes settings, it becomes critical to ensure that the quality of the resulting explanations is consistently high across various population subgroups including the minority groups. For instance, it should not be the case that explanations associated with instances belonging to a particular gender subgroup (e.g., female) are less accurate than those associated with other genders. However, there is little to no research that assesses if there exist such group-based disparities in the quality of the explanations output by state-of-the-art explanation methods. In this work, we address the aforementioned gaps by initiating the study of identifying group-based disparities in explanation quality. To this end, we first outline the key properties which constitute explanation quality and where disparities can be particularly problematic. We then leverage these properties to propose a novel evaluation framework which can quantitatively measure disparities in the quality of explanations output by state-of-the-art methods. Using this framework, we carry out a rigorous empirical analysis to understand if and when group-based disparities in explanation quality arise. Our results indicate that such disparities are more likely to occur when the models being explained are complex and highly non-linear. In addition, we also observe that certain post hoc explanation methods (e.g., Integrated Gradients, SHAP) are more likely to exhibit the aforementioned disparities. To the best of our knowledge, this work is the first to highlight and study the problem of group-based disparities in explanation quality. In doing so, our work sheds light on previously unexplored ways in which explanation methods may introduce unfairness in real world decision making.


INTRODUCTION
As machine learning (ML) models are increasingly being deployed to make consequential decisions in domains such as healthcare, finance, and policy, there is a growing need to ensure interpretable to ML practitioners and other domain experts (e.g., doctors, policy makers). Only if these practitioners have a clear picture of the behavior of these models can they assess when and how much to rely on them, and detect systematic errors and potential biases in them [28]. However, the increasing complexity as well as the proprietary nature of predictive models make it challenging to understand these complex black boxes, thus motivating the need for tools that can explain them in a faithful and human interpretable manner. To this end, several techniques have been proposed to explain complex models in a post hoc fashion.
Owing to their generality, post hoc explanation methods are increasingly being used to explain a number of complex models in high stakes domains such as medicine, finance, law, and science [30,36,76]. Therefore, it becomes critical to ensure that the explanations generated by these methods are of high quality. Prior research has studied several notions of explanation quality, including fidelity, stability, consistency, and sparsity [49,56,65,80]. In this work, we focus on disparities in the quality of explanations corresponding to different population subgroups.
To illustrate, consider a healthcare setting in which a predictive model is being used to aid doctors in diagnosing diseases ( Figure 1). Suppose the model incorrectly relies on spurious features (e.g., hospital name, zip code) to make predictions. Let us assume that post hoc explanations generated by a state-of-the-art method are being provided to doctors to explain each model prediction. Further, explanations corresponding to men are accurate, and rightly capture that the underlying model is relying on spurious features. Therefore, doctors who examine these explanations are likely to mistrust the model in the case of male patients and make better decisions by employing their own discretion. On the other hand, explanations for women are comparatively less accurate, and are incorrectly suggesting that the underlying model is relying on relevant features (e.g., fever, heart rate). So, doctors may trust model predictions in the case of female patients due to the misleading explanations, and may therefore make incorrect decisions leading to unfairness (women are more likely to be incorrectly diagnosed than men) in the overall decision making process.
The above scenario, while stylized, demonstrates how disparities in the quality of explanations may induce unfairness in real world decision making, and negatively impact the outcomes of vulnerable populations. Therefore, it is important to assess the extent to which such group-based disparities occur in the quality of the explanations output by state-of-the-art methods. However, there is little to no research that focuses on this critical problem.
In this work, we initiate the study of group-based disparities in explanation quality. More specifically, we make the following key contributions: • We formulate the problem of detecting group-based disparities in explanation quality. • We propose a novel evaluation framework which can quantitatively measure disparities in the quality of explanations output by state-of-the-art methods. We show an example of how to use this framework and some initial directions for inquiry using existing common metrics-specifically, fidelity disparity, stability disparity, consistency disparity, and sparsity disparity. • We leverage the aforementioned framework to carry out a rigorous empirical analysis with six state-of-the-art post hoc explanation methods, three real world datasets, and two different model classes (linear models and deep neural networks) to study if and when group-based disparities in explanation quality arise.
Results from our empirical analysis indicate that disparities in explanation quality are more likely to occur when the models being explained are complex and highly non-linear. Furthermore, we also observed that post hoc explanation methods such as Integrated Gradients and SHAP are more likely to generate explanations that are prone to the aforementioned disparities. Our analysis also revealed that group-based disparities are most prominent in case of two notions of explanation quality-namely, fidelity and sparsity.
Our findings not only shed light on the previously unexplored problem of group-based disparities in explanation quality, but also pave the way for rethinking the design and development of explanation methods in a way that such disparities can be minimized.

RELATED WORK
This work builds on extensive work in explainable machine learning. Crucially, the usage of explanations is often motivated by higherstakes settings, where the interpretation of the explanation has influence on how a human chooses to interact with the model. Common use-cases for explanations across a variety of stakeholders include model debugging and improvement, monitoring, building confidence, transparency, and auditing [16,34,72]. Most notably, explanations can be used for informing and justifying downstream actions for decision-making. For instance, based on the contents of the explanation, an individual might decide to follow the model's recommendation (or not); or, in a different application setting, an individual might decide to contest the result of a model's decision (or not). The related literature described in this section ultimately works towards these broader goals of usefulness for explanations, and it is these broader goals that our concerns about performance disparity ultimately address.

Inherently Interpretable Models and Post hoc Explanations
Many approaches learn inherently interpretable models such as rule lists [75,78], decision trees and decision lists [46], and others [17,18,42,50]. However, complex models such as deep neural networks often achieve higher accuracy than simpler models [59]. Thus, there has been significant interest in constructing post hoc explanations to understand their behavior. To this end, several techniques have been proposed in recent literature to construct post hoc explanations of complex decision models. For instance, LIME, SHAP, Anchors, BayesLIME, and BayesSHAP [51,59,60,65] are considered perturbation-based local explanation methods because they leverage perturbations of individual instances to construct interpretable local approximations (e.g., linear models). On the other hand, methods such as Gradient * Input, SmoothGrad, Integrated Gradients, and GradCAM [63,64,68,71] are referred to as gradient-based local explanation methods since they leverage gradients computed with respect to input features of individual instances to explain individual model predictions. An alternate class of methods referred to as global explanation methods attempt to summarize the behavior of black-box models as a whole rather than in relation to individual data points [12,44]. A more detailed treatment of this topic is provided in other comprehensive survey articles [9,23,33,48,53]. In contrast to the aforementioned works, which propose novel methods to explain models and their predictions, our work focuses on understanding group-based disparities in the quality of explanations generated by state-of-the-art post hoc local explanation methods.
Evaluating Explanations Prior research has proposed several metrics to determine if an explanation is reliable. An extensive survey of metrics for evaluating explanation methods can be found in Zhou et al. [80], who propose two high-level goals of explanation methods: interpretability (the clarity, simplicity, and broadness of the explanations), and fidelity (the completeness and soundness of explanations) [19,32]. Liu et al. [49] provide a synthetic benchmark for explanation evaluation which includes implementations of several metrics, as well as a discussion of how to choose metrics for evaluation. Furthermore, several prior works proposed different metrics for evaluating various aspects of explanation quality, such as fidelity, stability, consistency, and sparsity [6,31,35,44,49,56,65,80]. Recent research further leveraged the aforementioned properties and metrics to theoretically and empirically analyze the behavior of popular post hoc explanation methods [2,3,6,20,26,31,47,67]. In addition to quantitative metrics, human-grounded evaluation approaches emphasize how human users perceive and utilize explanations [28]. For example, Lakkaraju and Bastani [43] carry out a user study to understand if misleading explanations can fool domain experts into deploying racially biased models, while Kaur et al. [38] find that explanations are often over-trusted and misused. Similarly, Poursabzi-Sangdeh et al. [58] find that supposedlyinterpretable models can lead to a decreased ability to detect and correct model mistakes, possibly due to information overload. Jesus et al. [37] introduce a method to compare explanation methods based on how subject matter experts perform on specific tasks with the help of explanations. Lage et al. [41] use insights from rigorous human-subject experiments to inform regularizers used in explanation algorithms. Though our work does not involve human subject study, we see this line of human-centered investigation as a critical complement to our quantitative evaluation framework.

Limitations and Vulnerabilities of Post hoc Explanations
The aforementioned notions of explanation quality and the corresponding evaluation metrics were also leveraged to analyze the behavior of post hoc explanation methods and their vulnerabilitiese.g., Ghorbani et al. [31] and Slack et al. [66] demonstrated that methods such as LIME and SHAP may result in explanations that are not only inconsistent and unstable, but also prone to adversarial attacks. Furthermore, Lakkaraju and Bastani [43] and Slack et al. [66] showed that explanations which do not accurately represent the importance of sensitive attributes (e.g., race, gender) could potentially mislead end users into believing that the underlying models are fair when they are not [4,43,66]. This, in turn, could lead to the deployment of unfair models in critical real world applications. There is also some discussion about whether models which are not inherently interpretable ought to be used in high-stakes decisions at all. Rudin [61] argues that post hoc explanations tend to be unfaithful to the model to the extent that their usefulness is severely compromised. While this line of work demonstrate different ways in which explanations could potentially induce inaccuracies and biases in real world applications, they do not focus on analyzing group-based disparities in explanation quality, which is our focus.
The Intersection of Fairness and Explainability While fairness and explainability are often described as qualities which "responsible" models ought to have simultaneously [62,73], there is surprisingly little work exploring the intersection of the two. Begley et al. [13] extend SHAP as a tool to explain observed (un)fairness. Several recent works [1,55,69,70] propose methods to make specific algorithms simultaneously fair and explainable. As fairness is often explicitly referred to as a justification or motivation for the use of explanation methods [14,21,72], it is critical to also evaluate explanations from the perspective of fairness. Here, we make a distinction between whether an explanation accurately portrays the fairness of the underlying model, and whether the explanation method performs equally well on all groups. Several works address the former question-e.g., Aïvodji et al. [4] introduce the notion of "fairwashing, " in which a rule-based explanation method can be adversarially constructed such that an unfair model decision can be rationalized. Meanwhile, Slack et al. [66] illustrate how a model can be adversarially constructed such that LIME and SHAP hide the explicitly unfair behavior of the model. Alikhademi et al. [5] finds that existing explainability tools are ill-suited for evaluating fairness performance. Finally, Dodge et al. [25] conduct a human study investigating how explanations can be used to interpret the fairness performance of a model. While the aforementioned works address the question of whether an explanation accurately portrays the fairness of the underlying model, we investigate whether the explanation method performs equally well on all the groups. In concurrent work, Balagopalan et al. [10] are motivated by a similar question. Notably, they focus solely on fidelity, and empirically validate our findings of disparity in existing explanation methods.

OUR FRAMEWORK 3.1 Preliminaries and Notation
Let us consider a dataset D = {(x (1) , (1) ), . . . , (x ( ) , ( ) )} where x ( ) ∈ R is a vector of feature values and represents an instance in the data, and ( ) ∈ Y is the corresponding class label. Since our goal is to unearth group-based disparities in the quality of post hoc explanations, we consider the case where the dataset D comprises of some feature which corresponds to a discrete sensitive attribute (e.g., gender, race, age).
Let : R → Y be a predictive model which can make predictions on instances in D. Let E : (x, ) ↦ → w ∈ R be a local explanation method which takes as input an instance x and the predictive model , and outputs a vector of feature importances w. Note that the -th element of this vector (i.e., ) represents how important the -th feature is to the prediction (x). Finally, let M : (x, , E) ↦ → R denote a metric which measures some facet of the quality (e.g., fidelity or stability etc.) of a given local explanation E (x, ).

Metrics for Evaluating Explanations
In their survey, Zhou et al. [80] highlight two characteristics of a high-quality explanation: first, how well the explanation approximates the model, and second, human-understandability of the explanation. We focus on three metrics-fidelity, stability, and consistency-that measure the accuracy of the explanation's approximation, as well as a fourth-sparsity-that measures how easily the explanation is understood by a user.
3.2.1 Fidelity. Perhaps the most natural question when considering explanation quality is the extent to which it accurately represents the underlying decision-making process. In other words, are the "important features" designated in the explanation truly important features for the model? Explanations that accurately designate the important features to the underlying model are said to have high fidelity.
In our framework, we select two measures of fidelity: ground truth fidelity, which is applicable to models that represent their features as linear weights comparable with explanations, and prediction gap fidelity, which is applicable to a wider range of models like neural networks.
The ground truth fidelity metric applies solely to machine learning models which inherently encode some notion of feature importance as a vector of weights, such as in the coefficients of a logistic regression model. While therefore limited in practical applicability across most models, this metric is a useful baseline for determining whether an explanation method is correct when there exists a ground truth definition of features and their important. Specifically, suppose we have ground truth weights , such as the weights of a logisitic regression model. The ground truth fidelity metric is defined as the percent of the top features from explanation w which are also in the top features from . Definition 3.1. Let ∈ R be a vector denoting the importance of features in a model , where > ′ implies that the th feature is more important in than the ′ th feature. Let top( , ·) be a function that returns the indices of the largest elements of a given input vector. Then, given local explanation w = E (x, ) the ground truth fidelity for a given value of is defined as follows: The above definition requires that some agreed-upon measure of feature importance is available. For models that do not have such a quantity, such as deep neural networks, we measure prediction gap fidelity. The intuition behind this fidelity metric is that the model's prediction should not change substantially if minor alterations are made to features that are not important. If the prediction does change substantially, then this indicates that the explanation failed to capture features that were in fact important to the prediction, and is thus low fidelity. This follows a similar intuition to saliency in Dabkowski and Gal [24], and insertion/deletion in Petsiuk et al. [56]. This metric is applicable for probabilistic classifiers (x) = (ℎ(x)), where ℎ : R → [0, 1] generates predicted probabilities and : [0, 1] → {0, 1} maps the probability to a classification. The most important features of the model for x are identified from w = E (x, ). Then, a small amount of Gaussian noise with variance is added to all features of x outside the top to producex, and the difference between ℎ(x) and ℎ(x) is taken. The "prediction gap" for the single x is the expected value of this difference; practically, the procedure of adding noise and recording the gap in prediction-is repeated a total of times, and the "prediction gap" is computed as the average of all gaps.
Then prediction gap fidelity is defined as follows: We approximate prediction gap fidelity empirically. We generatex times, producing {x | 1 ≤ ≤ }. Then predication gap fidelity is approximated as follows:

Stability.
The idea that similar points should receive similar explanations-and the observation that popular explanations often do not satisfy this property-has been discussed extensively (common terms for this principle, in addition to stability, include robustness and insensitivity) [7,15,77]. Alvarez-Melis and Jaakkola [7] note that instability has been observed even when the underlying model is stable-calling into question the veracity of those explanations. In other words, given a set of differing explanations for similar points, it is not credible that those explanations are all correct.
To measure an explanation's stability at a point x, we add some noise to x to generate similar points, calculate explanations for those similar points, then take the average L1 distance between x's explanation and those of the similar points. This is similar to the definition of prediction gap fidelity, except now we add noise to all features and measure the difference in explanations rather than model predictions. This definition exactly mirrors the definition of stability in Yeh et al. [77] and others [6,15]. Then instability is defined as follows: We approximate instability empirically. We generatex times, producing {x | 1 ≤ ≤ }. Then instability is approximated as

Consistency.
Consistency captures the intuition that if an explanation for a single point is calculated multiple times, each of the calculated explanations should be similar. Inconsistent explanations for the same input x suggest that these explanations may be unreliable. This metric is motivated by concerns raised by empirical observations of the performance of many stochastic post-hoc explanation methods-requesting many explanations for the same point often resulted in drastically different explanations [6,45,45,65,79]. If the same point may result in a variety of very different explanations, it is unlikely that any individual explanation is correct, indicating a low approximation quality.
To measure consistency for a point x, we calculate several explanations for that same point; then, we take the average L1 distance between the first explanation and each of the new explanations.
Definition 3.4. Let an explanation method E be stochastic, such that E indicates an explanation generated with a random seed . The empirical inconsistency is defined as follows:

Sparsity.
Finally, we are also interested in how easily an explanation can be understood and interpreted by users. Extensive previous work drawing from cognitive psychology has discussed the question of whether a given explanation is actually understood by a human user [39,54,74]. One common theme is complexity leading to higher cognitive load. Interpretability is measured in Bhatt et al. [15] as the entropy of the explanation, where equal attribution to all features is considered to be the least-interpretable. In Lakkaraju et al. [44], interpretability is measured by counting discrete concepts. We measure sparsity by counting the number of features with an attributed importance greater than a threshold . Definition 3.5. Let be the threshold for which a feature importance is considered significant. Given local explanation w = E (x, ) and threshold parameter , complexity is defined as follows:

Measuring Disparity
For any given metric, we can empirically estimate disparity as follows. For each dataset, we partition the data into groups according to the sensitive attribute. In the datasets used for our experiments, we split the data into group 0 (majority group) and group 1 (minority group), though our framework is generic enough to handle multi-valued sensitive attributes. For each of these groups, we compute the mean values of the metric by averaging the metric values across instances in the respective groups. If there is a significant difference (as measured by a hypothesis test) between the mean metric values for each group, then there is a disparity in the quality of the explanations for that metric.

Consequences of Disparity
Though our broad proposal is to measure disparities in explanation quality in general, in this section we discuss some possible consequences of disparity in the specific metrics defined in the previous sections. We emphasize that disparity in additional metrics-or even different implementations of the same metrics-may have implications beyond what is discussed here; furthermore, this discussion is meant primarily as a preliminary exploration, and more tailored study (for example, with human subjects) for each case described here is warranted.
3.4.1 Fidelity Disparity. A significant disparity in explanation fidelity, or the closeness of an explanation to the model's actual behavior, can have concrete ramifications. For example, as in Figure 1, a domain expert such as doctor could make systematically worse decisions for a group of people because of a disparity in explanation quality.

Stability Disparity.
Like disparity in fidelity, disparity in stability can be consequential. Suppose the machine learning model that a doctor uses an explanation method that is not equally stable across groups. This disparity might lead to them rely, rightly or wrongly, more often on the model being explained for one group or the other. Measuring disparity in stability at development time, meanwhile, is important not just to prevent the above scenario, but to potentially identify problems with the model itself. Dooley et al. [27] observed that models can be unstable across groups; disparity in explanation stability may be a warning sign.

Consistency Disparity.
Disparity in consistency is also problematic when considering the user's interpretation of the explanations. A user may themselves be testing the explanation method by requesting an explanation multiple times, choosing to use the explanation for a given point only if they observe that the multiple explanations they received for that point are similar. If consistency disparity exists, therefore, the user may end up using explanations to aid decisionmaking much more often for one group than another. Returning to the example of the clinical setting, this may ultimately result in meaningfully disparate diagnoses.

Sparsity Disparity.
Unlike the three metrics discussed previously, disparity in sparsity across demographic groups is not inherently problematic. In fact, it may reflect the model's true behavior, e.g., if the model has a more complex decision boundary for one group than another, or if the model does rely more features for one group than another. Checking for disparity in sparsity, therefore, can be informative about the underlying model. One possible adverse consequence of disparity in sparsity is that there may be higher cognitive load when examining explanations for one group than another. However, if it is the case that the model truly is relying on more information to make predictions for one group, artificially ensuring equal sparsity may result in a substantial tradeoff with fidelity.

EMPIRICAL ANALYSIS
In this section, we demonstrate how to employ our explanation evaluation framework and interpret its outputs. First we generate a range of post-hoc explanations for various data and model combinations. We apply our evaluation framework to these explanations, computing the metrics of interest and analyzing them across data subgroups. Finally we examine the disparities that emerge, highlighting trends that may be of interest to practitioners.

Experimental Setup
Our evaluation framework is of particular importance in settings where explanation disparities emerge across protected attribute classes like sex and race. Accordingly, we choose the following benchmark datasets: German Credit [29], Student Performance [22,29], and COMPAS [8]. Though we acknowledge the limitations of COMPAS as a benchmark dataset [11] (and indeed, "fairness benchmarks" in general), we choose to evaluate and report explanation performance on it because we do not claim to successfully "achieve" any notion of fairness with respect to classification or explanation. We evaluate explanation disparities across one protected attribute in each dataset, namely sex in both German Credit and Student Performance and race in COMPAS. Datasets are divided into training and testing sets via an 80/20 random stratified split with respect to the target label.

Models.
We investigate whether explanation disparities can vary based on the complexity of model being explained. We train both linear and non-linear models, Logistic Regression (LR) and a 3-layer neural network (NN) respectively. For the latter, we use 50, 100, and 200 nodes in each consecutive layer, ReLU activation, binary cross entropy loss, Adam optimizer, and 100 training epochs.

Explanation Methods.
We investigate whether different explanation methods can produce different disparities. Most post-hoc explanation methds can be categorized into two families of approaches. One approach generates explanations by constructing locally interpretable model approximations whereas the other generates explanations by analyzing the behavior of the model's gradient. We choose LIME [59], SHAP [51], and MAPLE [57] from the former and SmoothGrad [68], Integrated Gradients (IntGrad) [71], and Vanilla Gradients (VanGrad) [64] from the latter. We employ the authors' implementations for LIME [59] and MAPLE [57]. We use Captum library [40] implementations of SHAP (sample size 1000), IntGrad, and VanillaGrad. We use default hyperparameter settings throughout unless otherwise specified. We implement SmoothGrad in PyTorch with standard normal noise and sample size 1000.

Metric Hyperparameters.
For both ground truth fidelity and prediction gap fidelity we fix = 5. Additionally, for prediction gap fidelity we fix = 1000 and = 0.1. For both instability and inconsistency we fix = 5. For sparsity we fix = 0.01.

Results & Insights
We compute set averages 0 and 1 for each metric, model, explanation method, and dataset combination. To identify when explanation disparity is statistically significant, we perform Mann-Whitney U tests [52] on each pair of 0 and 1 distributions and report the resulting p-values in Table 1. Of the 162 experimental combinations analyzed, 18.5% exhibited statistically significant explanation disparity. In Table 2 we aggregate these instances across metrics, counting the number of times significant explanation disparity occurs in each of the 36 explanation, model, and dataset combinations. Across these 36 combinations, explanation disparity occurs in at least one metric 56% of the time. Next we analyze explanation disparity across each of our framework metrics and summarize additional aggregate trends. Table  1, significant prediction gap fidelity disparity occurred 30.6% of the time. Significant ground truth fidelity disparity also occurred 11.1% of the time. Across both ground truth fidelity disparity and prediction gap fidelity disparity, significant fidelity disparity occurred most often with SHAP explanations. For prediction gap fidelity disparity, which can be computed for both LR and NN models, first notice that the prediction gap values are larger across both groups 0 and 1 in the NN setting, indicating worse fidelity. Next notice that the majority of significant disparities occur in the more complex NN model setting (63.6%). In addition to being more frequently significant, prediction gap fidelity explanation disparity appears more severe in complex model settings. This is illustrated in Figure 2 by the wider gaps between 0 and 1 means in complex NN model settings when compared to their simpler LR counterparts. Moreover, even when disparities are not statistically significant, we notice more severe disparity in complex NN model settings when compared to their simpler LR counterparts. In these NN settings with observably larger disparity, the lack of statistical significance is possibly due to larger variances. These larger variances themselves suggest that prediction gap fidelity is a less robust metric on complex models. In general, the trends of fidelity disparity being more frequently significant and more severe in complex models align with our expectations of explanation methods that employ local linear approximations. Table 1 (d), significant stability disparity occurred 13.9% of the time and most frequently with VanillaGrad explanations. The majority (80%) of instances of significant explanation disparity occurred on the complex NN model setting. In Figure 3, we notice that instability in general, across both groups on the German Credit dataset. On the German Credit dataset, like with fidelity disparity, in Figure 3, we notice that even when disparities are not significant, wider gaps between 0 and 1 means in complex NN model settings when compared to their simpler LR counterparts, indicating more severe disparity. Similarly, larger variances are again observed on more complex models, suggesting stability is a less robust metric in these settings.     Table 1 across metrics, counting the number of times significant explanation disparity occurs in the explanation/model/dataset combinations. Across these 36 combinations, explanation disparity occurs in at least one metric 56% of the time. Table 1 (e), significant consistency disparity occurred in one setting, LIME on the LR model trained on the German Credit dataset. Yet as observed in Figure 4, though there exists a statistically significant consistency disparity between group 0 and group 1 in this setting, it is not obvious because both groups 0 and 1 have very low inconsistency in general. In fact the only settings where inconsistency is often larger across both groups is with MAPLE explanations.   Table 1 (c), sparsity disparity occurred 30.6% of the time and most frequently with IntGrad explanations. Notice that, in contrast with previously observed trends,  instances of significant sparsity disparity are split more evenly between the NN and LR model settings, 45.5% and 54.5% respectively.

Sparsity disparity. As reported in
However, in Figure 5 we notice that the variance in average sparsity is typically larger in NN settings than in LR counterparts, similarly to those for fidelity disparity, perhaps contributing to the fewer instances of statistically significant disparity.

Additional aggregate trends.
To identify which explanation methods exhibit the most explanation disparity across our experiments, we read Table 1 columnwise, noting that each explanation method is employed on 27 metric, model, and dataset combinations. IntGrad exhibited significant explanation disparity on 33.3% of the combinations, followed by SHAP (29.6%), VanillaGrad (22.2%), LIME (11.1%), SmoothGrad (11.1%), and MAPLE (3.7%). In Table 2 we also notice that IntGrad is the only explanation method with nonzero entries in every row. This means that IntGrad exhibits significant explanation disparity in at least one dimension in every model/dataset setting tested and is the only explanation method that does so. As detailed in Section 5, investigating the underlying mechanisms in explanation methods that give rise to these disparities is a critical direction for future research.
To determine whether explanations disparity occurs across multiple metrics at once, we refer to Table 2. Of the 20 explanation, model, and metric settings in Table 2 with non-zero entries, indicating significant explanation disparity is observed at least once, in one half disparities occur in only one metric, and in the other half disparities co-occur in two metrics. This behavior suggests that practitioners should not rely on disparity in one metric to serve as a proxy for disparity along all metrics.

DISCUSSION
In this work, we proposed an evaluation framework for unearthing disparities in post hoc explanation quality. To the best of our knowledge, this work is the first to study the problem of group-based disparities across a variety of metrics for explanation quality. This is especially important when the black-box model is making predictions about humans, given the consequential use of explanations in this context: explanations are often used in conjunction with a model in order to shape decisions, and disparity in explanation quality across demographic groups may be yet another way that adverse downstream outcomes become baked into the technical system. There are many cases where disparity in performance does not occur. However, the prevalence of significant disparity throughout datasets, metrics, and methods in our results suggest that measuring disparity in explanation quality is still critical: more than half of all dataset-model combinations exhibited disparity.
Our work has lasting implications for several stakeholders. For researchers working on novel explanation methods, it may be useful to apply our framework on a test suite of datasets and models to identify whether the explanation method is susceptible to disparity in general, and if so, with respect to what metrics. For practitioners who are developing application-specific models and explanation methods, our framework may be useful to identify whether the resulting explanations exhibit quality disparities in the context of the application.
If disparity is in fact identified, what actions should stakeholders take? As discussed in Section 3, disparity in fidelity and consistency always indicates that the explanation method is amplifying or generating problems. Disparity in stability and sparsity, on the other hand, may either indicate an issue with the explanation method, or an issue with the underlying model to be explained. Further research is necessary in order to provide more prescriptive next steps. Our work points to several follow-up research directions: First, what conditions give rise to observed disparity? Are there ways in which we can characterize datasets, models, or explanation methods such that we can more systematically identify the causes of measured explanation disparity? Second, human-grounded evaluations of our setting are necessary. Given significant explanation disparities, how is human decision making affected? Finally, it would be interesting to develop explanation methods which are less susceptible to such disparities while still maintaining overall utility.