{'fraction_non_alphanumeric': 0.04040511161779522, 'fraction_numerical': 0.007980274669918871, 'mean_word_length': 4.965364542147714, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Evaluating Explainability for Graph Neural Networks

As post hoc explanations are increasingly used to understand the behavior of graph neural networks (GNNs), it becomes crucial to evaluate the quality and reliability of GNN explanations. However, assessing the quality of GNN explanations is challenging as existing graph datasets have no or unreliable ground-truth explanations for a given task. Here, we introduce a synthetic graph data generator, ShapeGGen, which can generate a variety of benchmark datasets (e.g., varying graph sizes, degree distributions, homophilic vs. heterophilic graphs) accompanied by ground-truth explanations. Further, the flexibility to generate diverse synthetic datasets and corresponding ground-truth explanations allows us to mimic the data generated by various real-world applications. We include ShapeGGen and several real-world graph datasets into an open-source graph explainability library, GraphXAI. In addition to synthetic and real-world graph datasets with ground-truth explanations, GraphXAI provides data loaders, data processing functions, visualizers, GNN model implementations, and evaluation metrics to benchmark the performance of GNN explainability methods.


Introduction
As graph neural networks (GNNs) are being increasingly used for learning representations of graphstructured data in high-stakes applications, such as criminal justice [1], molecular chemistry [2], and biological networks [3,4], it becomes critical to ensure that the relevant stakeholders can understand and trust their functionality. To this end, previous work developed several methods to explain predictions made by GNNs [5][6][7][8][9][10][11][12][13]. With the increase in newly proposed GNN explanation methods, it is critical to ensure their reliability. However, explainability in graph machine learning is still a nascent area and lacks both standardized evaluation strategies as well as reliable data resources to evaluate, test, and compare GNN explanations [14]. While several works have acknowledged this difficulty, they tend to base their analysis on specific real-world [2] and synthetic [15] datasets with limited ground-truth explanations. In addition, relying on these datasets and associated ground-truth explanations is insufficient as they are not indicative of diverse real-world applications [14]. To this end, developing a broader ecosystem of data resources for benchmarking state-of-the-art GNN explainers can support explainability research in GNNs.
A comprehensive data resource to correctly evaluate the quality of GNN explanations will ensure their reliability in high-stake applications. However, the evaluation of GNN explanations is a growing research area with relatively little work, where existing approaches mainly leverage ground-truth explanations associated with specific datasets [2] and are prone to several pitfalls (as outlined by Faber et al. [15]). Further, there could be multiple underlying rationales (redundant/nonunique explanations) generating the correct class labels, and a trained GNN model may only capture one or an entirely different rationale. In such cases, evaluating the explanation output by a state-of-the-art method using the ground-truth explanation is incorrect because the underlying GNN model does not rely on that ground-truth explanation. In addition, even if there is a unique ground-truth explanation that generates the correct class labels, the GNN model trained on the data could be a weak predictor which uses an entirely different rationale for making predictions.
The ground-truth explanation should also not be used to assess post hoc explanations of such models. Lastly, the ground-truth explanations corresponding to some of the existing benchmark datasets are not good candidates for reliably evaluating explanations as they can be recovered using trivial baselines (e.g., random node or edge as explanation). The above discussion highlights a clear need for general-purpose data resources which can evaluate post hoc explanations reliably across diverse real-world applications. While several diverse benchmark datasets (e.g., Open Graph Benchmark (OGB) [16], GNNMark [17], GraphGT [18], MalNet [19], Graph Robustness Benchmark (GRB) [20], Therapeutics Data Commons [21], and EFO-1-QA [22]) and programming libraries for deep learning on graphs (e.g., Dive Into Graphs (DIG) [23], Pytorch Geometric (PyG) [24], and Deep Graph Library (DGL) [25]) in graph machine learning literature exist, they are mainly used to only benchmark the performance of GNN predictors and are not suited to evaluate the correctness of GNN explainers because they do not capture ground-truth explanations.
Here, we address the above challenges by introducing a general-purpose data resource that is not prone to the ground-truth pitfalls (e.g., redundant explanations, weak GNN predictors, trivial explanations, etc.) and can cater to diverse real-world applications. To this end, we present SHAPEGGEN ( Figure 2), a novel and flexible synthetic XAI-ready (explainable artificial intelligence ready) dataset generator which can automatically generate a variety of benchmark datasets (e.g., varying graph sizes, degree distributions, homophilic vs. heterophilic graphs) accompanied by ground-truth explanations. SHAPEGGEN ensures that the generated ground-truth explanations are not prone to the pitfalls described in Faber et al. [15]), such as redundant explanations, weak GNN predictors, and trivially correct explanations. SHAPEGGEN allows us to evaluate the goodness of any given explanation (e.g., node feature-based, node-based, edge-based) across diverse real-world applications by seamlessly generating synthetic datasets that can mimic the properties of real-world data in various domains.
We incorporate SHAPEGGEN and several other synthetic and real-world graphs into GRAPHXAI, a general-purpose framework to systematically evaluate the quality of GNN explanations. GRAPHXAI also provides a broader ecosystem (Figure 1) of data loaders, data processing functions, visualizers, and a set of evaluation metrics (e.g., accuracy, faithfulness, stability, fairness) to reliably benchmark the quality of any given GNN explanation (node feature-based or node/edge-based). We leverage various synthetic and real-world datasets and evaluation metrics from GRAPHXAI to empirically assess the quality of explanations output by eight state-of-the-art GNN explanation methods. Across many GNN explainers, graphs, and graph tasks, we observe that state-of-the-art GNN explainers fail on graphs with large ground-truth explanations (i.e., explanation subgraphs with a higher number of nodes and edges) and cannot produce explanations that preserve fairness properties of underlying GNN predictors.

Results
To evaluate the utility and flexibility of GRAPHXAI, we show how GRAPHXAI enables a systematic benchmarking analysis of eight state-of-the-art GNN explainers on both SHAPEGGEN (in Methods section) and real-world graph datasets. We also explore the utility of the SHAPEGGEN generator to benchmark GNN explainers w.r.t. graphs with: i) homophilic vs. heterophilic, ii) small vs.
large, and iii) fair vs. unfair ground-truth explanations. We start with an outline of experimental setup, including giving details about performance metrics, GNN explainers, and underlying GNN predictors, and proceed with a discussion of benchmarking results.
A. Experimental setup GNN Explainers. The GRAPHXAI defines an Explanation class capable of storing multiple types of explanations produced by GNN explainers and provides a graphxai.BaseExplainer class that is a parent class to all explanation methods in our current release. We incorporate eight GNN explainability methods, including gradient-based: Grad [26], GradCAM [10], GuidedBP [5], Integrated Gradients [27]; perturbation-based: GNNExplainer [13], PGExplainer [9], SubgraphX [28]; and surrogate-based methods: PGMExplainer [12]. Finally, following Agarwal et al. [14], we consider random explanations as a reference: 1) Random Node Features, a node feature mask defined by an d-dimensional Gaussian distributed vector; 2) Random Nodes, a 1 × n node mask is randomly sampled using a uniform distribution, where n is the number of nodes in the enclosing subgraph; and 3) Random Edges, an N × N edge mask drawn from a uniform distribution over a node's incident edges. Implementation Details. We use a three-layer GIN model [29] as our GNN predictor f for all our experiments. We use a model comprising of three GIN convolution layers with ReLU non-linear activation function and a fully-connected linear classification layer with Softmax activations. The hidden dimensionality of the layers is set to 16. We follow an established approach for generating explanations [7,14] and use reference algorithm implementations. We select top-k (k = 25%) important nodes, node features, or edges, and use them to generate explanations for all graph explainability methods. For training GIN models, we use an Adam optimizer with a learning rate of 1 × 10 −2 , weight decay of 1 × 10 −5 , and the number of epochs to 1000. For the GNN explanation model, we set hyperparameters following the authors' recommendations.
Performance Metrics. In addition to the synthetic and real-world data resources, we consider four broad categories of performance metrics: i) Graph Explanation Accuracy (GEA); ii) Graph Explana-tion Faithfulness (GEF); iii) Graph Explanation Stability (GES); and iv) Graph Explanation Fairness (GECF, GEGF) to evaluate the explanations on the respective datasets. In particular, all evaluation metrics leverage predicted explanations, ground-truth explanations, and other user-controlled parameters, such as top-k features. Our GRAPHXAI package implements these performance metrics and additional utility functions within graphxai.Metrics class. Figure 3 shows a code snippet for evaluating the correctness of output explanations for a given GNN prediction in GRAPHXAI.
i) Graph Explanation Accuracy (GEA). We report graph explanation accuracy as an evaluation strategy that measures an explanation's correctness using the ground-truth explanation M g . In ground-truth and predicted explanation masks, every node, node feature, or edge belongs to {0, 1}, where '0' means that an attribute is unimportant and '1' means that it is important for the model prediction. To measure accuracy, we use Jaccard index [30] between the ground-truth M g and predicted M p : where TP denotes true positives, FP denotes false positives, and FN indicates false negatives. Most synthetic-and real-world graphs have multiple ground-truth explanations. For example, in the MUTAG dataset [31], carbon rings with both NH 2 or NO 2 chemical groups are valid explanations for the GNN model to recognize a given molecule as mutagenic. For this reason, the accuracy metric must account for the possibility of multiple equally valid explanations existing for any given prediction. Hence, we define ζ as a set of all possible ground-truth explanations, where |ζ| = 1 for graphs having a unique explanation. Therefore, we calculate GEA as: Here, we can calculate GEA using predicted node feature, node, or edge explanation masks. Finally, Equation 1 quantifies the accuracy between the ground-truth and predicted explanation masks, and higher values mean that a predicted explanation is more likely to match the ground-truth explanation.
ii) Graph Explanation Faithfulness (GEF). We extend existing faithfulness metrics [2,14] to quantify how faithful explanations are to an underlying GNN predictor. In particular, we obtain the prediction probability vectorŷ u using the GNN, i.e.ŷ u = f (S u ), and using the explanation, i.e.
y u = f (S u ), where we generate a masked subgraph S u by only keeping the original values of the top-k features identified by an explanation, and get their respective predictionsŷ u . Finally, we compute the graph explanation unfaithfulness metric as: where Kullback-Leibler (KL) divergence score quantifies the distance between two probability distributions, and the "||" operator indicates "divergence". Note that Equation 3 is a measure of the unfaithfulness of the explanation. So, higher values indicate a higher degree of unfaithfulness.
iii) Graph Explanation Stability (GES). Formally, an explanation is defined to be stable if the explanation for a given graph and its perturbed counterpart (generated by making infinitesimally small perturbations to the node feature vector and associated edges) are similar [14,32]. We measure graph explanation stability w.r.t. the changes in the model behavior. In addition to similar output labels for S u and the perturbed S u , we employ a second level of check where the difference between model behaviors for S u and S u is bounded by an infinitesimal constant δ, i.e. ||L Su −L S u || p ≤ δ.
Here, L(·) refers to any form of model knowledge like output logitsŷ u or embeddings z u . We compute graph explanation instability as: where D(·) represents the cosine distance metric, M p Su and M p S u are the predicted explanation masks for S u and S u , and B(·) represents a δ-radius ball around S u for which the model behavior is same. Equation 4 is a measure of instability, and higher values indicate a higher degree of instability. iv) Graph Explanation Fairness. To quantify graph explanation fairness, we report two metrics: counterfactual fairness mismatch [14] and group fairness mismatch [14].
iv-a) Counterfactual Fairness Mismatch. To measure counterfactual fairness, we verify if the explanations corresponding to S u and its counterfactual counterpart (where the protected node feature is modified) are similar (dissimilar) if the underlying model predictions are similar (dissimilar). We calculate counterfactual fairness mismatch as: where, M p and M p s are the predicted explanation mask for S u and for the counterfactual counterpart of S u . Note that we expect a lower GECF score for graphs having weakly-unfair ground-truth explanations because explanations are similar for both original and counterfactual graphs, whereas, for graphs with strongly-unfair ground-truth explanations, we expect a higher GECF score as explanations change when we modify the protected attribute. iv-b) Group Fairness Mismatch. We measure group fairness mismatch as: whereŷ K andŷ Eu K are predictions for a set of K graphs using the original and the important features identified by an explanation, respectively, and SP is the statistical parity. Finally, Equation 6 is a measure of group fairness mismatch of an explanation where higher values indicate that the explanation is not preserving group fairness.

B. Evaluation and Analysis of GNN Explainability Methods
Next, we discuss experimental results that answer key questions with respect to synthetic and real-world graphs and different kinds of ground-truth explanations.
Benchmarking GNN Explainers on Synthetic and Real-World Graphs. We evaluate performance of GNN explainers on SHAPEGGEN and molecular datasets using metrics described in the experimental setup. Results in Tables 1-4 show that, while no explanation method performs well across all properties, across different SHAPEGGEN node classification datasets (Table 1), gradient-based methods, such as GradCAM and GuidedBP, on average outperform other methods.
In particular, we observe that Grad produces most faithful explanations and, on average, outperforms other methods by 61.47%. Whereas, PGExplainer generates the least unstable explanations and, on average, achieves 56.05% more stability as compared to other GNN explainers.
Analyzing Homophilic vs. Heterophilic Ground-Truth Explanations. We compare GNN explainers by computing graph explanation unfaithfulness scores on homophilic and heterophilic graphs. We find that GNN explainers produce 55.98% more faithful explanations when groundtruth explanations are homophilic than when ground-truth explanations are heterophilic (i.e., low unfaithfulness scores for light green bars in Figure 4). These results reveal an important gap in existing GNN explainers. Namely, existing GNN explainers fail to perform well on diverse types of graphs, such as homophilic, heterophilic, and attributed graphs. This observation, enabled by the flexibility of SHAPEGGEN generator, highlights an opportunity for future algorithmic innovation in GNN explainability.
Characterizing the Reliability of Graph Explainers to Smaller vs. Larger Ground-Truth Explanations. Results in Figure 5 show that explanations from current GNN explainers are faithful (i.e., lower GEF scores) to the underlying GNN models when ground-truth explanations are smaller, i.e., S='triangle'. On average, across all 8 GNN explainers, explanations generated on 'triangular' (smaller) ground-truth explanations are 59.98% less unfaithful than explanations for 'house' (larger) ground-truth explanations. We find that most explanation methods are unfaithful to graphs with large ground-truth explanations (i.e., low unfaithfulness scores for light purple bars in Figure 5).
However, the Grad explainer, on average, achieves 9.33% lower unfaithfulness on large ground-truth explanations compared to other explanation methods. This limited behavior of existing GNN explainers has not been previously known and highlights an urgent need for additional analysis of GNN explainers.
Examining Fair vs. Unfair Ground-Truth Explanations. The fairness results in Figure 6 show that GNN explainers do not preserve counterfactual fairness and are highly prone to producing unfair explanations. We note that for weakly-unfair ground-truth explanations (light red in Figure 6), explanations M p should not change as the label generating process is independent of the protected attribute but we observe high GECF scores for most explanation methods. On the contrary, for strongly-unfair ground-truth explanations, we observe that explanations from most GNN explainers fail to capture (i.e., low GECF scores for dark red bars in Figure 6) the unfairness enforced using the protected attribute and generate similar explanations even when we flip/modify the respective protected attribute. We see that GradCAM and PGEx explanations outperform other GNN explainers in preserving counterfactual explanations for weakly-unfair ground-truth explanations, whereas PGMEx, on average, preserves counterfactual fairness better than other explanation methods on strongly-unfair ground truth explanations.

Discussion
The main objective of GRAPHXAI is to provide a comprehensive and general purpose framework that evaluates the quality of the GNN explanations output by state-of-the-art methods. GRAPHXAI provides a broader ecosystem of data loaders, data processing functions, visualizers, real-world graph datasets with ground-truth explanations, and a set of evaluation metrics to benchmark the quality of any given GNN explanation. Most importantly, our framework introduces a novel and flexible synthetic dataset generator called SHAPEGGEN to automatically generate a variety of benchmark datasets and corresponding ground truth explanations which are not prone to traditional pitfalls. Our experimental results show that existing GNN explainers perform well on graphs with homophilic ground-truth explanations but perform considerably worse on heterophilic and attributed graphs. Across multiple GNN explanation methods, graphs, and types of downstream prediction tasks, we find that existing GNN explainers fail on graphs with larger (i.e., a higher number of nodes and edges) ground-truth explanations and cannot produce explanations that preserve the fairness properties of underlying GNN predictors. This finding points to the need for methodological innovation and a thorough analysis of future explainability methods across multiple dimensions of performance.
GRAPHXAI provides a flexible framework for evaluating GNN explanation methods and promotes reproducible and transparent research in the area. We maintain GRAPHXAI as a centralized library for evaluating GNN explanation methods and plan to keep adding newer datasets, explanation methods, diverse evaluation metrics, and visualization features to our existing framework. We anticipate that GRAPHXAI can help algorithm developers and practitioners in the broad field of graph representation learning develop and evaluate principled explainability techniques.

Methods
SHAPEGGEN is a key component of GRAPHXAI and serves as a synthetic dataset generator of XAI-ready graph datasets. It is founded in graph theory and designed to address the pitfalls (see Introduction) of existing graph datasets in the broad area of explainable AI. As such, SHAPEGGEN can facilitate the development, analysis, and evaluation of GNN explainability methods (see Results).
We proceed with the description of SHAPEGGEN data generator.
A. Notation where C is the total number of labels.

GNN Explainability Methods.
Given the prediction f (u) for node u made by a GNN model, a GNN explainer O outputs an explanation mask M p that provides an explanation of f (u). These explanations can be given with respect to node attributes M a ∈ R d , nodes M n ∈ R N , or edges M e ∈ R N ×N , depending on specific GNN explainer, such as GNNExplainer [13], PGExplainer [9], and SubgraphX [28]. For all explanation methods, we use a graph masking function MASK that outputs a new graph G = (V G , E G , X G ) by performing element-wise multiplication operation between the masks (M a , M n , M e ) and their respective attributes in the original graph G, e.g. A = A M e . Finally, we denote the ground-truth explanation mask as M g that is used to evaluate the performance of GNN explainers.

B. SHAPEGGEN Dataset Generator
We propose a novel and flexible synthetic dataset generator called SHAPEGGEN that can automatically generate a variety of benchmark datasets (e.g., varying graph sizes, degree distributions, homophilic vs. heterophilic graphs) accompanied by ground-truth explanations. Furthermore, the flexibility to generate diverse synthetic datasets and corresponding ground-truth explanations allows us to mimic the data generated by various real-world applications. SHAPEGGEN is a generator of XAI-ready graph datasets supported by graph theory and is particularly suitable for benchmarking GNN explainers and studying their limitations.
Flexible Parameterization of SHAPEGGEN. SHAPEGGEN has tunable parameters for data generation. By varying these parameters, SHAPEGGEN can generate diverse types of graphs, including graphs with varying degrees of class imbalance and graph sizes. Formally, a graph is generated as: G =SHAPEGGEN (S, N s , p, n s , K, n f , n i , s f , c f , φ, η, L), where: • S is the motif, defined as a subgraph, to be planted within the graph.
• N s denotes the number of subgraphs used in the initial graph generation process.
• p represents the connection probability between two subgraphs and controls the average shortest path length for all possible pairs of nodes. Ideally, a smaller p value for larger N s is preferred to avoid low average path length and, therefore, the poor performance of GNNs.
• n s is the expected size of each subgraph in the SHAPEGGEN generation procedure. For a fixed S shape, large n s values produce graphs with long-tailed degree distributions. Note that the expected total number of nodes in the generated graph G is N × n s .
• K is the number of distinct classes defined in the graph downstream task.
• n f represents the number of dimensions for node features in the generated graph.
• n i is the number of informative node features. These are features correlated to the node labels instead of randomly generated non-informative features. The indices for the informative features define the ground-truth explanation mask for node features in the final SHAPEGGEN instance. In general, larger n i results in an easier classification and explanation task, as it increases the node feature-level ground-truth explanation size.
• s f is defined as the class separation factor that represents the level of separation between class labels for each node. Higher s f corresponds to a stronger signal in the node features, i.e., if a classifier is trained only on the node features, a higher s f value would result in an easier classification task.
• c f is the number of clusters per class. A larger c f value increases the difficulty of the classification task with respect to node features.
• φ ∈ [0, 1] is the protected feature noise factor that controls the strength of correlation r between the protected features and the node labels. This value corresponds to the probability of "flipping" the protected feature with respect to the node's label. For instance, φ=0.5 results in zero correlation (r=0) between the protected feature and the label (i.e. complete fairness), φ=0 results in a positive correlation (r=1), and φ=1 results in a negative correlation (r= − 1) between the label and the protected feature.
• η is the homophily coefficient that controls the strength of homophily or heterophily in the graph.
• L is the number of layers in the GNN predictor corresponding to the GNN's receptive field. For the purposes of SHAPEGGEN, L is used to define the size of the GNN's receptive field and thus the size of the ground-truth explanation generated for each node.
This wide array of parameters for SHAPEGGEN allows for the generation of graph instances with vastly differing properties.
Generating Graph Structure. Figure 2 summarizes the process to generate a graph G = (V G , E G , X G ). SHAPEGGEN generates N s subgraphs that exhibit the preferential attachment property [34], which occurs in many real-world graphs. Each subgraph is first given a motif, i.e., a subgraph S = (V S , E S , X S ). A preferential attachment algorithm is then performed on base structure S, adding n (n ∼ Poisson(λ = n s − |V S |)) nodes by creating edges to nodes in V S . The Poisson distribution is used for determining the sizes of each subgraph used in the generation process, with λ = n s − |V S |, the difference between the number of nodes in the motif and the expected subgraph size n s . After generating a list of randomly-generated subgraphs S = {S 1 , . . . , S N }, edges are created to connect subgraphs, creating the structure of an instance of SHAPEGGEN. Subgraph connections and local subgraph construction is performed in such a way that each node in the final graph G has between 1 and K motifs within its neighborhood, i.e., More details on the SHAPEGGEN structure generation can be found in Algorithm 2.
Generating Labels for Prediction. A motif defined as a subgraph S = (V S , E S , X S ) occurs randomly throughout G, with the set S = {S 1 , . . . , S N }. The task on this graph is a motif detection problem, where each node has exactly 1, 2, or K motifs in its 1-hop neighborhood. A motif S i is considered to be within the neighborhood of a node v ∈ V G if any node s ∈ V S i is also in the neighborhood of v, i.e., if |V S i ∩ N v | > 0. Therefore, the task that a GNN predictor needs to solve is defined by: Generating Node Feature Vectors. SHAPEGGEN uses a latent variable model [35] to create node feature vectors and associate them with network structure. The latent variable model creates n i informative features for each node based on the node's generated label and also creates noninformative features as noise. Having non-informative/redundant features allows us to evaluate GNN explainers, such as GNNExplainer [13], that formulate explanations as node feature masks.
SHAPEGGEN can generate graphs with both homophilic and heterophilic ground-truth explanations. We optimize between homophily vs. heterophily by taking advantage of redundant node features, i.e., features that do not associate with the generated labels, and manipulate them appropriately based on a user-specified homophily parameter η. The magnitude of the η parameter determines the degree of homophily/heterophily in the generated graph. The algorithm for node features is given in Algorithm 3. While not every node feature in the feature vector is optimized for homophily/heterophily indication, we experimentally verified the cosine similarity between node feature vectors produced by Algorithm 3 reveals a strong homophily/heterophily pattern.
Finally, SHAPEGGEN can generate protected features to enable the study of fairness [1]. By controlling the value assignment for a selected discrete node feature, SHAPEGGEN introduces bias between the protected feature and node labels. The biased feature serves as a proxy for a protected feature, such as gender or race. The procedure for generating node features is outlined in NODEFEATUREVECTORS within Algorithm 1.
Generating Ground-Truth Explanations. In addition to generating ground-truth labels, SHAPEGGEN has a unique capability to generate unique ground-truth explanations. To accommodate diverse types of GNN explainers, every ground-truth explanation in SHAPEGGEN contains information on a) the importance of nodes, b) the importance of node features, and c) the importance of edges. This information is represented by three distinct masks defined over enclosing subgraphs of nodes v ∈ V G , i.e., the L-hop neighborhood around the node. We denote the enclosing subgraph of node v ∈ V G for a given GNN predictor with L layers as: Let motifs within this enclosing subgraph be: S v = (V Sv , E Sv , X Sv ) = S ∩ SUB(v). Using this notation, we define ground-truth explanation masks: a) Node Explanation Mask. Nodes in SUB(v) are assigned a value of 0 or 1 based on whether they are located within a motif or not. For any node v i ∈ V SUB(v) , we set 1 V Sv (v i ) = 1 if v i ∈ V Sv and 0 otherwise. This function 1 V Sv is applied to all nodes in the enclosing subgraph of v to produce an importance score for each node, yielding the final mask as:

c) Edge Explanation Mask.
To each e = (v i , v j ) ∈ E SUB(v) we assign a value of either 0 or 1 based on whether e connects any two nodes in SUB(v). The masking function is defined as follows: This function 1 E Sv is applied to all edges e ∈ E SUB(v) to produce ground-truth edge explanation The procedure to generate these ground-truth explanations is thoroughly described in Algorithm 1.

C. Datasets in GRAPHXAI
We proceed with a detailed description of synthetic and real-world graph data resources included in GRAPHXAI.

C.1 Synthetic Graphs
The SHAPEGGEN generator outlined in the Methods section is a l dataset generator that can be used to generate any number of user-specified graphs. In GRAPHXAI, we provide a collection of pre-generated XAI-ready graphs with diverse properties that are readily available for analysis and experimentation.
Base SHAPEGGEN Graphs (SG-BASE). We provide a base version of SHAPEGGEN graphs. This instance of SHAPEGGEN is homophilic, large, and contains house-shaped motifs for ground-truth explanations, formally described as G = SHAPEGGEN (S='house', N s =1200, p=0.006, n s =11, K=2, n f =11, n i =4, s f =0.6, c f =2, φ=0.5, η=1, L=3). The node features in this graph exhibit homophily, a property commonly found in social networks. With over 10,000 nodes, this graph also provides enough examples of ground-truth explanations for rigorous statistical evaluation of explainer performance. The house-shaped motifs follow one of the earliest synthetic graphs used to evaluate GNN explainers [13].
Homophilic-and Heterophilic-SHAPEGGEN Graphs. GNN explainers are evaluated on homophilic graphs [1,[36][37][38] as low homophily levels in graphs can degrade the performance of GNN predictors [39,40]. To this end, there are no heterophilic graphs with ground-truth explanations in current GraphXAI literature despite such graphs being abundant in real-world applications [39]. To demonstrate the flexibility of the SHAPEGGEN data generator, we use it to generate graphs with: i) homophilic ground-truth explanations (SG-BASE) and ii) heterophilic ground-truth explanations (SG-HETEROPHILIC), i.e., G = SHAPEGGEN (S='house', N s =1200, p=0.006, n s =11, K=2, Weakly and Strongly Unfair-SHAPEGGEN Graphs. We utilize the SHAPEGGEN data generator to generate graphs with controllable fairness properties, i.e., leverage SHAPEGGEN to generate synthetic graphs with real-world fairness properties where we can enforce unfairness w.r.t. a given protected attribute. We use the SHAPEGGEN to generate graphs with: i) weakly-unfair groundtruth explanations (SG-BASE) and ii) strongly-unfair ground-truth explanations (SG-UNFAIR) φ=0.75, η=1, L=3). Here, for the first time, we generated unfair synthetic graphs that can serve as pseudo-ground-truth for quantifying whether current GNN explainers preserve counterfactual fairness.
Small and Large SHAPEGGEN Explanations. We explore the faithfulness of explanations w.r.t. different ground-truth explanation sizes. This is important because a reliable explanation should identify important features independent of the explanation size. However, current data resources only provide graphs with smaller size ground-truth explanations. Here, we use the SHAPEGGEN data generator to generate graphs having i) smaller ground-truth explanations size (SG-SMALLEX) φ=0.5, η=1, L=3) and ii) larger ground-truth explanations size (SG-BASE), i.e. house motifs.
BA-Shapes. In addition to SHAPEGGEN, we provide BA-SHAPES [13], a synthetic graph data generator for node classification tasks. We start with a base Barabasi-Albert (BA) [41] graph using N nodes and a set of five-node "house"-structured motifs K randomly attached to nodes of the base graph. The final graph is perturbed by adding random edges. The nodes in the output graph are categorized into four classes corresponding to nodes at the i) top, ii) middle, iii) bottom of houses, and iv) nodes that do not belong to a house.

C.2 Real-World Graphs
We proceed with the description of the real-world graph datasets with and without ground-truth explanations provided in GRAPHXAI. To this end, we provide data resources from crime forecasting, financial lending, and molecular chemistry and biology [1,2,31]. We consider these datasets as they are used to train GNNs for generating predictions in high-stakes downstream applications. In particular, we include chemical and biological datasets because they are used to identify whether an input graph (i.e., a molecular graph) contains a specific pattern (i.e., a chemical group with a specific property in the molecule). Knowledge about such patterns, which determine molecular properties, represents ground-truth explanations [2]. We provide a statistical description of real-world graphs in Tables 5-7. Below, we discuss the details of each of the real-world datasets that we employ: MUTAG. The MUTAG [31] dataset contains 1,768 graph molecules labeled into two different classes according to their mutagenic properties, i.e., effect on the Gram-negative bacterium S.
Typhimuriuma. Kazius et al. [31] identifies several toxicophores -motifs in the molecular graph -that correlate with mutagenicity. The dataset is trimmed from its original 4,337 graphs to 1,768, based on those whose labels directly correspond to the presence or absence of our chosen toxicophores: NH 2 , NO 2 , aliphatic halide, nitroso, and azo-type (terminology, as referred to in Kazius et al. [31]). Benzene. The Benzene [2] dataset contains 12,000 molecular graphs extracted from the ZINC15 [42] database and labeled into two classes where the task is to identify whether a given molecule has a benzene ring or not. Naturally, the ground truth explanations are the nodes (atoms) comprising the benzene rings, and in the case of multiple benzenes, each of these benzene rings forms an explanation. Recidivism.  Node feature vector Figure 2: Overview of SHAPEGGEN graph dataset generator. SHAPEGGEN is a novel dataset generator for graphstructured data that can be used to benchmark graph explainability methods using ground-truth explanations. Graphs are created by combining subgraphs containing any given motif and additional nodes. The number of motifs in a k-hop neighborhood determines the node label (in the figure, we use a 1-hop neighborhood for labeling, and nodes with two motifs in its 1-hop neighborhood are highlighted in red). Feature explanations are some masks over important node features (green striped), with an option to add a protected feature (shown in purple) whose correlation to node labels is controllable. Node explanations are nodes contained in the motifs (horizontal striped nodes) and edge explanations (bold lines) are edges connecting nodes within motifs. With just a few lines of code, one can calculate an explanation for a node or graph, calculate metrics based on that explanation, and visualize the explanation.

Weakly-unfair GT Explanation
Strongly-unfair GT Explanation Fairness Mismatch Figure 6: Counterfactual fairness mismatch scores across eight GNN explainers on graphs with weakly-unfair or stronglyunfair ground-truth (GT) explanations. Results show that explanations produced on graphs with strongly-unfair ground-truth explanations do not preserve fairness and are sensitive to flipping/modifying the protected node feature. Input: Graph G, node features X, node labels Y, homophily coefficient η Output: Optimized node features X C ← Sample connected set of nodes with same labels in Y D dis ← Sampled set of disconnected pairs of nodes with different labels D con ← Sampled set of connected pairs of nodes with different labels L(X) = L C (X) + L dis (X) + L con (X) Return arg min     Table 7: Statistics of real-world node classification datasets in GRAPHXAI without ground-truth (GT) explanations.