{'fraction_non_alphanumeric': 0.041809278079090094, 'fraction_numerical': 0.01363803907113896, 'mean_word_length': 4.9016469857054075, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Causal Attention for Interpretable and Generalizable Graph Classification

In graph classification, attention and pooling-based graph neural networks (GNNs) prevail to extract the critical features from the input graph and support the prediction. They mostly follow the paradigm of learning to attend, which maximizes the mutual information between the attended graph and the ground-truth label. However, this paradigm makes GNN classifiers recklessly absorb all the statistical correlations between input features and labels in the training data, without distinguishing the causal and noncausal effects of features. Instead of underscoring the causal features, the attended graphs are prone to visit the noncausal features as the shortcut to predictions. Such shortcut features might easily change outside the training distribution, thereby making the GNN classifiers suffer from poor generalization. In this work, we take a causal look at the GNN modeling for graph classification. With our causal assumption, the shortcut feature serves as a confounder between the causal feature and prediction. It tricks the classifier to learn spurious correlations that facilitate the prediction in in-distribution (ID) test evaluation, while causing the performance drop in out-of-distribution (OOD) test data. To endow the classifier with better interpretation and generalization, we propose the Causal Attention Learning (CAL) strategy, which discovers the causal patterns and mitigates the confounding effect of shortcuts. Specifically, we employ attention modules to estimate the causal and shortcut features of the input graph. We then parameterize the backdoor adjustment of causal theory -- combine each causal feature with various shortcut features. It encourages the stable relationships between the causal estimation and prediction, regardless of the changes in shortcut parts and distributions. Extensive experiments on synthetic and real-world datasets demonstrate the effectiveness of CAL.


INTRODUCTION
Graph neural networks (GNNs) [8,17] have exhibited impressive performance of graph classification across various domains, such as chemical molecules, social networks, and transaction graphs. Such a success mainly comes from the powerful representation learning of GNNs, which incorporates the graph structure and encodes them into the representations in an end-to-end way. Hence, it is crucial to emphasize the critical part of the input graph, while filtering the trivial part out [23,[37][38][39]. For example, when classifying the mutagenic property of a molecular graph [24], GNNs are expected to latch on the functional groups (i.e., nitrogen dioxide (NO 2 )), instead of the irrelevant patterns (i.e., carbon rings) [5,46]; when detecting fraud in a transaction network, malicious behaviors or coalitions of users are more informative than benign features.
Towards specifying the critical parts in graphs, some follow-on studies [10,16,35,45] adopt the paradigm of "learning to attend" [34,40] -maximizing the mutual information between the attended graph and the ground-truth label -to find the attended graph that maximizes the predictive performance. Specifically, there are two research lines in this paradigm: (1) Attention-based methods [4,16,22,33,35]. They often utilize the attention modules for nodes or edges to locate the attended graphs. These attention modules act like soft masks to identify the importance of each edge and node to the final representations and predictions. (2) Pooling-based methods [10,19,45,49]. They directly adopt hard masks to select a subset of nodes or edges as the attended graphs, to perform the information propagations. These attended graphs aim to approach the features that are beneficial for minimizing the training loss, instead of distinguishing the causal and noncausal effects.
Unfortunately, recent efforts [2,11,12,18] have shown that the current attention or pooling learning methods are prone to exploit the shortcut features to make decisions. These shortcuts usually come from the data selection biases, noisy features, or some trivial patterns from graphs, which are noncausal but discriminative in training data. Due to the existence of these shortcuts, models can capture shortcut features to finish the classification tasks without struggling to learn causal features. For example, instead of probing into the causal effect of the functional groups, the attended graphs prefer "carbon rings" as the cues of the "mutagenic" class, because most training "mutagenic" molecules are in the "carbon rings" context. While such correlations represent statistical relations inherent in the training data and are beneficial to the in-distribution (ID) test evaluations, they inevitably cause a huge performance drop in the out-of-distribution (OOD) test data that are at odds with the training distribution. Taking the molecule classification as an example again, when most test "non-mutagenic" molecules appear in the "carbon rings" context, the attended graphs mislead the GNNs to still predict "mutagenic". As the assumption that the test data conforms to the training distribution is often infeasible in real-world scenarios, the poor generalization of these methods hinders their deployment on critical applications.
To resolve this issue, we first take a causal look at the decisionmaking process of GNNs for graph classification, which delineates the relationships among the causal feature, shortcut feature, and prediction. With our causal assumption in Figure 1, the shortcut feature serves as a confounder [27]. It opens a backdoor path [26] and makes the causal feature and prediction spuriously correlated, e.g., misclassifying "non-mutagenic" molecules with "carbon rings" to the "mutagenic" molecules. Hence, mitigating the confounding effect is promising to exploit the causal features while filtering out the shortcut patterns, thereby enhancing the generalization.
Towards this end, we propose the Causal Attention Learning (CAL) strategy -maximizing the causal effect of the attended graph on predicting the label, while reducing the confounding effect of the shortcut features. Our attended graph aims to approach the causal features in the graph (e.g., nitrogen dioxide), while its complement targets the shortcut features (e.g., carbon rings). Specifically, we first apply attention modules to generate the estimations of the causal and shortcut features from the input graphs. We then parameterize the backdoor adjustment in the causal theory [26,27], which combines each causal estimation with various shortcut estimations and encourages these combinations to maintain a stable prediction. It encourages the invariant relationships between the causal patterns and the predictions, regardless of the changes in the shortcut parts and distribution shifts. We apply CAL to various GNN architectures for graph classification. Experimental results on numerous synthetic and real-world datasets demonstrate the better generalization and insightful interpretations of CAL.
Our technical contributions are summarized as: • We emphasize the generalization issue of current attention-and pooling-based GNNs in graph classification. From the causal perspective, we ascribe such an issue to the confounding effect of the shortcut features.
• We present a novel Causal Attention Learning (CAL) strategy for graph classification. It makes GNNs exploit the causal features while filtering out the shortcut patterns. • Extensive experiments on synthetic and real-world datasets justify the effectiveness of CAL. More visualizations with in-depth analyses demonstrate the interpretability and rationality of CAL.

PRELIMINARIES 2.1 Notations
We denote a graph by G = {A, X} with the node set V and edge set E. Let X ∈ R |V |×F be the node feature matrix, where x = X[ , :] is the F-dimensional attribute vector of node ∈ V. We use the adjacency matrix A ∈ R |V |×|V | to delineate the whole graph structure, where We define GConv(·) as a GNN layer module and denote the node representation matrix by H ∈ R |V |× , whose -th row h = H[ , :] denotes the representation of node .

Attention Mechanism in GNNs
In GNNs, attention can be defined over edges or nodes. For edgelevel attentions [4,16,21,33,35], they utilize weighted message passing and aggregation to update node representations H ′ : where M ∈ R |V |×|V | denotes the attention matrix that is often derived from trainable parameters and node representations. For node-level attention, several studies [18,19,22] define the selfattention mask to select the most attentive node representations: where M ∈ R |V |×1 represents the node-level attentions, which can be generated by a network (e.g., GNNs or MLPs); ⊙ is the broadcasted element-wise product. Hereafter, we can make further pooling operation [19] for the output node representations H and summarize the graph representation h G for graph G via the readout function readout (·). Then we use a classifier Φ to project the graph representation into a probability distribution z G : These methods follow the paradigm of "learning to attend" by minimizing the following empirical risk: where L CE is the cross-entropy loss over the training data D, and y G is the ground-truth label vector of G. However, this learning strategy heavily relies on the statistical correlations between the input graphs and labels. Hence, they will inevitably capture the noncausal shortcut features to make predictions.

METHODOLOGY
In this section, we first analyze the GNN learning from the perspective of causality. From our causal assumption, we identify the shortcut feature as a confounder. Then we propose the causal attention learning strategy to alleviate the confounding effect.

A Causal View on GNNs
We take a causal look at the GNN modeling and construct a Structural Causal Model (SCM) [27] in Figure 1. It presents the causalities among five variables: graph data , causal feature , shortcut feature , graph representation , and prediction , where the link from one variable to another indicates the cause-effect relationship: cause → effect. We list the following explanations for SCM: • ← → . The variable denotes the causal feature that truly reflects the intrinsic property of the graph data . While represents the shortcut feature which is usually caused by the data biases or trivial patterns. Since and naturally coexist in graph data , these causal effects are established. • → ← . The variable is the representation of the given graph data . To generate , the conventional learning strategy takes the shortcut feature and the causal feature as input to distill discriminative information. • → . The ultimate goal of graph representation learning is to predict the properties of the input graphs. The classifier will make prediction based on the graph representation .
Scrutinizing this SCM, we recognize a backdoor path between and , i.e., ← → → → , wherein the shortcut feature plays a confounder role between and . Even if has no direct link to , the backdoor path will cause to establish a spurious correlation with , e.g., making wrong predictions based on shortcut feature instead of causal feature . Hence, it is crucial to cut off the backdoor path and make the GNN exploit causal features.

Backdoor Adjustment
We have realized that shielding the GNNs from the confounder is the key to exploiting causal features. Instead of modeling the confounded ( | ) in Figure 1, we should achieve the graph representation learning by eliminating the backdoor path. But how to achieve this? Fortunately, causal theory [26,27] provides us with a feasible solution: we can exploit the do-calculus on the variable to remove the backdoor path by estimating ( | ) = ( | ( )). It needs to stratify the confounder between and . Therefore, we can obtain the following three essential conclusions: • The marginal probability ( = ) is invariant under the intervention, because the shortcut feature will not be affected by cutting off the backdoor path. Thus, ( ) = ( ). • The conditional probability ( | , ) is invariant, because 's response to and has nothing to do with the causal effect between and . Then we can get: ( | , ) = ( | , ). • Obviously, the variables and are independent under the causal intervention, which we have: Based on the above conclusions, we have: where T denotes the confounder set; ( | , ) represents the conditional probability given the causal feature and confounder ; ( ) is the prior probability of the confounder. Equation (5) is usually called backdoor adjustment [26], which is a powerful tool to eliminate the confounding effect. However, there exist two challenges for implementing Equation (5): i) The confounder set T is commonly unobservable and hard to obtain. ii) Due to the discrete nature of graph data, it seems difficult to directly manipulate the graph data, conditioning on domain-specific constraints (e.g., valency rules in molecule graphs). In section 3.4.3, we will introduce a simple yet effective solution to overcome these issues.

Causal and Trivial Attended-graph
Given a graph G = {A, X}, we formulate the soft masks on the graph structure and node feature as M ∈ R |V |×|V | and M ∈ R |V |×1 , respectively. Wherein, each element of the masks indicates the attention score relevant to the task of interest, which often falls into the range of (0, 1). Given an arbitrary mask M, we define its complementary mask as M = 1 − M, where 1 is the all-one matrix. Therefore, we can divide the full graph G into two attended-graphs: With the inspection on the data-generating process, recent studies [18,23,39,44] argue that the label of a graph is usually determined by its causal part. Considering a molecular graph, its mutagenic property relies on the existence of relevant functional groups [37]; Taking the digit image in the form of superpixel graph as another example, the coalition of digit-relevant nodes determines its label. Formally, given a graph G, we define the attended graph collecting all causal features as the causal attended-graph G , while the counterpart forms the trivial attended-graph G . However, the ground-truth attended-graph is usually unavailable in real-world applications. Hence, we aim to capture the causal and trivial attended-graph from the full graph by learning the masks: Learning to identify causal attended-graphs not only guides the representation learning of GNNs, but also answers "What knowledge does the GNN use to make predictions?", which is crucial to the applications on explainability, privacy, and fairness.

Causal Attention Learning
To implement the aforementioned backdoor adjustment, we propose the Causal Attention Learning (CAL) framework: 3.4.1 Estimating soft masks. Towards effective causal intervention, it is necessary to separate the causal and shortcut features from the full graphs. To this end, we hire attention modules, which yield two branches for the causal and trivial proposals. Given a GNN-based encoder (·) and a graph G = {A, X}, we can obtain Then we adopt two MLPs: MLP node (·) and MLP edge (·) to estimate the attention scores from two orthogonal perspectives: node-level and edge-level. For node and edge ( , ) we can obtain: , where (·) is softmax function, || denotes concatenation operation; , represent the node-level attention score for node and edge-level attention score for edge ( , ) in causal attendedgraph; analogously, , are for trivial attended-graph. Note that + = 1, and + = 1. These attention scores indicate how much the model pays attention to each node or edge in the corresponding attended-graph. Now we can construct the soft masks M , M , M , and M based on the attention scores , , , and , respectively. Finally, we can decompose the original graph G into the initial causal and trivial attended-graphs: 3.4.2 Disentanglement. Until now, we have distributed the attention scores at the granularity of nodes and edges to create the initial attended-graphs. Now we need to make the causal and trivial attended-graphs to capture the causal and shortcut features from the input graphs, respectively. Specifically, we adopt two GNN layers to obtain the representations of attended-graphs and make predictions via readout function and classifiers: The causal attended-graph aims to estimate the causal features, so we classify its representation to the ground-truth label. Thus, we define the supervised classification loss as: where L sup is the cross-entropy loss over the training data D. The trivial attended-graph aims to approach the trivial patterns that are unnecessary for classification. Hence, we push its prediction evenly to all categories and define the uniform classification loss as: where KL denotes the KL-Divergence, y unif represents the uniform distribution. By optimizing the above two objectives, we can effectively disentangle causal and trivial features. Please note that prior efforts [18,23,37,38,44] have shown that the mutual information between the causal part and label is greater than that between the full graph and label, due to the widespread trivial patterns or noise. Hence, the proposed disentanglement will not make the captured causal attended-graph converge to the full graph (noiseless full graph is a special case), which is not an optimal solution. See Section 4.5 for more supporting evidence and analyses. (5), one promising solution to alleviating the confounding effect is the backdoor adjustment -that is, stratifying the confounder and pairing the target causal attended-graph with every stratification of trivial attended-graph to compose the "intervened graphs". However, due to the irregular graph data, it is impossible to make the intervention on data-level, e.g., changing a graph's trivial part to generate a counterfactual graph data. Towards this end, we make the implicit intervention on representation-level and propose the following loss guided by the backdoor adjustment:

Causal intervention. As shown in Equation
where z G ′ is the prediction from a classifier Φ on "implicit intervened graph" G ′ ; h G is the representation of causal attended-graph G derived from Equation (9); while h G ′ is the representation of stratification G ′ obtained via Equation (10);T is the estimated stratification set of the trivial attended-graph, which collects the appearing trivial features from training data. In practice, we apply random addition to make the intervention in Equation (13). We define the Equation (14) as the causal intervention loss. It pushes the predictions of such intervened graphs to be invariant and stable across different stratifications, due to the shared causal features. Finally, the objective of CAL can be defined as the sum of the losses: where 1 and 2 are hyper-parameters that determine the strength of disentanglement and causal intervention, respectively. The detailed algorithm of CAL is provided in Appendix A.1, Alg.1, and the overview of CAL is depicted in Figure 2.

EXPERIMENTS
To verify the superiority and effectiveness of the proposed CAL, we conduct experiments to answer the following research questions: • RQ1: How effective is the proposed CAL in alleviating the outof-distribution (OOD) issue? • Synthetic graphs: Following [44], we create the synthetic dataset for graph classification, which contains a total of 8,000 samples with 4 classes, and keeps balance (2,000 samples) for each class. As shown in Figure 3, each sample consists of two parts: causal subgraph and trivial subgraph. More details about the causal and trivial subgraph are provided in Appendix A.2. The task is to predict the type of the causal part in the whole graph. For simplicity, we choose the "House" class to define the bias-level: = #Tree-House #House (16) where #Tree-House denotes the number of "House" causal subgraphs with the "Tree" trivial subgraphs, and #House presents the number of graphs in the "House" class, which is 2,000. We set the proportion of "Tree" in the other three classes to 1 − . Obviously, for the unbiased dataset, = 0.5. We abbreviate the synthetic dataset with bias-level as SYN-. We keep the same bias-level on the training/validation set and keep the testing set unbiased. Please refer to Appendix A.2 for more details. • Real-world graphs: We conduct experiments on three biological datasets (MUTAG, NCI1, PROTEINS), three social datasets (COLLAB, IMDB-B, IMDB-M) [24], and two superpixel datasets (MNIST, CIFAR-10) [18]. More details, such as statistics and splitting of datasets, are provided in Appendix A.2.

Performance on Synthetic Graphs (RQ1)
To explore whether CAL can alleviate the OOD issue, we first conduct experiments on SYN-with different biases: ∈ {0.1, 0.2, ..., 0.9}. The experimental results are summarized in Table 1 and Figure 4. We have the following Observations: Obs 1: Refining discriminative features without considering the causality leads to poor OOD generalization. For the unbiased dataset, most attention-and pooling-based baselines, such as GlobalAtt, SuperGAT, SortPool, Top-Pool, outperform GCN. It indicates the effectiveness of extracting discriminative features in the ID setting. However, as the bias-level goes to extremes, the performance dramatically deteriorates. For instance, the performance drop of attention-based methods ranges from 7.37% ∼ 12.75% on SYN-0.1, and 3.79% ∼ 13.79% on SYN-0.9; Pooling-based methods drop from 7.82% ∼ 14.24% and 3.99% ∼ 12.10% for SYN-0.1 and SYN-0.9. These indicate that simply extracting discriminative features by attention or pooling module is prone to capture the data biases. These are also beneficial for reducing the training loss but lead to poor OOD generalization. Taking SYN-0.9 as an example, most "House" co-occur with "Tree" in the training data, so the model will mistakenly learn shortcut features from the "Tree"-type trivial subgraphs to make predictions, instead of probing the "House"-type causal subgraphs. This will mislead the model to adopt the "Tree" pattern to make decisions in the inference stage.
Obs 2: GNNs with better ID performance tend to have worse OOD generalization. For the unbiased dataset, GIN achieves the best performance (96.74%), while GAT (92.69%) outperforms the GCN (90.94%). This indicates that the in-distribution (ID) performance of these models exhibits such an order: GIN > GAT > GCN. However, when the bias is changed to 0.1 and 0.9, the performance of GIN drops by 9.55% and 7.36%, GAT drops by 8.71% and 5.47% and GCN drops by 6.60% and 5.43%, respectively. It shows that the rankings of models' robustness against OOD issues are in the opposite order: GCN > GAT > GIN. This indicates that GNNs with better ID performance are prone to learn more shortcut features. Similar trends also occur in other baselines. After adopting the proposed CAL, this phenomenon is significantly alleviated, which verifies the effectiveness of CAL in overcoming the OOD issue.  Obs 3: Mitigating the confounder achieves more stable performance on OOD datasets. We first define the performance discount on SYN-as the accuracy on SYN-normalized by the accuracy on unbiased SYN-0.5. It indicates the degree of the performance degradation on biased synthetic datasets, without considering the model's ID generalization. We plot the performance discount curves on SYN-with ∈ {0.1, 0.2, ..., 0.9}. As depicted in Figure 4, we observe that pooling-based methods outperform GIN in a small range of bias-levels (0.2 ∼ 0.8), while the performance drops sharply when = 0.1 or 0.9. For example, the performance discount of Top-Pool drops from 0.95 to 0.88 as reduces from 0.2 to 0.1. Attentionbased methods perform worse than GIN when < 0.5. For > 0.5, AGNN achieves better performance than GIN, while GlobalAttention often performs worse. These results reflect that attention-or pooling-based methods all have their own weaknesses, such that they cannot consistently overcome the diverse distribution shifts. Equipped with CAL, GIN (red curve) consistently outperforms all the baselines on all ranges of bias-levels and obviously keeps a large gap, which further demonstrates the significance of mitigating the confounding effect, and the effectiveness of CAL. For comprehensive comparisons, we also plot two upper bound methods: IRM and DRO (dash lines), which require additional annotation information of trivial subgraphs for training. We observe that, even without additional information, CAL achieves comparable performance with these upper bound methods.

Performance on Real-world Graphs (RQ2)
Unlike synthetic graphs, there may not exist visible or specific patterns of the causal/trivial subgraphs in real-world graphs. However, there still exist irregular core-subgraphs [18,23,38,44] that determine the predictions, which will inevitably involve different degrees of biases caused by the complementary parts. Similar to SYN-, they mislead the GNNs to learn the spurious correlations. Hence, we verify the practicability of CAL on eight real-world datasets. We report the results of the baselines from the original papers by default and reproduce the missing results. The results are summarized in Table 2 and we make the following Observations: Obs 4: The OOD issue is widespread in real-world datasets. Attention-based and pooling-based methods are on a par with GNNs, and they both outperform graph kernel-based methods in most cases. It can be seen from the last six rows in Table 2, when CAL is applied to different GNN models, it consistently produces further performance improvements. It demonstrates that the distribution shifts also widely exist in real-world datasets. Specifically, we can find that GCN often performs worse than other GNNs, attention-based or pooling-based methods, while the performance significantly improves after adopting CAL. For instance, on IMDB-B and MNIST datasets, GCN+CAL achieves 1.92% and 4.52% relative improvements, respectively. This indicates that GCN is vulnerable to the distribution shift in certain datasets. Thanks to the causality, CAL will push GCN to pay more attention to causal features, which can establish robustness against the widespread OOD issues and achieve better generalization.

Ablation Study (RQ3)
In this section, we investigate the impact of the node/edge attention, random combination and the loss coefficients 1 and 2 .
Node Attention v.s. Edge Attention. Node Attention (NA) and Edge Attention (EA) refine the features from two orthogonal views: node-level and edge-level. Here we want to examine the effect of adopting NA or EA alone. We adopt GCN as the encoder to conduct experiments on four biased synthetic datasets and two real-world datasets. GCN+CAL w/o NA or EA represents the node/edge attention scores in Equation (7)/(8) are evenly set as 0.5. The experimental results are shown in Figure 5. We can find that: (1) Comparing NA with EA, the performance of CAL without NA is significantly worse than that without EA, which indicates that the node feature contains more significant information compared with graph structure. (2) Just adopting NA or EA alone still achieves better performance than baselines, which demonstrates that only applying NA or EA can also disentangle the causal/trivial attended-graph and achieve causal intervention to some extent. Random Combination. We need to stratify the confounder distribution for causal intervention. With the random combination, each causal feature will combine with different types of trivial patterns. To verify its importance, we change the "Random Addition" module in Figure 2 to "Addition", which just adopts the addition operation orderly, and we rename it as "GCN+CAL w/o RD". The experimental results are shown in Figure 5. We can find that: (1) The performance drops severely compared with GCN+CAL, which demonstrates the importance of the causal intervention. (2) GCN+CAL w/o RD can also outperform the GCN baselines. We conjecture that just implementing disentanglement makes GNN pay more attention to the causal features, which will slightly ignore the data biases or trivial patterns. These results also reflect that disentanglement and causal intervention will help each other to improve their own effectiveness.
Loss coefficients 1 and 2 . According to Equation (15), 1 denotes the strength of the disentanglement for the causal/trivial features, while 2 controls the strength of the causal intervention. To explore their impacts, we use GCN as the encoder and conduct experiments on two biased synthetic datasets and two real-world datasets. We fix one coefficient as 0.5 and change the other one in (0, 1) with a step size of 0.1. The experimental results are shown in Figure 6. We can find that: (1) 1 achieves better performance in a range of 0.3 ∼ 0.7. Too small or too large values will cause performance degradation. (2) 2 is not as stable as 1 . The optimal range is around 0.3 ∼ 0.5. It leads to a strong decline at 0.5 ∼ 0.8, which indicates that coefficient 2 should be set prudently.

Visualization and Analysis (RQ4)
Causal attended-graphs. We plot node/edge attention areas of the causal attended-graphs based on the attention scores in CAL. We adopt a GCN-based encoder and apply CAL on SYN-and MNIST superpixel graphs. The visualizations are shown in Figure 7. Nodes with darker colors and edges with wider lines indicate higher attention scores. We surprisingly find that almost all the darker colors and wider lines precisely distribute on the deterministic areas, such as the causal subgraphs we defined in the synthetic dataset and the nodes located on digit pixels in MNIST superpixel graphs. It further demonstrates that the proposed CAL can effectively capture the causal features with insightful interpretations. The explanation for performance improvements. Figure 8 displays the distribution of misclassification on SYN-. The abscissa represents the predictions, and the ordinate denotes the groundtruth types. The numbers in each row denote the proportion for each class. Figure 8 (Left) shows that the wrong predictions of graphs with "BA" are mainly distributed in "Cycle", "Grid" and "Diamond" classes, while the wrong predictions of graphs with "Tree" mainly concentrate on the "House" class (highlighted by the red circle). On one hand, most of the "House" co-occur with "Tree" in the training data, GCN tends to capture the shortcut features, e.g., "Tree" patterns, to make decisions. Therefore, the other three causal subgraphs with "Tree" will mainly be misclassified as "House" in the testing set. On the other hand, only a few "House" causal subgraphs co-occur with "BA", so the other three causal subgraphs with "BA" will almost not be misclassified as "House". In contrast, Figure 8 (Right) shows that, by applying CAL, the concentration of misclassification is obviously alleviated. This demonstrates that CAL improves performance by mitigating the confounding effect.

RELATED WORK
Attention Mechanism selects the informative features from data, which has obtained great success in computer vision [7,14,36,43] and natural language processing tasks [6,34]. In recent years, attention mechanism has gradually become prevalent in the GNN field. The attention modules for GNNs can be defined over edges [4,16,21,33,35] or over nodes [19,20,22]. Despite effectiveness, attention learning still stays at how to better fit the statistical correlations between data and labels. Hence, the learned attentions are inherently biased in OOD settings. Recent studies [36,43] propose the causal attention modules to alleviate the bias. CaaM [36] Figure 8: The misclassification distribution. Red circle highlights the concentration degree of misclassification.
adopts the adversarial training to generate the data partition in each iteration to achieve the causal intervention. CATT [43] proposes in-sample and cross-sample attentions based on front-door adjustment. However, they are both tailored for computer vision tasks, while cannot transfer to graph learning tasks, due to the irregular and challenging graph-structure data. Distinct from them, we utilize the disentanglement and causal intervention strategies to strengthen the attention modules for GNNs. OOD Generalization [2,13,28,29] has been extensively explored in recent years. IRM [2] minimizes the empirical risk under different environments. Group-DRO [29] adversarially explores the group with the worst risk and achieves generalization by minimizing the empirical risk of the worst group. Existing efforts [2,28,29] mainly focus on computer vision or natural language processing tasks, while the GNN field is of great need but largely unexplored. Furthermore, these methods require the environment or group prior information for each training sample, which is expensive in practice. To alleviate this dilemma, we adopt causal intervention to strengthen the causal relationship between the causal feature and prediction, thereby achieving better generalization. Causal Inferences [26,27] endows the model with the ability to pursue real causality. A growing number of studies [15,25,32,48] have shown that causal inference is beneficial to diverse computer vision tasks. CONTA [48] uses backdoor adjustment to eliminate the confounder in weakly supervised semantic segmentation tasks. DDE [15] proposes to distill the colliding effect between the old and the new data to improve class-incremental learning. Unlike computer vision, the application of causal intervention in the GNN community is still in its infancy. CGI [9] explores how to select trustworthy neighbors for GNN in the inference stage, and demonstrates its effectiveness in node classification. Recent work [47] studies the connection between GNNs and SCM from a theoretical perspective. Different from them, we introduce a causal attention learning strategy to mitigate the confounding effect for GNNs. It encourages GNNs to pay more attention to causal features, which will enhance the robustness against the distribution shift.
In this work, we revisit the GNN modeling for graph classification from a causal perspective. We find that current GNN learning strategies are prone to exploit the shortcut features to support their predictions. However, the shortcut feature actually plays a confounder role. It establishes a backdoor path between the causal feature and the prediction, which misleads the GNNs to learn spurious correlations. To mitigate the confounding effect, we propose the causal attention learning (CAL) strategy for GNNs. CAL is guided by the backdoor adjustment from the causal theory. It encourages the GNNs to exploit causal features while ignoring the shortcut parts. Extensive experimental results and analyses verify its effectiveness. Future studies include adopting powerful disentanglement methods and more advanced causal intervention strategies to improve the CAL. We will also make efforts to apply CAL to other graph learning tasks, such as node classification or link prediction.

A.2 Datasets Details
In this section, we give more details about the synthetic datasets and real-world datasets. 1) Synthetic graphs. For each synthetic graph instance, it consists of two subgraphs: trivial and critical subgraphs. We introduce the proposed trivial subgraph and critical subgraph as follows: • Trivial subgraph. There exist two types of trivial subgraphs: BA-SHAPES and Tree. The BA-SHAPES is a Barabási-Albert (BA) graph [3], and we abbreviate it as "BA" in this paper. The "Tree" graph is a base 12-level balanced binary tree [44]. To reduce the influence, we control the number of nodes in the two kinds of trivial subgraphs to be similar. • Causal subgraph. There are four types of causal subgraphs: "House", "Cycle", "Grid", "Diamond". The visualizations of these trivial subgraphs and causal subgraphs are depicted in Figure 3. For each synthetic graph instance, a causal subgraph is randomly attached on one node of a trivial subgraph. Then the resulting graph is further perturbed by adding 10% random edges. We take the one-hot form of the node degree as the node feature and set the dimension of node feature to 20. The synthetic graph examples are displayed in Figure 3. The statistics of the synthetic datasets are summarized in Table 3. We split the dataset into training, validation and testing set with the ratio of 7: 1: 2. 2) Real-world graphs. To demonstrate the practicality of the proposed CAL, we conduct experiments on TUDataset [24] and Superpixel graphs [18]. For TUDataset, we gather three biological datasets (MUTAG, NCI1, PROTEINS) and three social networks datasets (COLLAB, IMDB-B, IMDB-M), which are commonly used in graph classification benchmarks [8,41]. Following [8,41,45], we use 10-fold cross-validation and report average accuracy and standard deviation. The superpixel graphs [8,18] includes MNIST and CIFAR-10, which are classical image classification datasets converted into graphs using superpixels technology [1] and assigning each node's features as the superpixel coordinates and intensity. Following [8,18], we split the MNIST and CIFAR-10 to 55K training/5K validation/10K testing, and 45K training/5K validation/10K testing, respectively. All the detailed statistics about the real-world datasets are summarized in Table 3.

A.3 Hyper-parameters
As for training parameters, we train the models for 100 epochs with batch size of 128. We optimize all models with the Adam optimizer. For SYN-and TUDataset, we use GCN, GIN and GAT as GNN encoders with 3 layers and 128 hidden units. For Superpixel graphs MNIST and CIFAR-10, we use the GNN encoders with 4 layers and 146 hidden units as [8]. For all the baselines, we follow the default settings from original papers and reproduce the missing results. For the proposed CAL, we search 1 and 2 in (0.1, 1.0) with a step size of 0.1 and report the results with the best settings. We adopt NVIDIA 2080 Ti (11GB GPU) to conduct all our experiments, the training time comparison is shown as Table 4.