{'fraction_non_alphanumeric': 0.037196144174350375, 'fraction_numerical': 0.012023260687342833, 'mean_word_length': 4.6316565865171855, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Be Confident! Towards Trustworthy Graph Neural Networks via Confidence Calibration

Despite Graph Neural Networks (GNNs) have achieved remarkable accuracy, whether the results are trustworthy is still unexplored. Previous studies suggest that many modern neural networks are over-confident on the predictions, however, surprisingly, we discover that GNNs are primarily in the opposite direction, i.e., GNNs are under-confident. Therefore, the confidence calibration for GNNs is highly desired. In this paper, we propose a novel trustworthy GNN model by designing a topology-aware post-hoc calibration function. Specifically, we first verify that the confidence distribution in a graph has homophily property, and this finding inspires us to design a calibration GNN model (CaGCN) to learn the calibration function. CaGCN is able to obtain a unique transformation from logits of GNNs to the calibrated confidence for each node, meanwhile, such transformation is able to preserve the order between classes, satisfying the accuracy-preserving property. Moreover, we apply the calibration GNN to self-training framework, showing that more trustworthy pseudo labels can be obtained with the calibrated confidence and further improve the performance. Extensive experiments demonstrate the effectiveness of our proposed model in terms of both calibration and accuracy.


Introduction
Graphs are ubiquitous in the real world, including social networks, e-commerce networks, traffic networks, and so on. Recently, Graph Neural Networks (GNNs), which are able to effectively learn the node representations based on the message-passing manner, have attracted considerable attention in dealing with graph data [16,33,39,44,15,2,34]. To date, GNNs have been applied to various applications and achieved remarkable accuracy, e.g., node classification [16,33], link prediction [41] and graph classification [9].
However, it is well established that a model with good accuracy is not the only goal, but a trustworthy model is highly desired in many applications, especially in safety-critical fields [1]. Usually, a trustworthy model implies that it should know when it is likely to be incorrect, in other words, the probability, i.e., the confidence, associated with the predicted class label should reflect its ground truth correctness likelihood [12]. For example, in the scene of autonomous driving, the system will adopt the prediction given by the model only when the model has high confidence for its prediction. Otherwise, the decision-making power will be returned to the driver or the system adopts other safer strategies. Recently, the confidence calibration has attracted considerable attention in deep learning [12,40,19], which reveals that many modern neural network models are over-confident on the predictions, i.e., the prediction accuracy is lower than its confidence. However, it has not been studied in GNNs on the semi-supervised scenario, which gives rise to one fundamental question: will the current GNNs follow the same over-confident property as other neural networks? A well-informed answer can help us better understand GNNs and enable GNNs to be applied to various areas in a more reliable manner.
As the first contribution of this study, we present experiments assessing the relationship between the confidence and the accuracy of Graph Convolutional Networks (GCNs) [16] and Graph Attention Networks (GAT) [33] in the node classification task (more details can be seen in Section 2), respectively. Surprisingly, we discover that existing GNNs are far distant from being well-calibrated, and more importantly, GNNs tend to be under-confident in their predictions, which is very different from other modern deep learning models that are often over-confident [12,19]. GNNs being under-confident means that many predictions are distributed in the low-confidence range, and therefore, fewer predictions are available for safety-critical applications. Once the weakness is identified, another natural question is: how can we calibrate the confidence on predictions given by GNNs so as to make them more trustworthy?
Essentially, the confidence calibration is to calibrate the outputs (also known logits) of original models (e.g., GNNs), therefore, a straightforward manner is to employ temperature scaling (TS) [12], OP-families [25] to learn calibration function using a held-out dataset in a post-hoc way. However, when being applied to graphs, they all ignore the effect of topology, which will inevitably make mistakes during calibration. For example, considering that the logits of two nodes a and b are the same, but node a is similar to its neighbors while node b is not. Apparently, the predictions of GCNs for a should be more confident than b, while the traditional calibration methods, e.g., TS, will learn the same confidence for a and b, because it does not consider the effect of topology.
Moreover, most of them explore calibration functions only in the linear space [12,18] while it is well known that non-linear space contains more complex function transformation which is able to calibrate networks with complicated landscapes well. Even if some works have explored the non-linear space such as Matrix Scaling [12], they generally degrade the classification accuracy of the original classifier, while a good accuracy is still a basic requirement by many applications.
In this paper, we introduce a topology-aware post-hoc calibration method for GNNs. Specifically, for the logits given by the original classification GNNs, we employ another calibration GCN (CaGCN) to propagate confidence, naturally enabling that the confidence of topologically adjacent nodes becomes similar. CaGCN learns a unique temperature t for each node for temperature scaling, thus preserving the accuracy of the original classification GCN. In addition, based on our finding that large numbers of high-accuracy predictions are distributed in the low-confidence range, we design a calibrated self-training model CaGCN-st in which the confidence is firstly calibrated then used to generate pseudo labels with high confidence. The contributions of this paper are three-fold: • We study the trustworthy problem of GNNs, and discover one unique characteristic of GNNs, i.e., the predictions made by GNNs are usually under-confident. • We propose a novel trustworthy GNN model based on the confidence calibration. Our proposed calibration function has three features: topology-aware, non-linear, and accuracy-preserving. We further design a calibrated self-training GNN model, which can effectively utilize the predictions with high confidence. • Extensive experiments demonstrate the effectiveness of our proposed models in terms of both calibration and accuracy.

Notation and Preliminary Study
In this paper, we focus on the calibration of semi-supervised node classification in an undirected attributed graph G = (V, E) with the adjacent matrix A ∈ R N×N and the node feature matrix X = [x 1 , . . . , x N ] T . V is a set of nodes and E ⊆ V × V is a set of edges between nodes. N = |V | is the number of nodes. Here we give the definition of perfect calibration of GNNs as follows: Definition 1. Given random variables A, X, Y ⊆ {1, . . . , K} and a GNN model f θ where θ is the learnable parameters, for node i with label y i ∈ Y, z i = f θ (x i , A) = [z i,1 , . . . , z i,K ] T is the output of GNNs (i.e., the prediction probability), andŷ i = arg max k z i,k andp i = max k z i,k are the prediction and the confidence respectively. Then we define f θ to be perfectly calibrated as: The diagram is expected to plot an identity function of accuracy with respect to confidence. Any deviation from a perfectly diagonal (i.e., the difference between blue and red histogram) represents the miscalibration. According to Definition 1, GNN is perfectly calibrated only when the confidencep i is exactly equal to the true probability of getting a correct prediction for every node.
Next, we take two representative GNNs (GCN [16] and GAT [33]) as examples to analyze whether they are perfectly calibrated. Specifically, we apply GCN and GAT to four widely used datasets Cora [29], Citeseer [29], Pubmed [29], CoraFull [3], and examine whether their results satisfy Definition 1. To provide more results, we select three label rates for training set (i.e., 20, 40, 60 labeled nodes per class). All the experimental settings follow [16,33]. Since the true probability p cannot be exactly known, we take an approximate way to evaluate the calibration as in [12]. In particular, we first partition the [0,1] range of confidence into 20 equal bins and then we group the nodes into corresponding bins according to their confidence. After that we calculate the average accuracy of each bin. We expect the average accuracy is equal to the average confidence of each bin, which means the model is approximately perfectly calibrated. For example, if the average confidence of nodes in the bin [0.95, 1.0] is 0.96, and then the classification accuracy in this bin should be 96%.
We illustrate the results of label rate being 20 in Fig. 1 using Reliability Diagrams [23] here, where the x-axis is the confidence in 20 bins of equal size and y-axis is the average accuracy in each bin. The blue represents the classification accuracy of GCN and GAT while the red is our expectation. More results of label rate being 40, 60 and other GNN models can be seen in Fig. 8, Fig. 9, Fig. 12, Fig. 13 and Fig. 14 in the appendix. We can see that in all the datasets, the average accuracy of most bins is higher than the average confidence. In other words, these GNNs actually achieve remarkable performance, but they all output low confidence, i.e., the GNNs are usually under-confident. Please note that this phenomenon of GNNs is very different from other modern neural networks, which are generally known to be over-confident [12,19]. Moreover, as shown in Fig. 2, we also visualize the confidence distribution of test nodes, where the x-axis is the confidence and y-axis is the density [28]. The histogram height multiplied by the width is equal to the frequency. The blue represents the confidence distribution of correct predictions while the yellow represents that of incorrect predictions. More results of label rate being 40 and 60 can be seen in Fig. 10 and Fig. 11 in the appendix. We can see that a large quantity of correct predictions are distributed in the low confidence range. The results above indicate that the current GNNs are far from perfect calibration, leading to unreliable confidence.

Confidence Calibration on GCNs
In this section, we propose our method to calibrate current GNNs. Given A and X, for a l-layer GCN [16], the output of the GCN before the softmax layer can be obtained by: where W (l) is the weight matrix of l-th layer in GCN and σ (·) is the activation function. For each node i ∈ {1, · · · , N}, our goal is to learn a calibration function which is fed with v i (often known as the logit of node i) and outputs a calibrated confidence using a held-out dataset in a post-hoc way. The calibration function should satisfy three points below: (1) taking the network topology into account (2) non-linear (3) preserving the classification accuracy of the GCNs.  We assume that the ground-truth confidence distribution in a graph has homophily property, i.e., the confidence of neighboured nodes given by well-calibrated models should be similar, and thus we conduct an experiment to verify this. We employ the classic temperature scaling method [12] as our calibration function and use the total variation [27] of confidence as our evaluation, which sums the difference of confidence between all the neighboured nodes. We compare the total variation of confidence before and after confidence calibration, where the results are shown in Table 1. We can find that the total variation of confidence does decrease after temperature scaling, which verifies our assumption. This inspires us that if a GCN model is well-calibrated, then the confidence between neighbors should be more similar than before.

CaGCN: GCNs as Calibration Function
To this end, we find that GCN itself can play the role of calibration function that meets above requirement since GCN is able to propagate node features along the network topology and smooth similar information between neighboured nodes. Therefore, we can employ another l-layer GCN (CaGCN) as our calibration function to propagate the confidence along the network topology. Specifically, given the output V of the classification GCN, the logit v i and confidencep i for node i after calibration can be obtained by: where is the softmax operation. Then the total variation of confidence will surely become lower and the original classification GCN will be calibrated. Please note that although temperature scaling can be directly applied here, compared with GCN, it does not take the network topology into account, which may cause mistakes mentioned in Section 1. Moreover, temperature scaling only employs a linear transformation, and GCN is able to learn a non-linear calibration function.
For a comprehensive understanding of confidence propagation, we make a detailed and visible illustration here. As shown in Fig. 3, the logits of two nodes a and b are the same, but node a is similar to its neighbors while node b is not. Apparently, the predictions of GCNs for a should be more confident than b. Suppose that a, b and their neighbours are under-confident based on the observation above. If we continue to propagate their logits along the topology using another GCN, the logits of a and its neighbors will tend to be the same. Therefore, if one or more of these nodes are calibrated during the calibration process, all of them will be calibrated as well. The confidence is propagated in this way. On the other hand, looking at another node b, it is as difficult even for manual classification as it is for GCNs. Consequently, the confidence of b should stay still even be lower. However, it will become higher because of the influence from a if we use the traditional calibration method without considering the network topology. Instead, when the network topology is taken into account, the logit of b will be averaged by its neighboured and each dimension tends to 1/K. It will be correctly calibrated when other nodes in the same situation are well-calibrated.

The Accuracy-Preserving Property
Until now, we have proposed a non-linear calibration model CaGCN which can take the network topology into account, but the accuracy-preserving property cannot be satisfied. To address this problem, we firstly study the general accuracy-preserving calibration function.
shares the same order with v i as a result of the strictly isotonicity of s, the order between classes of the logit v i is unchanged, hence the accuracy of the prediction is preserved.
Temperature scaling [12] is the simplest accuracy-preserving calibration method using a scalar parameter t called temperature for all classes. Given the logit v i of node i, the confidence of the is the calibration function and s(x) = x/t is the strictly isotonic function.
However, we can find that temperature scaling (TS) [40] only performs the same linear transformation for all the nodes using the same t. As mentioned in Eq. 3, we propose to use CaGCN as our calibration function, while CaGCN is generally not isotonic, i.e., the order between classes of v i and v i is not the same, implying that after calibration by CaGCN, the accuracy of original GCN cannot be preserved. Instead, here we propose an improved CaGCN. Given the output V of the classification GCN, we firstly use a l-layer GCN to learn a unique temperature t i for each node i, then get a calibrated logit v i by transforming its original logit v i using t i in a temperature-scaling way, and finally obtain calibrated confidencep i as follows: where t i ∈ R is a scalar greater than zero and σ + (x) = log(1 + exp(x)) is an element-wise softplus activation [8]. The model proposed in Eq. 5 does not change the order between classes of v i and v i , implying that the accuracy of original GCN is preserved. Compared Eq. 5 with Eq. 3, we can find that Eq. 5 makes the same transformation on all the dimensions of v i , which will limit the learnable calibration function space. However, we will prove that actually Eq. 5 is the same with the model proposed in Eq. 3 on confidence calibration using the Proposition 2. Considering that for any logit v i , our expectation is in fact that the calibration model can output any confidencep i ∈ ( 1 K , 1). Please note thatp i ≥ 1 K , or the prediction will be changed. Since Eq. 3 has no limitation on the learnt calibration model, its outputp i can take any value from 1 K to 1. Therefore, if we can prove the outputp i in Eq. 5 can also traverse the interval ( 1 K , 1) for any v i , the equality between Eq. 3 and Eq.5 can be proved.

Proposition 2. Given the original logit
T of node i, assume v i, j not approaching infinity for each j ∈ {1, · · · , K}. The calibrated confidencep i in Eq. 5 can traverse the interval ( 1 K , 1) for node i.
Proof We set v i,1 > v i,2 > · · · > v i,K without loss of generality. For any v i ∈ R K , with the assumption of v i not approaching infinity, we have that (6) and Obviously, both The assumption about v i is easy to be satisfied since the L2-norm in GCN drives the weight matrix W approaching zero matrix and each element in node feature matrix X is not infinity. Therefore, based on Eq. 2, each element v i, j in V cannot approach infinity. From Proposition 2 we know that for any v i , there exactly exists such a unique temperature t i thatp i can take any value from 1/K to 1. In other words, the model can be perfectly calibrated.

Optimization Objective
Since NLL loss [10] can be decomposed into calibration loss and refinement loss [21], minimizing NLL loss benefits for confidence calibration. Therefore, we employ the NLL loss as our objective function with an additional regularization term. We use the prediction probability z i ∈ R K in Eq. 5 to calculate the NLL loss. Denote the K-class one-hot label for node i as y i = [y i,1 , · · · , y i,K ] T and suppose the size of the validation set is |D val |. Then the NLL loss over all validation nodes is represented as L nll where: Due to the under-confidence of GCNs, our goal is to increase the confidence of correct predictions while decreasing that of incorrect predictions. Considering that for incorrect predictions, the NLL loss cannot directly reduce their confidence, therefore, we design a regularization term for NLL loss as follows: where |cor| and |inc| are the number of nodes correctly and incorrectly predicted and z i,m and z i,s are the max and submax prediction probability. Intuitively, the confidence of incorrect predictions is decreased by reducing the gap between the max and the submax value of z i and vice versa. Combining L nll and L cal , we have the following overall objective function: where λ is the parameter of the regularization term. With the guide of labeled data, we can optimize CaGCN via back propagation and learn the calibrated confidence. The overall framework of CaGCN is shown in Fig. 4.

Self-training with Confidence Calibration
Here we propose a practical application of confidence calibration to improve the performance of self-training in GCNs. Self-training is to predict the labels for unlabeled data, and then add them to the training set, so as to achieve better performance. When applying self-training to GCN, we firstly obtain the predictionsŷ i and the confidencep i given by GCN and then add the most confident nodes to the training set with pseudo labelsŷ i based onp i . We continue to train until convergence. However, existing self-training methods perform not as expected with higher label rates [30]. Considering the under-confidence of existing GCNs, motivated by [26], we argue that the under-performance of Figure 4: The overall framework of CaGCN. Solid lines represent that we can backpropagate gradient here while dashed lines represent we cannot. We firstly train a classification GCN using the training set to obtain the logit V of all the nodes. Then we feed V to CaGCN to get the temperature t and transform V using t into V . Finally, the loss can be obtained using V after softmax according to Eq. 10 and CaGCN can be optimized with the guide of the validation set.  existing self-training methods originals from large numbers of high-accuracy predictions distributing in low-confidence intervals as shown in Fig. 2, causing that they cannot be added to the training set.
Consequently, we design a self-training model CaGCN-st where confidence is firstly calibrated then employed to generate pseudo labels for unlabeled nodes. Specifically, given an unlabeled dataset D U and a labeled dataset D L which has been divided into three parts D train , D val and D test , we firstly train a classification GCN using D train to get the logit of each node. Then all the logits will be fed into a CaGCN to train and we get a calibrated confidence for each node. It should be noted that instead of D val , we still employ D train to train our CaGCN. After that, the most confident predictions of D U will be adopted as the pseudo labels according to a threshold τ and added to the label set. The D train is enlarged in this way. The process above will be repeated s stages until convergence. Please note that our classification GCN and CaGCN are re-initialized in each stage.

Experiments
In this section, we evaluate the performance of CaGCN on confidence calibration and CaGCN-st on self-training respectively. We choose the commonly used citation networks Cora [29], Citeseer [29], Pubmed [29] and CoraFull [3] for evaluation, and more detailed descriptions are in Appendix B.

Confidence Calibration Evaluation
Baselines. Since our CaGCN is a general calibration model for GNNs, here we choose GCN [16] and GAT [33] as our classification models. For comparison, we choose the classic post-hoc calibration methods temperature scaling (TS) [12] and matrix scaling with off-diagonal regularization (MS) [18] as our baselines.
Experimental settings. For the base model GCN and GAT, i.e., the uncalibrated model, we follow parameters suggested by [16] and [33] and further carefully tune them to get optimal performance. For the post-hoc calibration technique, we follow the official implementation [12,18]. For our CaGCN, we train a two-layer GCN with the hidden layer dimension to be 16. We set λ = 0.5 for all datasets, weight decay to be 5e-3 for Cora, Citeseer, Pubmed and 0.03 for CoraFull. Other parameters of CaGCN follows [16]. We evaluate the performance of confidence calibration by ECE [22], NLL [10] and Brier Score (BS) [4], which we expect are smaller, and we set the bin number M = 20 for ECE (more details can be seen in Appendix A). For all methods, we randomly run 10 times and report the average results. More detailed experimental settings can be seen in Appendix B.
Results. Table 2 reports calibration results evaluated by ECE (more results on NLL and Brier Score are in Appendix C.1). We have the following observations: (1) Compared with uncalibrated models and other baselines, CaGCN is statistically significantly better at the * 0.05 level and ** 0.01 level.
(2) The ECE values on uncalibrated models are generally the highest, implying that GCN and GAT are poorly calibrated. (3) MS behaves badly on datasets with many classes, e.g., CoraFull. This is because the number of parameters for matrix scaling scales quadratically with the number of classes while the size of the validation set keeps unchanged. Therefore, it will over-fit to the small validation set when dataset has a great number of classes. However, CaGCN does not have this problem.
Additional analysis. In Section 2 we visualize the under-confidence problem of existing GNNs using reliability diagrams. Here we utilize the same visualization method to make a comparison before and after confidence calibration. As shown in Fig. 8, Fig. 9, Fig. 10 and Fig. 11 in the appendix, we can find that the confidence is well-calibrated after calibration.

Classification Evaluation of Self-Training
Baselines. Since self-training can be applied to any models, here we choose GCN and GAT as our base models, i.e., the original models (Orig.) without self-training, and we choose self-training (St.), co-training (Ct.), Union, Intersection (Inter.) methods proposed in [20] for comparison, which are commonly used as the baselines in self-training. Furthermore, we employ TS as the confidence calibration function in CaGCN-st as another baseline and we denote it by TS-st.  [20] and we further carefully tune them to get optimal performance. For all methods, we randomly run 10 times and report the average results.
Results. Table 3 summarizes the node classification accuracy on GCN and its self-training variants.
More results on GAT can be seen in Appendix C.2. We have the following observation: (1) CaGCN-st consistently outperforms all the baselines on all the datasets and label rates at the * 0.05 level. (2) Compared with the base model, self-training methods generally achieve better results, which proves their effectiveness. (3) Self-training methods with confidence calibration (i.e., TS-st and CaGCN-st) have better performance, which implies that confidence calibration scales more correct predictions to the high confidence range while keeps incorrect predictions basically unchanged, which we believe is beneficial for self-training.  Table 4, and we can observe that CaGCN-st consistently outperforms GCN-st on all the datasets, implying that self-training with calibrated confidence can generate more correct pseudo labels.
Additional analysis. We also investigate the changing trends of accuracy with respect to the threshold τ in CaGCN-st in Appendix C.2 and study why GCNs are poorly calibrated in Appendix D.

Related Work
Graph Neural Networks. Modern GCNs mimics CNNs to learn the local and global structural patterns of graphs through designed convolution and readout functions. [5] generalizes CNNs to graph signal based on the spectrum of graph Laplacian. ChebNet [7] uses Chebyshev polynomials to approximate the K-order localized graph filters and GCN [16] further employs the 1-order simplification of the Chebyshev filter. GAT [33] utilizes attention mechanisms to adaptively learn aggregation weights. GraphSAGE [13] uses various ways of pooling for aggregation. [20] introduces self-training to GCNs and [30] proposes a multi-stage self-supervised (M3S) self-training algorithm of GCNs. Both [20] and [30] focus on the few-shot learning and neither has ever explored self-training with higher label rates in GCNs. More works on GNNs can be found in surveys [36,43], however, to the best of our knowledge, current GNNs have not considered the confidence calibration.
Confidence Calibration. Confidence calibration has been studied for a long time in CV and NLP [12,23,25,14,19,37,40,42]. [12] discovers modern neural networks are poorly calibrated and study factors influencing calibration. Platt scaling [24] is a simple post-hoc calibration method for binary models, which transforms the logit using scalar parameters. Temperature scaling is the simplest multi-class extension of Platt scaling and matrix and vector scaling are another two extensions of platt scaling. [40] proposes Mix-n-Match calibration strategies which mix parameter methods with non-parameter methods. [25] explores the non-linear space for post-hoc calibration function using a neural network. Moreover, [32] points out GNNs can be miscalibrated in the supervised scenario and mainly focus on miscalibration originated from the imbalanced class distribution. However, none of them have considered the confidence calibration in GNNs on the common semi-supervised scenario.

Conclusion
Current efforts on advancing GNNs mostly focus on classification accuracy. However, when deploying GNNs to real-world applications, especially safety-critical fields, whether the results of GNNs are trustworthy is another important factor that cannot be neglected. In this paper, we study the confidence calibration problem in GNNs and discover existing GNNs are under-confident on their predictions. To solve this problem, we propose a novel trustworthy GNN model CaGCN which respects the homophily property of confidence in GNNs and preserves the classification accuracy. Moreover, we propose a novel self-training method CaGCN-st where confidence is first calibrated by CaGCN and then used to generate pseudo labels. Extensive experiments demonstrate the effectiveness of our proposed model in terms of both calibration and accuracy.
An interesting direction for future work is to extend CaGCN to other graph tasks, but more studies need to be conducted. We take the link prediction as an example, where we can regard the link prediction as a binary classification problem and the output as the confidence. Considering that the ground-truth confidence distribution for nodes should have the homophily property as is shown in Section 3.1, edges are likely to have the same property as well. As a result, we can employ CaGCN to propagate the confidence between edges by regarding the edges as the nodes. However, more exploration still needs to be conducted for the homophily property of edges.
Broader impact. Current efforts on advancing GNNs mostly focus on classification accuracy. However, when deploying GNNs to real-world applications, especially safety-critical fields, whether the results of GNNs are trustworthy is another important factor. The demands for a trustworthy model are universal and extensive such as in the field of disease prediction [31], traffic states prediction [6] and object detection [11] for autonomous driving, where estimating the true probability of getting a correct prediction is necessary. We take the disease prediction [31] as an example, where GNNs are utilized to encode the information of different symptoms, users and diseases. In this scenario, accurately and comprehensively predicting diseases at an early stage will help patients receive prevention treatments in a timely manner. Otherwise, the misdiagnosis and missed diagnosis will endanger the health of patients. Therefore, a trustworthy model is urgently needed. Our CaGCN can make a trustworthy prediction based on its confidence, and as a result, decrease the risk of misdiagnosis and missed diagnosis. We hope our work can provide insights for future improvements in tackling the trustworthiness problem in other saftey-critical fields.
Limitations. One potential issue of this work is that it provides a limited explanation to the underconfidence problem. We advocate peer researchers to look into this, making GNNs more reliable in different domains. Other than that, since this work is mostly on the discovery of the confidence calibration problem in GNNs and the theoretical aspect of improving calibration, we do not foresee any direct negative impacts on the society.

A.1 Expected Calibration Error (ECE)
Expected calibration error (ECE) [22] is a common calibration metric, which measures the difference in expectation between confidence and accuracy: wherep i is the confidence for node i,ŷ i is the prediction, y i is the label and p is the true probability thatŷ i is correctly predicted. However, we cannot exactly know the true probability p, thus Eq. 11 cannot be computed directly. The approximation to ECE is generally employed as the metric: where M is the number of equally-spaced bins (similar to the reliability diagrams as mentioned in Section 2 that predictions are partitioned into), |B m | is the number of predictions falling into the m-th bin according to their confidence and acc(B m ) = 1 |B m | ∑ i∈B m )pi represent the average accuracy and confidence in each bin respectively. The difference between acc and conf can be intuitively seemed as the deviation of the outputs to the diagonal in Fig. 1.

A.2 Brier Score (BS)
Brier Score (BS) [4] is another commonly used calibration metric, which measures the accuracy of probabilistic predictions. The higher the accuracy of predictions is, the lower BS is. For any given predictionŷ i , BS is the lowest when the prediction probability z i is exactly equal to the true probability thatŷ i is correct. Given the one-hot label y i for node i, BS can be represented as follows: B More Experimental Details

B.1 Datasets and Environment
We choose the commonly used Cora [29], Citeseer [29], Pubmed [29] and CoraFull [3] for evaluation, where nodes represent papers, edges are the citation relationship between papers, node features are comprised of bag-of-words vector of the papers and labels represent the fields of papers. We choose 500 nodes for validation, 1000 nodes for test and select three label rates for the training set (i.e., 20, 40, 60 labeled nodes per class). The details of these datasets are summarized in Table 5.
Our data are public and do not contain personally identifiable information and offensive content. The address of our data is https://docs.dgl.ai/en/latest/api/python/dgl.data.html# node-prediction-datasets and the license is Apache License 2.0. The environment where our code runs is shown as follows: • Operating system: Linux version 3.10.0-693.el7.x86_64 • CPU information: Intel(R) Xeon(R) Silver 4210 CPU @ 2.20GHz • GPU information: GeForce RTX 3090

B.3 Additional Experimental Details for Self-Training
For the four baselines self-training, co-training, union, intersection proposed in [20], the official implementation is in https://github.com/Davidham3/deeper_insights_into_GCNs and we implement them in Pytorch. For our new proposed TS-st, the learning rate is 0.001, maximum number of iteration is 50 for Cora, Citeseer, Pubmed and 25 for CoraFull. More detailed experimental settings can be seen in Table 9.

B.4 Other Source Code
The acquisition of all the code below complies with the provider's license and do not contain personally identifiable information and offensive content. Our code can be found in the supplemental material and all the related experimental details (e.g., environment, experimental settings for our methods and all the baselines) are included in README.

C.1 Additional Results for Calibration
NLL and BS. Since NLL and BS can be affected by the accuracy of predictions, we should keep the accuracy unchanged when evaluating the performance of calibration methods. However, the accuracy-preserving property cannot be satisfied by MS. Therefore, we omit it from our baselines for fairness. Table 6 and Table 7 report calibration results evaluated by NLL and BS. We find that CaGCN is generally better than other baselines at * 0.05 level and ** 0.01 level.
Reliability diagrams. In Section 2 we visualize the under-confidence problem of existing GNNs using reliability diagram. Here we still employ the same method to make a comparison before and after confidence calibration. Fig. 8 and Fig. 9 show the reliability diagrams on different models and networks of various label rates before (odd rows) and after (even rows) calibration. We can see that the confidence of predictions on all the datasets and models is well-calibrated. Moreover, for further verifying our conclusion that GNNs are under-confident, we also demonstrate the reliability diagrams of another four representative GNNs (GraphSAGE [13], APPNP [17], SGC [35], GIN [38]) on all the datasets with 20, 40, 60 label rates. The results are summarized in Fig. 12, 13, 14. Similarly, we can observe that in almost all the datasets, the average accuracy of most bins is higher than the average confidence, which means these models are also under-confident, verifying our conclusion again.
Confidence distribution. In Section 2 we visualize the confidence distribution as a supplement for the under-confidence problem of existing GNNs. Same as before, here we visualize the confidence distribution before (odd rows) and after (even rows) confidence on different models and networks of various label rates to make a comparison. As shown in Fig. 10 and Fig. 11, we can see that a large quantity of correct predictions have been transformed into a higher confidence range while incorrect predictions change little.

D Why GCNs are poorly calibrated
In this section, we focus on the reason why GNNs are poorly calibrated. It is inspired by the observation in [12] that modern neural networks can overfit to NLL without overfitting to the accuracy. Since NLL can be used to measure model calibration as mentioned in Section 3.3, [12] gives an explanation of miscalibration: modern neural networks achieve better classification accuracy at the expense of well-calibrated probabilities. Similarly, we conduct an experiment to explore the relationship between NLL and the accuracy of GNNs.
We take the representative GCN as the example and apply it to Cora, Citeseer, CoraFull with label rate L/C = 20. We employ the validation set for early stopping in training with a window size of 100 and carefully tune the parameter α for weight decay to obtain the best accuracy and NLL on the test set respectively. Other parameters follow [16]. The changing trends of accuracy and NLL with respect to epoch are shown in Fig. 7, where NLL is scaled by a constant to fit the figure. Intuitively, GCN should achieve the best accuracy when NLL is the lowest. However, we find that GCN does not achieve the best performance when NLL is the lowest. Taking the Cora dataset in Fig. 7(a) as an example, we find that GCN achieves the best accuracy when α = 5e − 4 but NLL still under-fits at this time, i.e., NLL has not achieved the lowest value. If we tune α to be 1e − 5, NLL generally achieves the best result, i.e., GCN is better calibrated at this time, while accuracy drops from 81.5% to 79.5%. This gives an explanation of miscalibration for GNNs: GNNs learn better classification accuracy at the expense of well-modeled probabilities, i.e., GNNs under-fit to NLL without under-fitting to accuracy.