{'fraction_non_alphanumeric': 0.03193013577130381, 'fraction_numerical': 0.00828955447908849, 'mean_word_length': 4.911474087517645, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Bayesian Graph Contrastive Learning

Contrastive learning has become a key component of self-supervised learning approaches for graph-structured data. Despite their success, existing graph contrastive learning methods are incapable of uncertainty quantification for node representations or their downstream tasks, limiting their application in high-stakes domains. In this paper, we propose a novel Bayesian perspective of graph contrastive learning methods showing random augmentations leads to stochastic encoders. As a result, our proposed method represents each node by a distribution in the latent space in contrast to existing techniques which embed each node to a deterministic vector. By learning distributional representations, we provide uncertainty estimates in downstream graph analytics tasks and increase the expressive power of the predictive model. In addition, we propose a Bayesian framework to infer the probability of perturbations in each view of the contrastive model, eliminating the need for a computationally expensive search for hyperparameter tuning. We empirically show a considerable improvement in performance compared to existing state-of-the-art methods on several benchmark datasets.


Introduction
Self-supervised contrastive methods have shown great promise to produce high-quality node representations by learning to be invariant to different augmentations in graph domain [24,30,29,28]. While these contrastive learning methods have achieved great empirical performance, to the best of our knowledge, they are limited to learn deterministic node embeddings. Such deterministic representations lack the capability of modeling uncertainty, which is a natural consideration when having multiple information sources, i.e. node attributes and graph structure, and is of crucial importance in many real-life applications such as healthcare and autonomous driving [12,18]. Another main challenge in Graph Contrastive Learning (GCL) methods is the proper choice of augmentation types and their hyperparameters. There has been recent works that attempt to address this issue by either assuming augmentation operation to be an invertible function [26] or only learning the probability of picking a specific type of augmentation and do not learn the parameters of the augmentations [28].
In this paper: 1) Instead of applying augmentations over input to make different views of data for GCL training, we propose a novel generalized augmentation which is applied to every layer of encoders. Our proposed method can be viewed as a stochastic regularization technique rather than an augmentation method. We theoretically prove that this is equivalent to having stochastic encoders. We further show that existing augmentation techniques are special cases of the proposed generalized augmentation, eliminating the need for choosing augmentation type. 2) We propose Bayesian Graph Contrastive Learning (BGCL) which is the first model in contrastive learning literature that enables Bayesian interpretability and provides uncertainty estimates for latent representations and downstream predictions. We theoretically prove that a GCL model with our generalized augmentation is an approximation of a Bayesian model, hence equipped with natural uncertainty quantification capabilities. 3) We propose the first Bayesian approach for learning parameters of augmentation.
Instead of a computationally expensive grid search for finding the optimal augmentation hyperparameters in existing works [28], BGCL learns the parameters of augmentation together with node embeddings (end-to-end training). We empirically demonstrate advantages of BGCL over existing methods on several benchmark dataset, in terms of uncertainty quantification and interpretability of latent representation.

Graph contrastive learning
Inspired by the success of contrastive methods in vision, graph contrastive learning methods have been introduced to learn node embeddings by minimizing a contrastive loss between two augmented views of the same graph. Random corruptions are applied to input graph to obtain augmented views. The corruptions are chosen form a set of corruption types which usually includes T = {NodeDrop, EdgeDrop, FeatureDrop, SubgraphSelection}. Although several graph contrastive losses have been proposed [27,30,25], the main goals of these objective functions are: 1) enforcing the representations of the same node in the two views of the graph to be close to each other, and 2) pushing apart representations of every other node.
To achieve the first goal, the vast majority of the methods include a term that measures the similarity between the representations of the same nodes in the two views. For the second goal, different approaches have been proposed. For example, GRACE [30] deployed negative samples, like SimCLR [2], while BGRL [25] uses different encoders for each view in the same way as BYOL [8]. Without loss of generality, we carry out our analysis based on GRACE. We note that the same analysis could be applied to other methods as well.
Given a graph G = (A, X) with A ∈ R N ×N as adjacency matrix and X ∈ R N ×F0 as attributes matrix, GRACE generate two views of the graphG o = (Ã o ,X o ) andG t = (Ã t ,X t ) by sampling from a random corruption function. Subscripts o and t are used for indexing two views. Next, a graph neural networks (GNN) f , is deployed to map the augmented graphs to latent representations, H o and H t , independently. The parameters of encoder (i.e. f ) are learned by minimizing the following loss function: where τ is a hyperparameter, λ is a 2 weight decay hyperparameter, g is a similarity function, L is the number of layers in encoder, and W (j) represents the parameters of the encoder at j-th layer. It has been shown that L CNT in the above equation is a lower bound for InfoNCE objective [30].

Method
In this section, we introduce Bayesian Graph Contrastive Learning. Our proposed method: 1) infers an implicit distribution as latent representation of each node, as opposed to deterministic vector representations in existing methods, allowing to capture uncertainty; 2) models augmentation masks as discrete random variables and infers their distribution using variational inference; 3) provides an approach to translate the uncertainty of representation to the downstream predictive models.

Bayesian graph contrastive learning
We are interested in developing a Bayesian interpretation of contrastive models which can be used to quantify uncertainty of representations. To that end, we will: 1) introduce a generalized augmentation model and show how the stochasticity of augmentations can be perceived as randomness in the parameters of the encoders; 2) show that training a graph contrastive model with our generalized augmentation model is an approximation of a Variational Graph Auto-Encoder (VGAE) [14] with stochastic functions as encoders.

Generalized augmentation model
We first define a generalized augmentation method that includes the existing augmentation types as special cases. We note that all of the augmentations in T (section 2) are simply parameterized by a random binary matrix. For example, FeatureDrop can be formulated asX = X Z , where is the Hadamard product, andZ is a sample drawn from a random Bernoulli matrix.
Unlike existing works where augmentations are only applied to the first layer (i.e. input data) of encoders, we extend the notion of augmentation and propose to multiply the output of each layer of encoders by a random mask. These masks could be drawn from different types and/or have different probability distributions. This extension allows us to view augmentation as a stochastic regularization technique for GNNs. Indeed, conventional augmentation is a stochastic regularization that is only applied to the first layer of encoders.
Furthermore, to increase flexibility of the model, we propose to multiply a different random mask with the adjacency matrix for every input-output feature pair [10]. More specifically, the (l + 1)-th layer of graph neural network in an encoder with generalized augmentation is defined as follows: where U (l) ∈ R N ×F l is the matrix of features at layer l with U (0) = X, F l is the number of features at layer l, σ is the activation function, g a is the adjacency matrix normalization operator [15], and the elements ofZ (l,i,j) are drawn from Bernoulli(π (l) ). It can be easily shown that existing augmentations are special cases of our proposed method. For example, choosingZ (l,i,j) = 1 N ×N diag(Z (l) [:,i] ) and π (0) = 1 while {π (l) = 1} L−1 l=1 leads to FeatureDrop. We emphesize that our proposed generalization eliminates the need for choosing augmentation type.
Having defined our generalized augmentation model, we show that the randomness could be transferred from input to parameters of encoder. To that end, we rewrite and reorganize equation 2 to have a node-wise view of the layers. More specifically, where c v is a constant depending on the graph structure, N (v) is the first hop neighborhood of v, and W . This can be interpreted as learning a different random weight for each connection in a GNN layer. We refer toW (l,u,v) ] as connection-specific weights.

Bayesian interpretation
Bayesian models offer a mathematically grounded framework to reason about model uncertainty [5]. If we show that GCL with generalized augmentation is an approximation of a Bayesian model, then we can quantify model uncertainty. To that end, we show that a GCL with generalized augmentation is a special case of VGAE by proving L CNT + L WD (equation 1) is a valid ELBO.
Consider a VGAE with two latent variables H o and H t and two graph neural networks, with connection-specific weights , and the approximate posterior is factorized as We can write the ELBO for this VGAE as follows: (3) Note that we used Jensen's inequality in the derivation of ELBO above. Following the same procedure as Aitchison [1], we substitute the following likelihood: in the definition of L VGAE and derive: The last term in the above equation, i.e. log p true (G), is independent of parameters, hence it does not affect the optimization and could be treated as a constant. Next, we parameterize the prior distribution latent representations in the same way as Aitchison [1]: (6) It is straightforward to show that the first term in the definition of L w BGCL (equation 5) is equal to infinite limit of I NCE or equivalently −L CNT in equation 1.
Finally, we parameterize prior and posterior of W o and W t such that is proportional to L WD in equation 1. More specifically, we parameterize the posterior of weights as follows: where δ(·) is the Dirac delta function, and {π are the parameters of the posterior. Note that we use the same form of parametrization for both views and for brevity we wrote the posterior for one view only. We share θ w = {M (l) } L l=1 between two views while having separate drop probabilities for each view. To be able to evaluate the KL term analytically, the discrete quantised Gaussian can be adopted as the prior distribution as in [6]. Then, we can derive the KL divergence between the prior and posterior as: where H is the Shannon entropy of Bernoulli random variable. When the augmentation probabilities are fixed, H(π (l) t ) and H(π (l) o ) are constants, hence the KL divergence is only proportional to 2 norm of the weights. Therefore, with appropriate choice of 2 regularization hyperparameter, it will be equal to L WD in equation 1. Our analysis in this subsection showed that training a GCL model with generalized augmentation is an approximation of a VGAE with stochastic encoders.
The proposed BGCL offers a novel and interesting perspective of contrastive learning. Specifically, it could be interpreted that BGCL has no augmentation over the input and the embeddings are generated by feeding the original data to two different stochastic functions (i.e. moving stochasticity from data to function). Indeed, our unique construction of stochastic encoders allows the model to mimic data augmentation. Whether any other type of stochastic function, such as neural processes, works the same way or not is an interesting avenue for future works.

Learnable Bayesian graph contrastive learning
Finding optimal parameters for augmentation distributions is subject to computationally expensive and time consuming search over a set of possible hyperparameters [28]. Here, we propose a Bayesian approach with variational inference to learn the parameters of our generalized augmentation. We impose independent hierarchical beta-Bernoulli priors over augmentations and assume where c is a hyperparameter. Furthermore, we parametrize the posterior using Kumaraswamy-Bernoulli hierarchical distribution [16] as follows: where a o are variational parameters. Note that we use the same form of parameterization for prior and posterior of both views and for brevity we wrote the prior and posterior for one view only. We denote the parameters of our augmentation model by θ a = θ a,o ∪ θ a,t = {a l=1 . Furthermore, we represent the set of latent augmentation masks by , and the set of augmentation drop rates by The main goal here is learning "good" augmentations which should be reflected in the likelihood function. However, before defining the likelihood, we have to define what makes "good" augmentations. Tian et al. [26] argues that a good set of views are those that share the minimal information necessary to perform well at the downstream task which is equivalent to maximizing contrastive loss. Following the same principle, we define a likelihood that places more mass on views with lower mutual information. Let's consider the same VGAE that was introduced in the last subsection. While there we assumed that weights are random variable with fixed augmentation drop rates, here we assume that θ w = {M (l) } L l=1 are fixed and θ a are variational parameters. Following the same analysis as last section, we can write the ELBO for this model as follows: Let's assume that the prior of latent representation are defined as follows: By substituting the above priors in equation 11, and using Jensen's inequality, we can see that the first term in L a BGCL is bounded by the infinite limit of −I NCE or equivalently L CNT . Furthermore, we can expand the KL terms in L a BGCL as follows: Since the conditional posterior and prior of masks have the same distribution, i.e. Bernoulli, the KL divergence between them is zero. Hence, the KL term only depends on the KL divergence between posterior and prior of augmentation drop rates, which can be expanded as follows: where c o is a hyperparameter, γ is the Euler-Mascheroni constant and Ψ(·) is the digamma function.
Note that for brevity we wrote the last two equations for one view only.
Due to the discrete nature of the random masks Z, we cannot directly apply reparameterization trick to calculate the gradient of the KL term. Hence, we use continuous approximation, i.e. concrete relaxation [11,6], of Z instead of discrete variables. Given a drop probability π and a sample from uniform distribution u ∼ Unif[0, 1], a sample from concrete distribution can be calculated as follows: where t is temperature parameter. We emphasize that to the best of our knowledge, this is the first method that models augmentations as discrete random variables which offers modeling more diverse types of augmentation.
Having defined the augmentation learning method, we describe the overall learning procedure. We have two objective functions: 1) the objective function for contrastive model (equation 1 or equivalently equation 5) which is used for learning the parameters of encoders θ w , and 2) the ELBO for our Bayesian augmentation learning method (equation 11) which is used for learning the parameters of augmentation θ a . Combining these objective functions, we derive the overall objective of our model as: min In each training epoch, we first optimize the parameters of encoders by minimizing the contrastive loss. Then we optimize the parameters of augmentations by maximizing the ELBO of our Bayesian augmentation learning method.

Downstream predictive model
In section 3.1.2, by performing varitional inference for BGCL, we infer the posterior distribution of weights of model. By sampling from posterior of the weights of the model and passing the input graph to the model, we will have samples from the approximate posterior of node representations. In fact, these are samples from an implicit variational approximation Q(H) of the true posterior distribution. Let's consider the node classification task where y = {y v } v∈V are the set of observed node labels in the graph. We model the following probability distribution: where H is the matrix of node representations, W c is the parameters of the multinomial logistic regression, and p cat is probability mass function of categorical distribution. We can calculate the likelihood of y as follows: p(y) = p cat (y; softmax(H W c )) Q(H) dH.
Similar formulation can also be obtained for the regression task by replacing the categorical distribution with normal density with a fixed variance and ignoring softmax function for the mean.
In the non-Bayesian contrastive learning setup, a common practice for learning the parameters W c is via maximum log-likelihood. We also take the same approach, however, we do not have a closed-form likelihood. We circumvent this problem by using the Monte Carlo estimate of the likelihood (equation 14) as: and learn parameters of logistic regression W c by maximizing the logarithm of the above term via batch-optimization. Our model is more flexible with higher expressive power compared to the existing methods. More specifically, based on equation 15, the proposed framework learns a mixture of categorical (normal) distributions rather than a single one for classification (regression) task. Therefore, it is less prone to misspecification, providing more reliable predictions.

Experiments
Datasets. We use three standard graph datasets to benchmark the performance of BGCL as well as baselines. Cora [22], Citeseer [22], and Amazon-Photos [17]. Cora and Citeseer are citation datasets where each node represents a document and the edges indicate the citation relations. Each node is provided with a multi-dimensional binary attribute vector, where each dimension of the attribute vector indicates the presence of a word from a dictionary in the publication. We follow the same pre-processing steps as [30] and randomly split the nodes into train (10%) and test (90%) sets. Amazon-Photo is from the Amazon co-purchase graph where nodes represent products and edges are between pairs of products frequently purchased together. Products are classified into eight classes, and node attributes are a bag-of-words representation of a product's reviews. We follow the same pre-processing steps as [25] and randomly split the nodes into train/validation/test sets (10/10/80%), respectively.
Baselines and experimental setups. Since GRACE achieves the state-of-the-art results in graph contrastive learning [25], we use it as the base contrastive model in our BGCL. More specifically, we use our learnable Bayesian generalized augmentation approach on top of GRACE. We emphasize that our proposed model could also be used on top of any graph contrastive model which deploys T as the set of augmentation types. For a fair comparison with baselines, we use the same set of hyperparameters as the ones reported in GRACE original publication [30] for BGCL except for the augmentation drop rates which are learned in the training process. The prior distribution for all of the drop rates, π (l) s, is Beta(1, 1). The temperature in concrete distribution is 0.3. The linear classifiers are trained for 50 different random splits and random initializations. We implemented our model in PyTorch [19]. All of our simulations are conducted on a single NVIDIA Tesla P100 GPU node.
We compare our BGCL with several baselines in terms of node classification accuracy. More specifically, we compare our BGCL with two contrastive methods (GRACE [30] and DGI [24]), two auto-encoders (VGAE and GAE [14]), two conventional node embedding methods (node2vec [9] and DeepWalk [21]), and supervised GCN [15]. We use the same setup for baselines as described in [30]. Since there are no contrastive models with uncertainty quantification capabilities, we compare our BGCL with a supervised model, Bayesian GCN with Graph DropConnect (GDC) [10]. We use the same setup as [10] to quantify uncertainty for datasets. More details on experimental setups and datasets are included in the supplementary materials.

Uncertainty quantification in node classification
Ideally, if a model is confident about its prediction, the prediction should be accurate (i.e. high p(accurate | certain)). Also, if a model makes a mistake in the prediction, it should be uncertain about its output (i.e. high p(uncertain | inaccurate)). These two probabilities can be combined into one to form metric for quality of uncertainty called Patch Accuracy Vs Patch Uncertainty (PAVPU) [18]. More specifically, PAVPU = (n ac + n iu )/(n ac + n au + n ic + n iu ), where n ac is the number of accurate and certain predictions, n au is the number of accurate and uncertain predictions, n ic is the number of inaccurate and certain predictions, and n iu is the number of inaccurate and uncertain predictions. A model with higher PAVPU quantifies uncertainty more accurately. We use predictive entropy (as defined in [18]) as a measure for certainty.  Figure 2 shows performance of BGCL and a supervised Bayesian GCN model in uncertainty quantification on benchmark datasets. Comparing these two models, we see that BGCL is outperforming (or performs on par with) competing method on most of the certainty thresholds. This is indeed very impressive considering that BGCL is a self-supervised model and baseline is a supervised model. This empirically proves that our novel Bayesian formulation of graph contrastive learning allows us to accurately capture uncertainty of latent representations.
PAVPU alone may not depict the full picture of performance of a model. Therefore, we also report the classification accuracy for our model and compare it with the existing baselines in Table 1. The results show that BGCL outperforms the unsupervised and self-supervised baselines in all datasets. We emphasize that our goal for reporting classification accuracy is not claiming stateof-the-art performance in node classification. Rather, we want to show that our learnable generalized augmentation scheme is able to learn optimal drop rates without any supervision and obviates the need for ad-hoc procedure for selecting augmentation type. Indeed there are other supervised methods that outperform contrastive models. However, the main focus of this paper is self-supervised setting and the reported supervised GCN (like majority of works in graph contrastive learning) is merely for showing the relative performance of self-supervised techniques with a well-known supervised model.

Representation uncertainty
We provide results showing how uncertainty of representations increases the interpretability of our model. Specifically, we show that adding noise to input data (which can be seen as out of distribution samples), will increase the uncertainty of representation.
To that end, we first train three BGCL models on Cora, Citeseer and Amazon-photo datasets. Then, we synthesize noisy versions of these datasets by randomly selecting a few nodes (10 for Cora and Citeseer, and 100 for Amazon-photo), and replace their attributes with normally distributed noise (N (0, 1) for Cora and Citeseer, and N (0, 1) and N (0, 10) for Amazon-photo). We further train three other BGCL models on these noisy datasets. Then, we examine the difference between the Average Standard Deviation (ASTD) of latent representations of the randomly selected nodes in the two cases for each dataset. We do the same examination for k-hop neighbors (for k = 1, · · · , 5) of  We can see that the noisy nodes have the highest increase in ASTD. The reason for higher ASTD in noisy cases is that information mismatch between the attribute of noisy nodes and their neighbors. As we go farther away from the noisy nodes, the difference gets closer to zero. Noting that GCNs [15] are low-pass graph filters, this observation is consistent with evolution of a diffusion process over graph. Figure  4 shows the results for Amazon-photo dataset.
We can see that adding N (0, 1) does not effect the ASTD significantly. This can be explained by higher average degree of Amazon-photo (31.13) compared to Cora (3.90) and Citeseer (2.77). Having more neighbors allows the model to filter out noise, by taking advantage of the noise-free attributes of the neighbors via message passing, and learns embeddings with higher confidence. By increasing the standard deviation of noise to 10, we reduce the signal-to-noise ration showing an increase in uncertainty of the latent representations. This is consistent with our observation for Cora and Citeseer.

Conclusions
In this paper, we proposed Bayesian Graph Contrastive Learning (BGCL). First, we introduced a generalized augmentation method by applying random augmentation to each layer of encoders, obviating the need to choose augmentation type. Next, we show that a graph contrastive learning model with the proposed augmentation method, is an approximation of a VGAE with Bayesian neural networks as encoders. As a result, our BGCL represents each node by a distribution in the latent space instead of deterministic node embeddings. Hence, BGCL provides predictive uncertainty in downstream graph analytics tasks. Furthermore, we developed a Bayesian framework to infer the augmentation drop rates in each view of the contrastive model, eliminating the need for a computationally expensive search for hyperparameter tuning.

A Further results on learnable augmentation
While the main focus of the experiments/discussions in the main manuscript was uncertainty quantification, here, we show that our learnable generalized augmentation can be deployed in non-Bayesian scenarios as well. To deploy our proposed model in a non-Bayesian setting, we train our model as before. However, in testing phase, we don't sample from generalized augmentation. Instead, we feed the input graph to encoder without augmentation. This yields deterministic node representations. These deterministic representations could be used in downstream tasks in the same way as other graph contrastive models.
We report the results of our model in non-Bayesian setting for three additional datasets: WikiCS [20], Amazon-Computers [17], and CoauthorCS [23]. The details of datasets are reported in Table 3.
We use cluster batching [3] (as implemented in PyTorch Geometric package) during training of our model on these datasets. The number of batches for WikiCS, Amazon-Computers, and CoauthorCS are 5, 5, and 10, respectively. Rest of hyperparameters are reported in Table 4. For these datasets, we use the same experimental settings as [25], and report the baseline results from [25]. The results, reported in Table 2, show that our learnable generalized augmentation achieves on-par performance with the base model (GRACE) without any need for an expensive grid search to find optimal augmentation type or augmentation drop rates in a non-Bayesian setting.

B Sampling complexity
As described in the main manuscript, in our generalized augmentation model, we need to draw F l × F l+1 × E with E as number of edges. This could potentially be a large number. To alleviate this issue, we use a single sample for a block of features as opposed to drawing a new sample for every input-output feature pairs. This reduces the number of samples to B × E with B being number of blocks. The number of blocks used for each experiment is reported in Table 4.

C Further implementation details
All of the datasets used in our experiments are publicly available in PyTorch Geometric [4] package. Table 4 shows the hyperparameters used in our experiments for each dataset. LR w in Table 4 refers to learning rate for the first step of optimization while LR a denotes the learning rate for the second step of optimization. Note that, for a fair comparison, hidden dimension, latent dimenstion, activation function, and τ are the same as experiments in GRACE [30]. For simplicity, we assumed that the augmentation in all layers of a view have the same drop rate. All models are initialized with Xavier initialization [7], and trained using Adam optimizer [13]. For down-stream classification task, we have implemented our linear classifier in PyTorch. For all of the datasets, we train the classifier for 150 epochs using Adam optimizer with the learning rate of 0.1.
To calculate PAVPU, we first draw 500 random augmentations from the trained BGCL model leading to 500 embeddings for each node. Next, we train a classifier (as described in section 3.3) using 10 samples. Then, we deploy the trained classifier to predict the nodes' labels using the remaining 49 sets of 10 samples. Having 49 prediction for each node, we follow the same procedure as in [18] to calculate PAVPU.