{'fraction_non_alphanumeric': 0.0368577645543763, 'fraction_numerical': 0.010968515475820418, 'mean_word_length': 4.837221358216692, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Towards Robust Graph Contrastive Learning

We study the problem of adversarially robust self-supervised learning on graphs. In the contrastive learning framework, we introduce a new method that increases the adversarial robustness of the learned representations through i) adversarial transformations and ii) transformations that not only remove but also insert edges. We evaluate the learned representations in a preliminary set of experiments, obtaining promising results. We believe this work takes an important step towards incorporating robustness as a viable auxiliary task in graph contrastive learning.


INTRODUCTION
Imagine we have unlimited labeled training data and a prediction task to solve. Due to the results from recent years, our first approach is to try a deep-learning-based model. In various domains, including computer vision, natural language processing, or-more recentlygraphs, deep learning models have proven to set the state of the art, given they receive sufficient data.
How should we proceed when there is not enough data? In this scenario, the picture becomes less clear. Generally, finding the data is often not the main issue, but finding the labels for this data is. Recent developments in deep learning attempt to tackle this problem by resorting to self-supervised learning, where the labels are obtained by exploiting the internal structure of raw data. As one instance of this approach, contrastive learning methods have recently achieved impressive results [2,18]. On a high level, contrastive learning attempts to learn the representations by applying transformations to the input without fundamentally changing it. The goal is to make the representations of a single input under different transformations similar, while the representations of different inputs should differ. The field of graphs lends itself particularly well to this setup. Notably, the Web enables us to mine massive graphs from, for example, the Web structure itself or social networks. On the other hand, labeling graph data is challenging as labels should reflect the complex network structure. Zhu et al. [39] show that we can use contrastive learning successfully in this setting.
Similarly to contrastive learning, the subject of adversarial attacks also evolves around identity-preserving (imperceptible) transformations. However, adversarial attacks aim to find imperceptible transformations that, despite looking innocent, cause a misprediction in the model. It was shown that even highly accurate neural networks are vulnerable to such attacks [7,25,31], and thus unreliable, which is an especially important issue when they are used in safety-critical systems such as autonomous vehicles or face recognition systems. This lead to a great interest in building robust models, i.e., those less susceptible to adversarial attacks. In the contrastive learning setting, Kim et al. [19] recently demonstrate that using adversarial transformations allows us to learn robust representations. However, while the vulnerability to attacks is widely present in the graph domain [3,41], an investigation of adversarial transformations to learn robust representations is so far missing. In this paper, we explore this, and consider the use of adversarial transformations within the graph contrastive learning setting. We make the following contributions: (1) We propose Graph Robust Contrastive Learning (GROC), a fully self-supervised graph algorithm aiming to achieve robustness to adversarial attacks. To the best of our knowledge, we are the first to integrate adversarial transformations into the graph contrastive learning framework. (2) We conduct an evaluation of GROC on several popular transductive node classification datasets. The preliminary results show that GROC improves the robustness against adversarial attacks while maintaining a comparable performance on clean examples. (3) We outline possible future directions. We plan to extend our work to improve the efficiency of our method and extend our experiments to include a more comprehensive set of baselines and evaluation settings.

RELATED WORK 2.1 Graph Self-Supervised Learning
Early work on graph self-supervised learning focuses mostly on generative approaches. Hu et al. [13] propose to improve the performance of graph neural networks on downstream applications by using the tasks of link prediction, node ranking, and cluster recovery. Hu et al. [12] propose the pretraining tasks of attribute and edge generation, improving model performance on downstream tasks, including node classification, link prediction, and community detection. While the above approaches improve graph representations, recent breakthroughs in contrastive learning, mainly within computer vision [2,10], motivate the study of similar algorithms in the graph domain [9,27,28,30,33,36,39,40]. In contrastive learning, the model is supposed to discriminate between positive (similar) and negative (dissimilar) pairs. In the formulation that we focus on, Zhu et al. [39], further improved in Zhu et al. [40], create two views of Figure 1: A high-level illustration of our method, GROC. (a) We start by choosing a set of anchor nodes from the graph. (b) Then, we apply two stochastic transformations ′ 1 and ′ 2 that randomly mask node features, to obtain two distinct views. The graph is reduced to the union of receptive fields of anchor nodes-here 1−hop neighborhoods. (c) We apply adversarial transformations ′′ 1 and ′′ 2 to these two views, removing and inserting edges based on gradient signals. (d) Through contrastive learning, we force the embeddings of each anchor node in both views together (green arrows), and force the embeddings of different anchor nodes apart (red arrows).
the same graph by randomly removing edges and masking node features. In their work, positive pairs are the corresponding nodes in two views, while negative pairs are all other nodes in the same view (intra-view pairs) or the other view (inter-view pairs). You et al. [36] use a similar procedure but focus on graph-level representations. Note that none of these works, as opposed to ours, have robustness as one of their goals during training.
Orthogonally to our contributions, Grill et al. [8] recently suggest a novel self-supervised method that does not require negative pairs, improving the results of contrastive methods. This approach was subsequently adapted to the domain of graphs [1,32].

Graph Adversarial Attacks & Defenses
The main insights regarding robustness, that even highly accurate networks are vulnerable to adversarial attacks, directly transfer to graphs. Dai et al. [3] propose RL-S2V, an attack which learns to create misclassifications through edge insertion and removal with reinforcement learning. Nettack [41] crafts adversarial examples by perturbing the graph structure and altering the node features. Zügner and Günnemann [42] propose Meta-Attack, employing meta-learning to produce adversarial examples.
In response to these attacks, researchers propose various graph purification procedures [4,34], as well as adversarial defenses, training procedures designed to improve the robustness of neural networks, usually achieved by adversarially augmenting clean examples during training [22]. However, due to the discrete nature of edges and nodes in graphs, generating adversarial examples efficiently during training is hard [15,16]. Dai et al. [3] mitigate this problem by using examples with randomly dropped edges. Zhu et al. [38] propose Robust GCN, which absorbs the adversarial perturbations using Gaussian distributions as node representations in each layer of the network. Jin et al. [17] improve the adversarial robustness of graph neural networks by cleaning perturbed graphs through the intrinsic properties of real-world graphs such as low-rank adjacency matrices, sparse graphs, or homophily. Most recently, Zhang and Zitnik [37] suggest a more general approach that is able to defend against attacks on heterophily graphs. This goal of improving the robustness of networks extends to self-supervised contrastive learning methods as well. Recently, researchers argue that by using adversarial transformations during contrastive learning, a deep neural network model can achieve stateof-the-art robustness against image adversarial attacks [11,14,19]. In the graph domain, You et al. [36] include the evaluation of their contrastive learning method in the setting of adversarial attacks. However, they focus solely on graph classification, and more notably, they do not explicitly include adversarial robustness as a goal of their training.

GRAPH ROBUST CONTRASTIVE LEARNING
In this section, we introduce our method: Graph Robust Contrastive Learning (GROC) (Fig. 1). GROC builds on top of previous work in contrastive learning on graphs, aiming to improve graph neural networks' robustness against adversarial attacks.
We train in the graph contrastive learning framework [36,39,40] inspired by Chen et al. [2]. The key idea is to treat G as merely one view on the underlying input graph, not necessarily a unique one. We then define a family of identity-preserving transformations , where two such transformations, 1 , 2 ∈ map G to two new views of the same underlying graph, that is 1 and 2 do not change the fundamental structure of G and the node identities. Therefore, we expect the embeddings of the same node under 1 and 2 to be similar. At the same time, we expect the embeddings of different nodes to be dissimilar across and within two graph views.

Algorithm 1: Graph Robust Contrastive Learning (GROC)
Input: G = ( , ∈ R × , ∈ R × ) Output: Embedding matrix , for use in a downstream task 1: Initialize the parameters of an -layer GNN encoder 2: For each node , precompute ( ) and ( ), the set of nodes (resp. edges) within hops of that affect its final embedding 3: for Randomly split in / batches of size 5: for each batch do 6: Apply stochastic transformations ′ 1 , ′ 2 to G to obtain two views, masking features independently with probability 1 (resp. 2 ) 7: − ← ∈ ( ) ⊲ Candidate set for edge removal 8: For each ∈ ,^( ) ← ( ′ ∈ ( ′ )) \ ( ) , and + ← {( , ) | ∈ , ∈^( )} ⊲ Candidate set for edge insertion 9: Temporarily add + to both views with weights 1/| + |, removing them after the next step 10: For both views, apply and compute the contrastive loss as in Eq. (1), considering only the nodes in 11: Backpropagate the loss to obtain a gradient intensity value ( ) at each edge 12: Further transform the views by applying the adversarial transformations ′′ 1 and ′′ 2 , where each ′′ removes − · | − | edges from − with the minimal ( ) and adds + · | + | edges from + with the maximal ( ) 13: For both views, apply and compute the contrastive loss as in Eq. (1), considering only nodes in 14: Update by applying gradient descent to minimize the contrastive loss be the embeddings of nodes other than in both graph views and a similarity metric. We can obtain through the following optimization: where 1 ≡ ( 1 ( )) and 2 ≡ ( 2 ( )), and The above optimization is intractable to solve due to the massive search space of transformations T and a lack of an optimization algorithm. We follow the approach of Zhu et al. [39] to tackle this problem. We realize as the cosine similarity between two embeddings after being fed through a 2−layer MLP. Sampling two transformations 1 , 2 from , we can define a contrastive loss L ( , 1 , 2 ) for each node as follows: where is a temperature parameter. Finally, to derive a gradientbased update for , we aim to minimize 1 2 Zhu et al. [39] and Zhu et al. [40] follow this framework, considering the transformations that randomly remove a fraction of edges and randomly mask a fraction of node features with 0. In GRACE [39], the edges are removed uniformly. In GCA [40], the authors investigate three variants where the edge removal probability is inversely proportional to the degree-based, eigenvector-based, or PageRank-based centrality scores of the edge.

Motivation
While the previously described contrastive learning methods obtain impressive results on a wide variety of tasks despite having no access to labels, their accuracy swiftly drops under adversarial attacks, as we later demonstrate in Section 4.
Recall that the transformations used in contrastive learning aim to produce a view which is distinct from the input but is also imperceptible, i.e., the transformation should not fundamentally alter its identity, or in the case of graphs, the node identities. For most domains, various ways to define the notion of imperceptibility arise naturally, including norm perturbations and various image transformations. However, properly defining this notion for graphs is still an open challenge due to their discrete nature. Many methods resort to independent perturbations of features and edges, often simply performing random edge removal and random feature masking. In the context of adversarial defenses, we value transformations that increase the loss our optimization procedure attempts to minimize. We find the previously described choices lacking in this regard, which negatively impacts adversarial robustness. To partially alleviate this issue, we introduce several improvements to the choice of transformations 1 , 2 .

Method
We represent each ∈ as a composition = ′ • ′′ . Namely, we generate two distinct views by first applying stochastic transformations ′ 1 , ′ 2 , followed by adversarial transformations ′′ 1 , ′′ 2 . For ′ we simply employ random feature masking. For ′′ we employ two types of edge-based transformations.
First, we perform edge removal as before. However, inspired by similar methods from other domains, we use the gradient information to make a more informed choice of edges to remove. Namely, we perform one preliminary forward-backward pass after applying ′ to obtain the gradients on edges. As we are minimizing Eq. (1), we remove a subset of edges with minimal gradient values. We discuss the choice of the number of edges to remove later. Second, we introduce edge insertion, once again using the gradient information to choose the edges to insert. However, to obtain the gradients on the edges from our candidate set for insertion + , we need to include those edges in the graph with a nonzero weight. Using all absent edges as the candidate set is impractical. To solve this, we tweak the training procedure by processing the nodes in randomized batches of size , considering only the nodes in the current batch (anchors) when constructing ( ) and later Eq. (1). With this setup, we restrict + to the set of edges ( , ), where is an anchor node, and is within the -hop neighborhood of some anchor ′ ≠ , but not within the -hop neighborhood of . We temporarily insert all these edges into the graph with weights 1/| + |, and after the preliminary pass, remove them, apart from a subset of edges with maximal gradient values. Note that | + | is upper bounded by , which for small values of is a significant improvement over O ( 2 ), and additionally reduces the impact of the candidate edges on the result of the preliminary pass. Further, we hypothesize that there is an additional benefit to node batching, as this greatly reduces the number of negative examples in ( ) for each , focusing more on its representation in the other view. The GROC algorithm is illustrated in Fig. 1 and detailed in Algorithm 1.

EXPERIMENTS
We provide a preliminary evaluation of GROC on a set of transductive node classification tasks. In this setting, given a partially labeled graph, the task consists of learning to fill in the missing labels. In our self-supervised setup, we first use the features of all nodes, but notably no labels, to learn high-level representations of each node. Then, we follow the linear evaluation protocol [33], and train a simple linear classifier on the produced embeddings, using the labels of training nodes for supervision.
We report standard classification accuracy and robust accuracy, so far assuming the threat model of Nettack [41], a common targeted gray-box attack, here used in an evasion setting. We vary the perturbation budget from 1 to 5 and use a 2-layer GCN [20] as the surrogate model. The robust accuracy is reported on a set of 10 the most easily attacked nodes (the ones with the lowest surrogate margin), 10 the least easily attacked nodes, and 20 additional random nodes from the test set. In the future, we plan to investigate additional attack methods, such as RL-S2V [3] or Meta-Attack [42].
For our implementation we use PyTorch [26], heavily relying on the PyTorchGeometric [6] library. For Nettack, we use the reference implementation from the DeepRobust [21] library. We perform all experiments on a single GPU.

Networks and Datasets
As our encoder , we use a 2-layer GCN with layer sizes respectively 2 ℎ and ℎ , and activation , with the concrete values shown in Table 3. We evaluate on the following five datasets: • The standard citation network benchmarks Cora, Citeseer and Pubmed [29], where nodes and edges represent the documents and the citations between them. We follow the dataset splits of Kipf and Welling [20] based on the setup of Yang et al. [35], using 20 nodes per class for training, 500 nodes for the validation, and 1000 nodes for the test set. • AmazonPhoto [23], a segment of the Amazon co-purchase graph. Nodes represent products, and edges imply that two products are frequently purchased together. We randomly construct the training and validation sets with 10% of the nodes each; the remaining nodes constitute the test set. • WikiCS [24], a dataset of computer science Wikipedia articles with edges based on hyperlinks between them. We use all 20 dataset splits provided in the original paper and report the average results. To meet our evaluation's assumptions, we preprocess each dataset if needed to ensure that the features are binary (thresholding at 0) and the graph is undirected with no multiple edges.

Investigated Methods
We compare GROC against two other self-supervised methods, GRACE [39] and GCA [40]. We show comparisons with the degreebased variant of GCA (GCA-DE), as we did not observe significant differences compared to other variants. Furthermore, we evaluate GRACE-ADV, an extension of GRACE that, instead of removing edges randomly, uses gradient signals as in GROC. Finally, we include a fully supervised baseline GCN, where we train the entire network with the supervision from node labels.

Hyperparameters
For GCN, we match the setup of Kipf and Welling [20], using a 2layer GCN with the hidden layer size of 16. For all self-supervised baselines we follow the hyperparameter choices from GRACE and GCA to choose the GCN parameters ( ℎ and ), the training parameters ( ℎ , the initial learning rate for the Adam optimizer , and the L2 penalty parameter ), the contrastive loss temperature , as well as the feature masking rates 1 and 2 and the edge removal rates − 1 and − 2 . Notably, for GRACE-ADV we reduce − 1 and − 2 on two datasets as necessary to obtain convergence. For GROC, we use the same set of hyperparameters, again reducing − 1 and − 2 as in GRACE-ADV and additionally using significantly fewer epochs for training as it converges early. Additionally, GROC introduces three more hyperparameters: the edge insertion rates + 1 and + 2 and , the size of node batches. We experimentally find that a set of 10 anchor nodes per batch works well across all datasets. We tune the edge insertion rates separately for each dataset. In Table 3 we show all hyperparameter choices.

Results
The results of our experiments are shown in Tables 1 and 2. As a sanity check, we see that all self-supervised methods achieve standard accuracy comparable with the supervised GCN. As a second sanity check, we see that the accuracy rapidly drops for all baseline methods as we increase the perturbation budget of Nettack from 1 to 5. This confirms that they are indeed susceptible to adversarial attacks and motivates our focus on building a more robust self-supervised method.
Notice that GRACE-ADV often shows an improvement in adversarial robustness over the vanilla version of GRACE, demonstrating the efficacy of introducing adversarial transformations into the training framework, i.e., removing those edges that hurt the contrastive loss the most, instead of randomly.
Finally, we observe that the combination of edge insertion and adversarial transformations leads to the most robust method overall, GROC, which consistently boosts the robustness of the baseline methods. Note that graph attacks often rely heavily on edge insertion 1 , which explains its importance as part of the transformation    in the contrastive learning setting. GROC is the only method that learns to incorporate adversarial edge insertion into its representation, thus allowing for higher adversarial robustness.

Limitations
While GROC succeeds in improving the adversarial robustness of graph contrastive learning methods, we point out that more work is needed, as the robustness of GROC currently comes at a price. While, due to batching, one epoch carries more information (and we can thus converge in fewer epochs), the complete training procedure of GROC takes an order of magnitude longer in total time. Therefore, we identify the work on optimizing and improving the method's performance as an important direction for future work. Additionally, to make the evaluation thorough and gain more complete insights into the proposed method's effects, we plan to include more datasets, inductive classification settings, and evaluate the adversarial robustness under a larger set of adversarial attacks. We further plan to include supervised adversarial defenses [5,15] in our evaluation.

CONCLUSION
In this work, we focused on the issue of adversarial robustness of self-supervised learning methods on graphs. We suspected, and later experimentally confirmed, that the previously introduced contrastive learning methods are vulnerable to adversarial attacks. As a first step towards achieving robustness in this setting, we introduced a novel method, GROC, that enhances the generation of graph views by introducing adversarial transformations and edge insertion. We confirmed that this approach can improve the adversarial robustness of the produced representations through a preliminary set of experiments. We hope that this work will ultimately lead to more successful and more robust contrastive learning algorithms on graphs.