{'fraction_non_alphanumeric': 0.04307093612625814, 'fraction_numerical': 0.012323140118667275, 'mean_word_length': 4.857605177993528, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 2, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Self-supervised Graph Learning for Recommendation

Representation learning on user-item graph for recommendation has evolved from using single ID or interaction history to exploiting higher-order neighbors. This leads to the success of graph convolution networks (GCNs) for recommendation such as PinSage and LightGCN. Despite effectiveness, we argue that they suffer from two limitations: (1) high-degree nodes exert larger impact on the representation learning, deteriorating the recommendations of low-degree (long-tail) items; and (2) representations are vulnerable to noisy interactions, as the neighborhood aggregation scheme further enlarges the impact of observed edges. In this work, we explore self-supervised learning on user-item graph, so as to improve the accuracy and robustness of GCNs for recommendation. The idea is to supplement the classical supervised task of recommendation with an auxiliary self-supervised task, which reinforces node representation learning via self-discrimination. Specifically, we generate multiple views of a node, maximizing the agreement between different views of the same node compared to that of other nodes. We devise three operators to generate the views -- node dropout, edge dropout, and random walk -- that change the graph structure in different manners. We term this new learning paradigm as \textit{Self-supervised Graph Learning} (SGL), implementing it on the state-of-the-art model LightGCN. Through theoretical analyses, we find that SGL has the ability of automatically mining hard negatives. Empirical studies on three benchmark datasets demonstrate the effectiveness of SGL, which improves the recommendation accuracy, especially on long-tail items, and the robustness against interaction noises. Our implementations are available at \url{https://github.com/wujcan/SGL}.


INTRODUCTION
Learning high-quality user and item representations from interaction data is the theme of collaborative recommendation. Earlier work like matrix factorization (MF) [34] projects single ID of each user (or item) into an embedding vector. Some follow-on studies [20,25] enrich the single ID with interaction history for learning better representations. More recently, representation learning has evolved to exploiting higher-order connectivity in user-item graph. The technique is inspired from the graph convolution networks (GCNs), which provide an end-to-end way to integrate multi-hop neighbors into node representation learning and achieve state-of-the-art performance for recommendation [19,38,46,50].
Despite effectiveness, current GCN-based recommender models suffer from some limitations: • Sparse Supervision Signal. Most models approach the recommendation task under a supervised learning paradigm [19,21,34], where the supervision signal comes from the observed user-item interactions. However, the observed interactions are extremely sparse [3,18] compared to the whole interaction space, making it insufficient to learn quality representations. • Skewed Data Distribution. Observed interactions usually exhibit a power-law distribution [9,30], where the long tail consists of low-degree items that lack supervision signal. In contrast, high-degree items appear more frequently in neighborhood aggregation and supervised loss, thus exert larger impact on representation learning. Hence, the GCNs are easily biased towards high-degree items [5,36], sacrificing the performance of low-degree (long-tail) items. • Noises in Interactions. Most feedback that a user provides is implicit (e.g., clicks, views), instead of explicit (e.g., ratings, likes/dislikes). As such, observed interactions usually contain noises, e.g., a user is misled to click an item and finds it uninteresting after consuming it [44]. The neighborhood aggregation scheme in GCNs enlarges the impact of interactions on representation learning, making the learning more vulnerable to interaction noises. In this work, we focus on exploring self-supervised learning (SSL) in recommendation, to solve the foregoing limitations. Though being prevalent in computer vision (CV) [11,40] and natural language processing (NLP) [10,26], SSL is relatively less explored in recommendation. The idea is to set an auxiliary task that distills additional signal from the input data itself, especially through exploiting the unlabeled data space. For example, BERT [10] randomly masks some tokens in a sentence, setting the prediction of the masked tokens as the auxiliary task that can capture the dependencies among tokens; RotNet [11] randomly rotates labeled images, training the model on the rotated images to get improved representations for the mask task of object recognition or image classification. Compared with supervised learning, SSL allows us to exploit the unlabeled data space via making changes on the input data, achieving remarkable improvements in downstream tasks [6].
Here we wish to bring the SSL's superiority into recommendation representation learning, which differs from CV/NLP tasks since the data are discrete and inter-connected. To address the aforementioned limitations of GCN-based recommendation models, we construct the auxiliary task as discriminating the representation of a node itself. Specifically, it consists of two key components: (1) data augmentation, which generates multiple views for each node, and (2) contrastive learning, which maximizes the agreement between different views of the same node, compared to that of other nodes. For GCNs on user-item graph, the graph structure serves as the input data that plays an essential role for representation learning. From this view, it is natural to construct the "unlabeled" data space by changing the graph adjacency matrix, and we develop three operators to this end: node dropout, edge dropout, and random walk, where each operator works with a different rationality. Thereafter, we perform contrastive learning based on the GCNs on the changed structure. As a result, SGL augments the node representation learning by exploring the internal relationship among nodes.
Conceptually, our SGL supplements existing GCN-based recommendation models in: (1) node self-discrimination offers auxiliary supervision signal, which is complementary to the classical supervisions from observed interactions only; (2) the augmentation operators, especially edge dropout, helps to mitigate the degree biases by intentionally reducing the influence of highdegree nodes; (3) the multiple views for nodes w.r.t. different local structures and neighborhoods enhance the model robustness against interaction noises. Last but not least, we offer theoretical analyses for the contrastive learning paradigm, finding that it has the side effect of mining hard negative examples, which not only boosts the performance but also accelerates the training process.
It is worthwhile mentioning that our SGL is model-agnostic and can be applied to any graph-based model that consists of user and/or item embedding. Here we implement it on the simple but effective model, LightGCN [19]. Experimental studies on three benchmark datasets demonstrate the effectiveness of SGL, which significantly improves the recommendation accuracy, especially on long-tail items, and enhance the robustness against interaction noises. We summarize the contributions of this work as follows: • We devise a new learning paradigm, SGL, which takes node selfdiscrimination as the self-supervised task to offer auxiliary signal for representation learning. • In addition to mitigating degree bias and increasing robustness to interaction noises, we prove in theory that SGL inherently encourages learning from hard negatives, controlled by the temperature hyper-parameter in the softmax loss function. • We conduct extensive experiments on three benchmark datasets to demonstrate the superiority of SGL.

PRELIMINARIES
We first summarize the common paradigm of GCN-based collaborative filtering models. Let U and I be the set of users and items respectively. Let O + = { | ∈ U, ∈ I} be the observed interactions, where indicates that user has adopted item before. Most existing models [19,38,46] construct a bipartite graph G = (V, E), where the node set V = U ∪ I involves all users and items, and the edge set E = O + represents observed interactions. Recap GCN. At the core is to apply the neighborhood aggregation scheme on G, updating the representation of ego node by aggregating the representations of neighbor nodes: where Z ( ) denotes the node representations at the -th layer, Z ( −1) is that of previous layer, and Z (0) is the ID embeddings (trainable parameters). denotes the function for neighborhood aggregation, which is more interpretable from the vector level: To update the representation of ego node at the -th layer, it first aggregates the representations of its neighbors N at the ( − 1)-th layer, then combines with its own representation z ( −1) . There are a number of designs for aggregate (·) and combine (·) [12,15,41,48].
We can see that the representations of the -th layer encode the -order neighbors in the graph. After obtaining layers representations, there may be a readout function to generate the final representations for prediction: Common designs include last-layer only [38,50], concatenation [46], and weighted sum [19]. Supervised Learning Loss. A prediction layer is built upon the final representations to predict how likely would adopt . A classical solution is the inner product, which supports fast retrieval: To optimize model parameters, existing works usually frame the task as one of supervised learning, where the supervision signal comes from the observed interactions (also the edges of G). For example, encouraging the predicted valueˆto be close to the ground truth value and selecting negative examples from missing data [21]. Besides above point-wise learning, another common choice is the pairwise Bayesian Personalized Ranking (BPR) loss [34], which enforces the prediction of an observed interaction to be scored higher than its unobserved counterparts:

METHODOLOGY
We present the proposed Self-supervised Graph Learning (SGL) paradigm, which supercharges the main supervised task with self-supervised learning. Figure 1 illustrates the working flow of SGL. Specifically, the self-supervised task (also termed as pretext task or auxiliary task) is to construct supervision signal from the correlation within the input data. Specifically, we introduce how to perform data augmentation that generates multiple representation views, followed by the contrastive learning based on the generated representations to build the pretext task. SSL is combined with classical GCN in a multi-task learning manner. Thereafter, we conduct theoretical analyses on SSL from the gradient level, revealing the connection with hard negative mining. Lastly, we analyze the complexity of SGL.

Data Augmentation on Graph Structure
Directly grafting the data augmentation adopted in CV and NLP tasks [6,10,17,47] is infeasible for graph-based recommendation, due to specific characteristics: (1) The features of users and items are discrete, like one-hot ID and other categorical variables. Hence, the augmentation operators on images, such as random crop, rotation, or blur, are not applicable. (2) More importantly, unlike CV and NLP tasks that treat each data instance as isolated, users and items in the interaction graph are inherently connected and dependent on each others. Thus, we need new augmentation operators tailored for graph-based recommendation.
The bipartite graph is built upon observed user-item interactions, thus containing the collaborative filtering signal. Specifically, the first-hop neighborhood directly profiles ego user and item nodesi.e., historical items of a user (or interacted users of an item) can be viewed as the pre-existing features of user (or item). The secondhop neighboring nodes of a user (or an item) exhibit similar users w.r.t. behaviors (or similar items w.r.t. audiences). Furthermore, the higher-order paths from a user to an item reflect potential interests of the user on the item. Undoubtedly, mining the inherent patterns in graph structure is helpful to representation learning. We hence devise three operators on the graph structure, node dropout, edge dropout and random walk, to create different views of nodes. The operators can be uniformly expressed as follows: and Random Walk (right). For Random Walk, the graph structure keeps changing across layers as opposed to Edge Dropout. As a result, there exists a three-order path between node 1 and 1 that does not exist in Edge Dropout.
where two stochastic selections 1 and 2 are independently applied on graph G, and establish two correlated views of nodes Z ( ) 1 and Z ( ) 2 . We elaborate the augmentation operators as follows: • Node Dropout (ND). With the probability , each node is discarded from the graph, together with its connected edges.
In particular, 1 and 2 can be modeled as: where M ′ , M ′′ ∈ {0, 1} |V | are two masking vectors which are applied on the node set V to generate two subgraphs. As such, this augmentation is expected to identify the influential nodes from differently augmented views, and make the representation learning less sensitive to structure changes. • Edge Dropout (ED). It drops out the edges in graph with a dropout ratio . Two independent processes are represented as: where M 1 , M 2 ∈ {0, 1} | E | are two masking vectors on the edge set E. Only partial connections within the neighborhood contribute to the node representations. As such, coupling these two subgraphs together aims to capture the useful patterns of the local structures of a node, and further endows the representations more robustness against noisy interactions. • Random Walk (RW). The above two operators generate a subgraph shared across all the graph convolution layers. To explore higher capability, we consider assigning different layers with different subgraphs. This can be seen as constructing an individual subgraph for each node with random walk [31] (see Figure 2 as an example). Assuming we choose edge dropout at each layer (with different ratio or random seeds), we can formulate RW by making the masking vector to be layer sensitive: where M ( ) 1} | E | are two masking vectors on the edge set E at layer .
We apply these augmentations on graph structure per epoch for simplicity -that is, we generate two different views of each node at the beginning of a new training epoch (for RW, two different views are generated at each layer). Note that the dropout and masking ratios remain the same for two independent processes (i.e., 1 and worthwhile mentioning that only dropout and masking operations are involved, and no any model parameters are added.

Contrastive Learning
Having established the augmented views of nodes, we treat the views of the same node as the positive pairs (i.e., {(z ′ , z ′′ )| ∈ U}), and the views of any different nodes as the negative pairs (i.e., {(z ′ , z ′′ )| , ∈ U, ≠ }). The auxiliary supervision of positive pairs encourages the consistency between different views of the same node for prediction, while the supervision of negative pairs enforces the divergence among different nodes. Formally, we follow SimCLR [6] and adopt the contrastive loss, InfoNCE [14], to maximize the agreement of positive pairs and minimize that of negative pairs: where (·) measures the similarity between two vectors, which is set as cosine similarity function; is the hyper-parameter, known as the in softmax. Analogously, we obtain the contrastive loss of the item side L . Combining these two losses, we get the objective function of self-supervised task as L = L + L .

Multi-task Training
To improve recommendation with the SSL task, we leverage a multi-task training strategy to jointly optimize the classic recommendation task (cf. Equation (5)) and the self-supervised learning task (cf. Equation (10)) where Θ is the set of model parameters in since introduces no additional parameters; 1 and 2 are hyperparameters to control the strengths of SSL and 2 regularization, respectively. We also consider the alternative optimization -pre-training on L and fine-tuning on L . See more details in Section 4.4.2.

Theoretical Analyses of SGL
In this section, we offer in-depth analyses of SGL, aiming to answer the question: how does the recommender model benefit from the SSL task? Towards this end, we probe the self-supervised loss in Equation (10) and find one reason: it has the intrinsic ability to perform hard negative mining, which contributes large and meaningful gradients to the optimization and guides the node representation learning. In what follows, we present our analyses step by step. Formally, for node ∈ U, the gradient of the self-supervised loss w.r.t. the representation ′ is as follows: where L ( ) is the individual term for a single node in Equation (10); ∈ U \ { } is another node which serves as the negative view for node ; ( ) and ( ) separately represent the contribution of positive node and negative nodes { } to the Figure 3: Function curve of ( ) when = 1 and = 0.1, together with the logarithm of the maximum value of ( ) w.r.t. and its optimal position, i.e., ln ( * ) and * ( ).
gradients w.r.t. ′ : where ; ′ = ′ ∥ ′ ∥ and ′′ = ′′ ∥ ′′ ∥ are the normalized representations of node from different views; similar notations to node . Afterwards, we focus on the contribution of negative node (cf. Equation (14)), the 2 norm of which is proportional to the following term: As ′ and ′′ are both unit vectors, we can introduce another variable = ′ ′′ ∈ [−1, 1] to simplify Equation (15) as follows: where directly reflects the representation similarity between the positive node and the negative node . According to the similarity , we can roughly categorize the negative nodes into two groups: (1) Hard negative nodes, whose representations are similar to that of the positive node (i.e., 0 < ≤ 1), thus making it difficult to distinguish from in the latent space; (2) Easy negative nodes, which are dissimilar to the positive node (i.e., −1 ≤ < 0) and can be easily discriminated. To investigate the contributions of hard and easy negatives, we plot the curves of ( ) over the change of the node similarity in Figures 3a and 3b, by setting = 1 and = 0.1 respectively. Clearly, under different conditions of , the contributions of negative nodes differ dramatically with each others. Specifically, as Figure 3a shows, given = 1, the values of ( ) fall into the range of (0, 1.5) and slightly change in response to . This suggests that negative samples, no matter hard or easy, contribute similarly to the gradient. In contrast, as Figure 3b displays, when setting = 0.1, the values of ( ) at hard negatives could reach 4, 000, while the contribution of easy negatives is vanishing. This indicates that hard negative nodes offer much larger gradients to guide the optimization, thus making node representations more discriminative and accelerating the training process [33].
These findings inspire us to probe the influence of on the maximum value of ( ). By approaching ( )'s derivative to zero, we can obtain * = ( √ 2 + 4 − )/2 with maximum value ( * ). To see how ( * ) changes with , we represent its logarithm as: We present the curves of * and ln ( * ) in Figures 3c and  3d, respectively. With the decrease of , the most influential negative nodes become more similar to the positive node (i.e., * approaches 0.9), moreover, their contributions are amplified super-exponentially (i.e., * is close to 8 ). Hence, properly setting enables SGL to automatically perform hard negative mining. It is worth mentioning that, our analyses are inspired by the prior study [24], but there are major differences: (1) [24] defines hard negatives as samples whose similarity are near-to-zero to the positive sample (i.e., ≈ 0), and easy negatives as obviously dissimilar samples (i.e., ≈ −1); (2) It leaves the region of > 0 untouched, which is however crucial in our case. Innovatively, we provide a more fine-grained view in the region of > 0 to highlight the critical role of , the temperature hyper-parameter in softmax, in mining hard negatives.

Complexity Analyses of SGL
In this subsection, we analyze the complexity of SGL with ED as the strategy and LightGCN as the recommendation model; other choices can be analyzed similarly. Since SGL introduces no trainable parameters, the space complexity remains the same as LightGCN [19]. The time complexity of model inference is also the same, since there is no change on the model structure. In the following part, we will analyze the time complexity of SGL training.
Suppose the number of nodes and edges in the user-item interaction graph are | | and | | respectively. Let denote the number of epochs, denote the size of each training batch, denote the embedding size, denote the number of GCN layers,ˆ= 1 − denote the keep probability of SGL-ED. The complexity mainly comes from two parts: • Normalization of adjacency matrix. Since we generate two independent sub-graphs per epoch, given the fact that the number of non-zero elements in the adjacency matrices of full training graph and two sub-graph are 2 | | , 2ˆ| | and 2ˆ| | respectively, its total complexity is (4ˆ| | + 2 | |). • Evaluating self-supervised loss. We only consider the inner product in our analyses. As defined in Equation (10), we treat all other user nodes as negative samples when calculating InfoNCE loss of user side. Within a batch, the complexity of numerator and denominator are ( ) and ( ), respectively, where is the number of users. And hence the total complexity of both user and item side per epoch is (| | (2 + | |)). Therefore, the time complexity of the whole training phase is (| | (2 + | |) ). An alternative to reduce the time complexity is treating only the users (or the items) within the batch as negative samples [6,49], resulting in total time complexity of (| | (2 + 2 ) ). We summarize the time complexity in training between LightGCN and SGL-ED in Table 1. The analytical complexity of LightGCN

EXPERIMENTS
To justify the superiority of SGL and reveal the reasons of its effectiveness, we conduct extensive experiments and answer the following research questions: • RQ1: How does SGL perform w.r.t. top-recommendation as compared with the state-of-the-art CF models? • RQ2: What are the benefits of performing self-supervised learning in collaborative filtering? • RQ3: How do different settings influence the effectiveness of the proposed SGL?

Experimental Settings
We conduct experiments on three benchmark datasets: Yelp2018 [19,46], Amazon-Book [19,46], and Alibaba-iFashion [8] 1 . Following [19,46], we use the same 10-core setting for Yelp2018 and Amazon-Book. Alibaba-iFashion is more sparse, where we randomly sample 300k users and use all their interactions over the fashion outfits. The statistics of all three datasets are summarized in Table 2. We follow the same strategy described in [46] to split the interactions into training, validation, and testing set with a ratio of 7:1:2. For users in the testing set, we follow the all-ranking protocol [46] to evaluate the top-recommendation performance and report the average Recall@ and NDCG@ where = 20.

Compared Methods.
We compare the proposed SGL with the following CF models: • NGCF [46]. This is a graph-based CF method largely follows the standard GCN [12]. It additionally encodes the second-order feature interaction into the message during message passing. We tune the regularization coefficient 2 and the number of GCN layers within the suggested ranges. • LightGCN [19]. This method devises a light graph convolution for training efficiency and generation ability. Similarly, we tune the 2 and the number of GCN layers. • Mult-VAE [28]. This is an item-based CF method based on the variational auto-encoder (VAE). It is optimized with an additional reconstruction objective, which can be seen as a special case of SSL. We follow the suggested model setting and tune the dropout ratio and . • DNN+SSL [49]. This is a state-of-the-art SSL-based recommendation method. With DNNs as the encoder of items, it adopts two augmentation operators, feature masking (FM) and feature dropout (FD), on the pre-existing features of items. In our cases where no item feature is available, we apply the augmentations on ID embeddings of items instead. We tune the DNN architecture (i.e., the number of layers and the number of neurons per layer) as suggested in the original paper.
We discard potential baselines like MF [34], NeuMF [21], GC-MC [38], and PinSage [50] since the previous work [19,28,46] has validated the superiority over the compared ones. Upon LightGCN, we implement three variants of the proposed SGL, named SGL-ND, SGL-ED, and SGL-RW, which are equipped with Node Dropout, Edge Dropout, and Random Walk, respectively.  Table 3 shows the result comparison between SGL and LightGCN. We find that:

Comparison with LightGCN.
• In most cases, three SGL implementations outperform LightGCN by a large margin, indicating the superiority of supplementing the recommendation task with self-supervised learning. • In SGL family, SGL-ED achieves best performance in 10 over 18 cases, while SGL-RW also performs better than SGL-ND across all three datasets. We ascribe these to the ability of edge dropoutlike operators to capture inherent patterns in graph structure. Moreover, the performance of SGL-ED is better than that of SGL-RW in the denser datasets (Yelp2018 and Amazon-Book), while slightly worse in the sparser dataset (Alibaba-iFashion).
One possible reason is that, in sparser datasets, ED is more likely to blocks connections of low-degree nodes (inactive users and unpopular items), while RW could restore their connections at different layers, as the case of node 1 and 1 shown in Figure 2. • SGL-ND is relatively unstable than SGL-ED and SGL-RW. For example, on Yelp2018 and Amazon-Book, the results of SGL-ED and SGL-RW increases when layers go deeper while SGL-ND exhibits different patterns. Node Dropout can be viewed as a special case of Edge Dropout, which discards edges around a few nodes. Hence, dropping high-degree nodes will dramatically change the graph structure, thus exerts influence on the information aggregation and makes the training unstable. • The improvements on Amazon-Book and Alibaba-iFashion are more significant than that on Yelp2018. This might be caused by the characteristics of datasets. Specifically, in Amazon-Book and Alibaba-iFashion, supervision signal from user-item interactions is too sparse to guide the representation learning in LightGCN.
Benefiting from the self-supervised task, SGL obtains auxiliary supervisions to assist the representation learning. • Increasing the model depth from 1 to 3 is able to enhance the performance of SGL. This indicates that exploiting SSL could empower the generalization ability of GCN-based recommender models -that is, the contrastive learning among different nodes is of promise to solving the oversmoothing issue of node representations, further avoiding the overfitting problem.   Table 4, we summarize the performance comparison with various baselines. We find that: (1) SGL-ED consistently outperforms all baselines across the board. This again verifies the rationality and effectiveness of incorporating the self-supervised learning. (2) LightGCN achieves better performance than NGCF and Mult-VAE, which is consistent to the claim of LightGCN paper. The performance of Mult-VAE is on par with NGCF and LightGCN on Alibaba-iFashion, while outperforms NGCF on Amazon-Book. (3) DNN+SSL is the strongest baseline on Amazon-Book, showing the great potential of SSL in recommendation. Surprisingly, on the other datasets, DNN+SSL performs much worse than SGL-ED. This suggests that directly applying SSL on ID embeddings might be suboptimal and inferior than that on graph representations. (4) Moreover, we conduct the significant test, where -value < 0.05 indicates that the improvements of SGL-ED over the strongest baseline are statistically significant in all six cases.

Benefits of SGL (RQ2)
In this section, we study the benefits of SGL from three dimensions: (1) long-tail recommendation; (2) training efficiency; and (3) robustness to noises. Due to the limited space, we only report the results of SGL-ED, while having similar observations in others.

Long-tail Recommendation.
As Introduction mentions, GNN-based recommender models easily suffer from the long-tail problem. To verify whether SGL is of promise to solving the problem, we split items into ten groups based on the popularity, meanwhile keeping the total number of interactions of each group the same. The larger the GroupID is, the larger degrees the items have. We then decompose the Recall@20 metric of the whole dataset into contributions of single groups, as follows: where is the number of users, and are the items in the top-recommendation list and relevant items in the testing set for user , respectively. As such, ( ) measures the recommendation performance over the -th group. We report the results in Figure 4 and find that: • LightGCN is inclined to recommend high-degree items, while leaving long-tail items less exposed. Specifically, although only containing 0.83%, 0.83% and 0.22% of item spaces, the 10-th group contributes 39.72%, 39.92% and 51.92% of the total Recall scores in three datasets, respectively. This admits that, LightGCN hardly learns high-quality representations of long-tail items, due to the sparse interaction signal. Our SGL shows potentials in alleviating this issue: the contributions of the 10-group downgrade to 36.27%, 29.15% and 35.07% in three datasets, respectively. • Jointly analyzing Table. 3 and Figure 4, we find the performance improvements of SGL mainly come from accurately recommending the items with sparse interactions. This again verifies that the representation learning benefits greatly from auxiliary supervisions, so as to establish better representations of these items than LightGCN. 4.3.2 Training Efficiency. Self-supervised learning has proved its superiority in pre-training natural language model [10] and graph structure [23,31]. Thus, we would like to study its influence on training efficiency. Figure 5 shows the training curves of SGL-ED and LightGCN on Yelp2018 and Amazon-Book. As the number of epochs increases, the upper subfigures display the changes of training loss, while the bottoms record the performance changes in the testing set. We have the following observations: • Obviously, SGL is much faster to converge than LightGCN on Yelp2018 and Amazon-Book. In particular, SGL arrives at the best performance at the 18-th and 16-th epochs, while LightGCN takes 720 and 700 epochs in these two datasets respectively. This suggests that our SGL can greatly reduce the training time, meanwhile achieves remarkable improvements. We ascribe such speedups to two factors: (1) SGL adopts the InfoNCE loss as the SSL objective which enables the model to learn representations from multiple negative samples, while the BPR loss in LightGCN only uses one negative sample which limits the model's perception field; (2) As analyzed in Section 3.4, with a proper , SGL benefits from dynamic hard negative mining, where the hard negative samples offer meaningful and larger gradients to guide the optimization [54]. • Another observation is that the origin of the rapid-decline period of BPR loss is slightly later than the rapid-rising period of Recall. This indicates the existence of a gap between the BPR loss and ranking task. We will conduct in-depth research on this phenomenon in our future work.

Robustness to Noisy
Interactions. We also conduct experiments to check SGL's robustness to noisy interactions. Towards this end, we contaminate the training set by adding a certain proportion of adversarial examples (i.e., 5%, 10%, 15%, 20% negative user-item interactions), while keeping the testing set unchanged. Figure 6 shows the results on Yelp2018 and Amazon-Book datasets.
• Clearly, adding noise data reduces the performance of SGL and LightGCN. However, the performance degradation of SGL is lower than that of LightGCN; moreover, as the noise ratio increases, the gap between two degradation curves becomes more apparent. This suggests that, by comparing differently augmented views of nodes, SGL is able to figure out useful patterns, especially informative graph structures of nodes, and reduce dependence on certain edges. In a nutshell, SGL offers a different angle to denoise false positive interactions in recommendation. • Focusing on Amazon-Book, the performance of SGL with 20% additional noisy interactions is still superior to LightGCN with noise-free dataset. This further justifies the superiority and robustness of SGL over LightGCN.  • We find that SGL is more robust on Yelp2018. The possible reason may be that Amazon-Book is much sparser than Yelp2018, and adding noisy data exerts more influence on graph structure of Amazon-Book than that of Yelp2018.

Study of SGL (RQ3)
We move on to studying different designs in SGL. We first investigate the impact of hyper-parameter . We then explore the potential of adopting SGL as a pretraining for existing graph-based recommendation model. Lastly, we study the influence of negatives in the SSL objective function. Due to the space limitation, we omit the results on iFashion which have a similar trend to that on Yelp2018 and Amazon-Book.
4.4.1 Effect of Temperature . As verified in Section 3.4, play a critical role in hard negative mining. Figure 7 shows the curves of model performance w.r.t. different . We can observe that: (1) Increasing the value of (e.g., 1.0) will lead to poorer performance and take more training epochs to converge, which fall short in the ability to distinguish hard negatives from easy negatives. (2) In contrast, fixing to a too small value (e.g., 0.1) will hurt the model performance, since the gradients of a few negatives dominate the optimization, losing the supremacy of adding multiple negative samples in the SSL objective. In a nutshell, we suggest to tun in the range of 0.1 ∼ 1.0 carefully.

Effect of Pre-training.
The foregoing experiments have shown the effectiveness of SGL, where the main supervised task and the self-supervised task are jointly optimized. Here we would like to answer the question: Can the recommendation performance benefit from the pre-trained model? Towards this goal, we first pre-train the self-supervised task to obtain the model parameters, use them to initialize LightGCN, and then fine-tune the model via optimizing the main task. We term this variant as SGL-pre and show the comparison with SGL in Table 5. Clearly, although SGLpre performs worse than SGL-ED on both datasets, the results of SGL-pre are still better than that of LightGCN (cf. Table 3). Our selfsupervised task is able to offer a better initialization for LightGCN, which is consistent to the observation in previous studies [6]. However, the better performance of joint training admits that the representations in the main and auxiliary tasks are mutually enhanced with each other.

4.4.3
Effect of Negatives. Moreover, we also study the effect of the choice of negative samples in the auxiliary task. Two variants are considered: (1) SGL-ED-batch, which differentiates node types and treat users and items in a mini-batch as negative views for users and items, separately, and (2) SGL-ED-merge, which treat nodes within a mini-batch as negative, without differentiating the node types. We report the comparison in Table 5. SGL-ED-batch performs better than SGL-ED-merge, which indicates the necessity for distinguishing types of heterogeneous nodes. Moreover, SGL-ED-batch is on a par with SGL-ED that treats the whole spaces of users and items as negative. It suggests that training the SSL task in mini-batching is an efficient alternative.

RELATED WORK
In this section, we separately review two tasks related to our work: graph-based recommendation and self-supervised learning.

Graph-based Recommendation
Previous studies on graph-based recommendation can be categorized into: model-level and graph-level methods, regarding their focus. The model-level methods focus on model design for mining the useritem graph. The research attention has evolved from the random walk that encodes the graph structure as transition probabilities [2], to GCN that propagates user and item embeddings over the graph [19,38,46,50]. Recently, attention mechanism is introduced into GCN-based recommendation models [7], which learns to weigh the neighbors so as to capture the more informative user-item interactions. A surge of attention has also been dedicated to graphlevel methods, which enriches the user-item graph by accounting for side information apart from user-item interactions, which ranges from user social relations [1,32,53], item co-occurrence [2], to user and item attributes [27]. Recently, Knowledge Graph (KG) is also unified with the user-item graph, which enables considering the detailed types of linkage between items [4,43,45].
Despite the tremendous efforts devoted on these methods, all the existing work adopts the paradigm of supervised learning for model training. This work is in an orthogonal direction, which explores self-supervised learning, opening up a new research line of graph-based recommendation.

Self-supervised Learning
Studies on self-supervised learning can be roughly categorized into two branches: generative models [10,29,39] and contrastive models [6,11,17,40]. Auto-encoding is the most popular generative model which learns to reconstruct the input data, where noises can be intentionally added to enhance model robustness [10]. Contrastive models learn to compare through a Noise Contrastive Estimation (NCE) objective, which can be in either global-local contrast [22,40] or global-global contrast manner [6,17]. The former focuses on modeling the relationship between the local part of a sample and its global context representation. While the latter directly performs comparison between different samples, which typically requires multiple views of samples [6,17,37]. SSL has also been applied on graph data [51,52,56]. For instance, InfoGraph [35] and DGI [42] learns node representations according to mutual information between a node and the local structure. In addition, Hu et al. [23] extend the idea to learn GCN for graph representation. Furthermore, Kaveh et al. [16] adopt the contrastive model for learning both node and graph representation, which contrasts node representations from one view with graph representation of another view. Besides, GCC [31] leverages instance discrimination as the pretext task for graph structural information pre-training. These studies focused on general graphs while leaving the inherent properties of bipartite graph unconsidered.
To the best of our knowledge, very limited works exist in combining SSL with recommendation to date. A very recent one is 3 -Rec [55] for sequential recommendation which utilizes the mutual information maximization principle to learn the correlations among attribute, item, subsequence, and sequence. Another attempt [49] also adopts a multi-task framework with SSL. However, it differs from our work in: (1) [49] uses two-tower DNN as encoder, while our work sheds lights on graph-based recommendation, and devised three augmentation operators on graph structure. (2) [49] utilizes categorical metadata features as model input, while our work considers a more general collaborative filtering setting with only ID as feature.

CONCLUSION AND FUTURE WORK
In this work, we recognized the limitations of graph-based recommendation under general supervised learning paradigm and explored the potential of SSL to solve the limitations. In particular, we proposed a model-agnostic framework SGL to supplement the supervised recommendation task with self-supervised learning on user-item graph. From the perspective of graph structure, we devised three types of data augmentation from different aspects to construct the auxiliary contrastive task. We conducted extensive experiments on three benchmark datasets, justifying the advantages of our proposal regarding long-tail recommendation, training convergence and robustness against noisy interactions.
This work represents an initial attempt to exploit selfsupervised learning for recommendation and opens up new research possibilities. In future work, we would like to make SSL more entrenched with recommendation tasks. Going beyond the stochastic selections on graph structure, we plan to explore new perspectives, such as counterfactual learning to identify influential data points, to create more powerful data augmentations. Moreover, we will focus on pre-training and fine-tuning in recommendation, -that is, to pre-train a model which captures universal and transferable patterns of users across multiple domains or datasets, and fine-tune it on upcoming domains or datasets. Another promising direction is fulfilling the potential of SSL to address the long-tail issue. We hope the development of SGL is beneficial for improving the generalization and transferability of recommender models.

A GRADIENT OF INFONCE LOSS W.R.T. NODE REPRESENTATION
Here we derive the Equation (12). Since we adopt cosine similarity function in the contrastive loss, we here decouple the normalization to show its superiority inspired from [24]. For a user ∈ U, the contrastive loss is as follows: