{'fraction_non_alphanumeric': 0.030557143764690934, 'fraction_numerical': 0.007432818753573471, 'mean_word_length': 5.373279352226721, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
GRecX: An Efficient and Unified Benchmark for GNN-based Recommendation

In this paper, we present GRecX, an open-source TensorFlow framework for benchmarking GNN-based recommendation models in an efficient and unified way. GRecX consists of core libraries for building GNN-based recommendation benchmarks, as well as the implementations of popular GNN-based recommendation models. The core libraries provide essential components for building efficient and unified benchmarks, including FastMetrics (efficient metrics computation libraries), VectorSearch (efficient similarity search libraries for dense vectors), BatchEval (efficient mini-batch evaluation libraries), and DataManager (unified dataset management libraries). Especially, to provide a unified benchmark for the fair comparison of different complex GNN-based recommendation models, we design a new metric GRMF-X and integrate it into the FastMetrics component. Based on a TensorFlow GNN library tf_geometric, GRecX carefully implements a variety of popular GNN-based recommendation models. We carefully implement these baseline models to reproduce the performance reported in the literature, and our implementations are usually more efficient and friendly. In conclusion, GRecX enables uses to train and benchmark GNN-based recommendation baselines in an efficient and unified way. We conduct experiments with GRecX, and the experimental results show that GRecX allows us to train and benchmark GNN-based recommendation baselines in an efficient and unified way. The source code of GRecX is available at https://github.com/maenzhier/GRecX.


INTRODUCTION
Personalized recommendation is an important yet challenging task, which has attracted substantial attention in the past decade. Most traditional approaches consider recommendation as a matching task [16], and can be solved by estimating the matching score based upon semantic representations of users and items [5][6][7]. Recently, graph representation learning approaches are emerging tools to pursue a meaningful vector representation for each node in graphs, which can effectively model users, items and their corresponding relationships. Graph Neural Networks (GNNs), such as GCN [10], GraphSAGE [3] and GAT [13], have shown impressive performance in aggregating feature information of neighboring nodes. In recommender systems, the interactions between users and items can be represented as a bipartite graph and the goal is to predict new potential edges (i.e., which items could a user be interested in), which 1 https://github.com/CrawlScript/tf_geometric can be achieved with GNNs, which are called GNN-based recommendation methods. GNN-based recommendation techniques has attracted researchers and engineers from a variety of fields, and they have been utilized to build various real-world applications such as medicine recommendation [8], micro-video recommendation [2,15], and social recommendation [1,17,18]. Although many existing approaches provide official implementations, it is still difficult to build efficient and unified benchmarks due to the following limitations: (1) The evaluation of GNN-based recommendation approaches may involve many computationally expensive operations, which should be optimized to perform efficient benchmarking. However, most official implementations usually ignore the problem, and some of the computationally expensive operations are still widely used by these implementations. For example, these implementations usually rely on the matrix multiplication between the user representation matrix and the item representation matrix to perform the topK retrieval of items for users, which may result in large time and space complexity. (2) It is difficult to build a unified benchmark since different baselines may adopt different loss functions, different negative sampling strategies, etc. For example, UltraGCN [12] adopts NGCF [14] and LightGCN [4] as baselines. However, the negative sampling strategy of Ultra-GCN [12] is different from that of NGCF [14] and LightGCN [4]. As shown in Table 1, NGCF and LightGCN adopt a negative sampling strategy that uses only one negative sample, while negative sampling strategy of UltraGCN , which represents the standard version of UltraGCN, samples more than 500 negative samples. It is well known that GNN-based recommendation models can be improved by simply increasing the number of negative samples [8].
As a result, the experimental results reported by UltraGCN [12] cannot verify the effectiveness of the model. Note that we implement the UltraGCN model as a baseline. In this paper, we present GRecX, an open-source TensorFlow framework for benchmarking GNN-based recommendation models in an efficient and unified way. To address the efficiency problem, we develop core libraries to provide essential components for build efficient benchmarks, including FastMetrics (efficient metrics computation libraries), VectorSearch (efficient similarity search libraries for dense vectors), BatchEval (efficient mini-batch evaluation libraries), and DataManager (unified dataset management libraries). To build a unified benchmark for the fair comparison of different complex GNN-based recommendation models, we design a new metric named GRMF-X and integrate it into the FastMetrics component. In addition, we also provide efficient implementations of a variety of popular GNN-based recommendation models, which enable us to build a more comprehensive benchmark. We conduct experiments with GRecX, and the experimental results show that GRecX allows us to train and benchmark GNN-based recommendation baselines in an efficient and unified way. All features of GRecX and a collection of examples are provided with the source code, which is available at https://github.com/maenzhier/GRecX. Figure 1 shows the overall framework of GRecX, which mainly consists of core libraries and implementations of popuplar GNN-based recommendation models. In this section, we provide an overview of the framework of GRecX.

Core Libraries
The core libraries provide essential components including FastMetrics (efficient metrics computation libraries), VectorSearch (efficient similarity search libraries for dense vectors), BatchEval (efficient mini-batch evaluation libraries), and DataManager (unified dataset management libraries). There components enable us to build efficient and unified benchmarks. In this section, we will introduce each component of GRecX's core libraries in detail.

FastMetrics.
FastMetrics provides efficient implementations for various widely-used recommendation metrics, such as NDCG@N, Precision, and Recall. It is non-trivial to implement efficient metrics computation libraries for recommendation due to the complexity of real-world data. Moreover, for fair comparison, we design a new metric named GRMF-X and integrate it into FastMetrics. GRMF-X is the abbreviation for Gain Relative to MF in terms of the metrics X, and it is defined as follows: is the evaluated score of model under the metric and context , and denotes a tuned Matrix Factorization (MF) [11] model. We introduce this naive evaluation metric since it can effectively help us to verify the superiority of GNN-based recommendation models.
There are mainly two reasons why our unified benchmark can benefit from GRMF-X: (1) We observe that although some research's experimental results show that their proposed model outperforms all the baselines, the reported performance of the baselines or the proposed models may not be competitive with the simple welltuned MF model. Note that although some research provide the performance of MF, they may employ a MF model that is not well tuned, which usually show poor performance. This shows that the authors do not conduct experiments with well-implemented baselines, and thus the experimental results are not convincing. (2) We also observe that some research do not conduct experiments of different baselines under the same context. Here we use context to denote the some important settings beyond the GNN architectures, such as the negative sampling strategies. For example, as shown in Table 1, NGCF and LightGCN employ a negative sampling strategy that uses only one negative sample, while Ultra-GCN's negative sampling strategy samples more than 500 negative samples. It is well known that GNN-based recommendation models can be improved by simply increasing the number of negative samples [8]; therefore, directly adopting the performance of the official implementations may result in unfair comparison. In Equation 1, we implicitly constrain that different models are compared under the same context . Note some hyper-parameters such as the learning rate and L2 coefficient are not considered as the context. For these hyper-parameters, different models may rely on different parameter settings to achieve their best performance, and we should carefully tune these hyper-parameters to obtain the best performance.

VectorSearch.
Many GNN-based recommendation approaches perform recommendation by similarity based searching, which ranks items based on the similarity scores between the dense vector representation of users and items. Most evaluation metrics require a global scan over all the items, which may result in large time and space complexity. We leverage industrial solutions such as Faiss [9] to build efficient similarity search libraries for dense vectors named VectorSearch, which can perform dense vector searching efficiently   with millions of candidate items. Furthermore, we improve the Vec-torSearch module with several re-implemented similarity search methods, such as binary vectors and compact quantization codes, to make searching more efficient and flexible.

BatchEval.
The evaluation of GNN-based recommendation models can benefit from mini-batch techniques, which can take advantage of GPU's parallel processing ability to improve the efficiency of the evaluation. It requires a lot of tricks to design minibatch implementations on irregular real-world recommendation data. To provide friendly and handy mini-batch solutions, we design BatchEval. The users only need to provide the learned representations or the similarity computation function, and BatchEval can automatically perform efficient mini-batch based evaluation with the provided information.

DataManager.
DataManager provide abstract Dataset class as interfaces for users to custom handy dataset APIs. The abstract Dataset class can automatically handle the whole lifecycle of data processing, such as data downloading, data preprocessing, data caching, etc. Usually, users can easily custom their Dataset class by subclassing the abstract Dataset class, providing the download urls of raw dataset, and overriding the preprocessing process. Then, the abstract Dataset class will handle the rest of data processing process. In addition, we already implement several widely-used recommendation datasets as Dataset classes, which can be directly used to load these datasets.

Implementations of GNN-based Recommendation models
In this paper, we implement a basic MF model [11] and the state-ofthe-art GNN-based recommendation algorithms (e.g. NGCF [14], LightGCN [4], UltraGCN [12]) as baselines. Especially, we carefully implement these baseline models to reproduce the performance reported in the literature, and our implementations are usually more efficient and friendly.

EXPERIMENTS
We conduct experiments on several benchmark datasets to provide a fair comparison of some popular baselines with GRecX. The complete experimental results are at https://github.com/maenzhier/GRecX. We are still working on some baselines to provide more comprehensive benchmark. Here, we only list partial experimental results on two benchmark datasets to demonstrate the performance of models implemented by GRecX and our new evaluation metrics GRMF-X.

Datasets and Baselines
We use three datasets: yelp2018, gowalla, and amazon-book. Note that some dataset may have different versions, and we use the version used by LightGCN [4]. The statistics of the three datasets is listed in Table 2. In terms of the baselines, we choose a basic model MF [11], and three state-of-the-art GNN-based recommendation models UltraGCN [12], NGCF and LightGCN [4].

Evaluation Metrics and Parameter Setting
Here we employ a widely-used metrics NDCG@20 and our new metrics GRMF-X (GRMF-NDCG@20) for evaluation. As mentioned in Section 2.1.1, for other hyper-parameters, we carefully tune these hyper-parameters and report the performance with them.
We use BCE and BPR as the ranking losses respectively and experimental results are shown in Table 3 and Table 4. For all comparable models, we set different parameters including number of negative samples and dimensionality of representation. In terms of number of negative samples, we set it to 1 and 800 to align with the original experimental settings of NGCF (one negative sample), LightGCN (one negative sample) and UltraGCN (800 negative samples). Note that UltraGCN means the UltraGCN _ model with using one negative sample in the training phase. For the dimensionality of representation setting, we set the dimensionality of learned user/item representations to 64 for all the models. Specifically, we find that NGCF model concatenates the output embedding of each graph convolution layer including input layer to construct the final users/items' representations, which are then combined with the CF mechanism for recommendations. Taking NGCF model with three layers as an example, its dimensionality of final output representations is equal to 256 (3 * 64 + 64), which may be unfair to other models. So, we design MLP+MF model which just replaces all graph convolution layer of NGCF model with an MLP layer. And we also show results of MF model with 64-and 256-dimensional representations.

Performance
In this paper, we implement a simple well-tuned MF model as a important baseline. And we introduce a naive evaluation metric GRMF-X for verifying the superiority of GNN-based recommendation models effectively. The results for recommendations on two datasets in terms of the BCE loss and BPR loss are reported in Table 4 and 3, respectively. Note that the experimental results are preliminary and will be updated continuously.
In addition, compared with results of UltraGCN _ in table 4, results of LightGCN with a same negative sampling startegy in table 3 shows better performances. Interestingly, compared with results of NGCF in table 3 and table 4, MF model (64-and 256dimensional representations) and MLP+MF model achieve better results on three datasets in most cases.
Moreover, we conduct experiments on three datasets to analyze the time of model inference in terms of different LightGCN implement codes, including GrecX, RecBole, and original LightGCN codes. Experimental results are shown in Fig 2. From the histogram results, we can clearly see that, compared to RecBole code and original lightGCN code, the Inference efficiency of the lightGCN code implemented by our GrecX framework has been significantly improved, especially on the two datasets Yelp2018 and Gowalla.

CONCLUSIONS
In this paper, we present GRecX, an open-source TensorFlow framework for benchmarking GNN-based recommendation models in an efficient and unified way. GRecX consists of core libraries for building GNN-based recommendation benchmarks, as well as the implementations of popuplar GNN-based recommendation models. With GRecX, we can efficiently perform fair comparison between different GNN-based recommendation models in a unified benchmark. In the future, we will integrate more baselines into GRecX and further improve the performance of both the core libraries and implementations.