{'fraction_non_alphanumeric': 0.043564945692605266, 'fraction_numerical': 0.011575658384169022, 'mean_word_length': 4.893721501227639, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 4, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Improving Peer Assessment with Graph Convolutional Networks

Peer assessment systems are emerging in many social and multi-agent settings, such as peer grading in large (online) classes, peer review in conferences, peer art evaluation, etc. However, peer assessments might not be as accurate as expert evaluations, thus rendering these systems unreliable. The reliability of peer assessment systems is influenced by various factors such as assessment ability of peers, their strategic assessment behaviors, and the peer assessment setup (e.g., peer evaluating group work or individual work of others). In this work, we first model peer assessment as multi-relational weighted networks that can express a variety of peer assessment setups, plus capture conflicts of interest and strategic behaviors. Leveraging our peer assessment network model, we introduce a graph convolutional network which can learn assessment patterns and user behaviors to more accurately predict expert evaluations. Our extensive experiments on real and synthetic datasets demonstrate the efficacy of our proposed approach, which outperforms existing peer assessment methods.


INTRODUCTION
Peer assessment systems have emerged as a cost-effective and scalable evaluation mechanism in many multi-agent settings such as peer grading in large (online) classes, peer review in conferences, peer art evaluation, etc. In these systems, peers assess each others' work (e.g., assignments, papers, etc.) in lieu of a set of pre-appointed experts responsible for evaluation (e.g., instructors, teaching assistants, program committee members, etc.). These peer assessment systems not only make the evaluation of thousands of contributions plausible, but also help to deepen peers' understanding [35], and facilitate peers providing feedback to each other [31]. These benefits of peer assessment systems have come with some social and technical challenges, which impact their reliability and robustness.
The reliability of peer assessment systems is directly impacted by the accuracy of peers in their assessments. Peers might lack knowledge or motivation to accurately evaluate others, or they might be strategic in their assessments for their own gain. Two classes of approaches are taken to address these challenges. One primarily focuses on designing strategy-proof peer assessment mechanisms, which incentivize peers to accurately assess each other [10,13,18,27,38,48,52]. The other class of approaches-most relevant to our work-emphasizes learning peer aggregation mechanisms, which aggregate noisy peer assessments for an item (e.g., assignment or paper) as an estimate of its ground-truth valuation (or expert evaluation) [8,11,30,42,46].
The learning methods for peer assessment aggregation fall into unsupervised [9,30,42], and semi-supervised [11,46] approaches based on whether or not a subset of ground-truth labels are used for training in addition to peer assessment data. These models usually possess particular inductive biases such as peer's accuracy in assessment is correlated with his/her item's ground-truth valuations (e.g., the grade of his/her assignment) [11,30,42]; or peer's accuracy in an assessment depends on the extent of its agreement with others' assessments or ground-truth valuations [46]. However, these machine learning methods are empirically shown to be only as effective as simple aggregation mechanisms such as averaging [36]. Moreover, these approaches are not flexible and general enough to accommodate a wide variety of peer assessment modes (e.g., when an individual assesses the group contribution of others or self assessments). Our focus in this paper is to develop a semi-supervised aggregation mechanism without any specific or restrictive inductive bias, accommodating various modes of peer assessments.
We first introduce our graph representation model of peer assessment, which we call social-ownership-assessment network (SOAN). Our SOAN model can express a wide variety of peer assessment setups (e.g., self-assessment and peer assessment for both individual or group contributions) and represent conflict-of-interest relations between peers using auxiliary information (e.g., social networks). Leveraging our SOAN model, we then introduce a semi-supervised graph convolution network (GCN) approach, called GCN-SOAN, which can learn assessment patterns and behaviors of peers, without any restrictive inductive bias, to predict the ground-truth valuations. We run extensive experiments on real-world and synthetic datasets to evaluate the efficacy of GCN-SOAN. Our GCN-SOAN outperforms a wide variety of baseline methods (including simple heuristics, semi-supervised, and unsupervised approaches) on the same real-world dataset [36], which was shown to be challenging for machine learning approaches. We further analyze the robustness of GCN-SOAN on a wide range of synthetic data, which captures strategic assessment behavior between users, follows the assumptions of competitor baselines, or considers strict and generous graders. In all those datasets, even those that were tailored to the baselines, our GCN-SOAN outperforms others. Our GCN-SOAN approach can be a stand-alone approach or possibly be integrated with some existing mechanisms for incentivizing accurate assessments (e.g., [9,48,52]).

RELATED WORK
We review the related work on peer assessment aggregation methods, strategic behavior in peer assessment systems, incentivizing peers for accurate assessment, and node representation learning in graphs.
Peer assessment aggregation methods. Peer assessment aggregation methods fall into two categories from the machine learning perspective: unsupervised [8,30,42] and semi-supervised [11,46]. In an unsupervised approach, the goal is to aggregate peer assessments for an item (e.g., assignment) to estimate its ground-truth valuation (or grade) without using ground-truth labels. Average and Median-the most simple yet effective form of unsupervised aggregation methods-take the average and median of the peer assessments to estimate ground-truth valuations. In addition to these methods, there is a growing body of unsupervised methods which use the grade of each user's submission as a proxy of his/her accuracy in grading others [30,42]. These methods use various techniques such as iterative methods similar to PageRank [42], Bayesian inference by sampling [30], or expectation maximization [8]. In semi-supervised aggregation methods, the goal is to find the aggregation method for predicting ground truth valuations while using all peer assessments and a small subset of ground-truth labels as training data. Some models in this category are again based on the assumption that the user's submission's grade should be predictive of his/her grading accuracy [11]. However, some others obtain the grading accuracy of graders by comparing their assessments directly and indirectly to ground truth grades of a set of items [46].
Our work is a semi-supervised peer assessment aggregation method. However, our work is different from other aggregation methods in several ways: (i) it doesn't make any specific inductive assumption (e.g., correlation of grader's accuracy with his/her grades of items); (ii) it can accommodate various modes of assessment ranging from self-evaluation to peer assessment of group work; (iii) our solution benefits from auxiliary information (e.g., social networks) to capture conflict-of-interest relations between peers and correct its prediction.
Strategic behaviors and incentivizing accurate assessment. Strategic manipulation in peer assessment systems are those errors and inaccuracies in peer evaluations or systems, which are intentionally introduced by the users to get a more favorable outcome or assessment [2-4, 18, 19]. In a peer selection setting, an individual might misreport his/her ranking of other peers or their work to increase the chance of selection of someone's else (e.g., committee selection, award selection, etc.) or that person's work (e.g., conference review) [2,3,5,19,24,39,51]. Even in those settings that users do not compete with each other (e.g., classrooms), users might mutually agree to strategically inflate or deflate grades to reach their desirable outcome or assessment [33]. Since peer assessment is time-consuming, users might not put the effort to carefully evaluate peers' items and might give them higher/lower evaluation [13]. To motivate accurate grading behaviors, many solutions are proposed for rewarding graders based on whether their assessments agreed with other peers' assessments or ground-truth grades [13,44,46,48,52]. These solutions have a close connection to peer prediction models with the focus on designing the mechanisms to motivate users to put effort in truthful disclosure of private information [1,7,23,27,32,37,37,53].
Our work differentiates from this literature in several ways. Our goal is not to incentivize accurate assessment, nor to detect strategic behaviours. We aim to learn semi-supervised aggregation functions using graph neural networks, which learns to aggregate inaccurate assessments (even strategic assessments) to predict the groundtruth valuations. Our solution can be stand-alone or possibly be integrated with some existing mechanisms for incentivising accurate assessments (e.g, [9,48,52]).
Node representation learning in graphs. The goal of node representation learning in a graph is to encode each node into a lowdimensional embedding space such that similar nodes in the graph are embedded close to each other. Node embedding approaches can be categorized into semi-supervised [22,41] or unsupervised [15,29]. DeepWalk [29] and Node2Vec [15], two popular examples of unsupervised methods, use random walks starting from each node to generate a set of walks (or node's neighborhood). These random walks are then deployed analogous to sentences in Word2Vec [26] to extract the representation of each node, capturing both local and global information of each node in the graph. Graph Neural Networks (GNNs) [14] learn node embeddings by iteratively aggregating nodes' information from its local neighborhood. This iterative aggregation allows the node to gradually aggregate more information from farther nodes in the graph. The first iteration only captures the one-hop neighborhood; but after iterations, the information from the k-hop neighborhood is encoded to node embeddings. GNNs differentiate by their aggregation functions. Some notable examples include Graph Convolutional Network (GCN) [22], GraphSAGE [17], and Graph Attention Networks [41]. The GCN is one of the most popular semi-supervised GNNs that uses a symmetric normalization technique coupled with self-loops to aggregate and update node embeddings. Our work is built upon GCN.

PROPOSED MODELS AND ALGORITHMS
Our goal is to predict the ground-truth assessments (e.g., expert assessments of educational or professional work) from noisy peer assessments. We here first discuss our proposed graph representation model, social-ownership-assessment network (SOAN), for capturing the peer grading patterns and behavior. 1 Then, we present a modified graph convolution network (GCN) approach, which leverages our SOAN model, to predict the ground-truth assessments. We call this approach GCN-SOAN.

Social-Ownership-Assessment Model
We assume that a set of users U (e.g., students or scholars) can assess a set of items I (e.g., a set of educational, professional, or intellectual work). The examples cover various applications ranging from peer grading in classrooms to peer reviewing scientific papers, professional work, or research grant applications. We also consider each item ∈ I possesses a (possibly unknown, but verifiable) ground-truth value ∈ R + (e.g., staff grade for a course work, or expert evaluation of intellectual or professional work).
The user-item assessments can be represented by assessment where is the assessment (e.g., grade or rating) of user ∈ U for item ∈ I. We let = 0 when the user 's assessment for item is missing; otherwise ∈ R + . As the assessment matrix A is sparse, we equivalently represent it by an undirected weighted bipartite graph, consisting of two different node types of users U and items I, and weighted assessment edges between them (see Figure 1a as an example).
We introduce a social-ownership-assessment network (SOAN), an undirected weighted multigraph, consisting of three types of social, ownership, and assessment relationships on two node types of users and items. In addition to the assessment matrix A, this network consists of two other adjacency matrices: social matrix The social matrix S, by capturing the friendship and foe relationships between users U, can accommodate "conflict of interest" information. The ownership matrix O, by capturing which users to what extent own or contributed to an item, not only completes conflict of interest information but also provides flexibility of modeling group contributions, self-evaluation, etc. We let = (S, O, A) denote the tuple of all three networks of SOAN. Figure 1 demonstrates some instantiations of our models for various settings. Our SOAN model offers important advantages over the existing peer assessment models (e.g., [8,11,30,42]

):
Expressiveness. Our model is more expressive as it facilitates the representation of many various peer assessment settings that could not be accommodated in the existing models. Its expressive power can be realized in the settings such as self assessments (Figure 1b), peer assessments for both solo and group work (Figures 1e and 1f), and the mixtures of peer and self assessments for solo and group work (Figures 1c and 1d). For all of these settings, our SOAN model can also express conflict of interest (which is neglected in other models) through a social network (see Figure 1g).
Less Assumptions. Dissimilar to some existing models (e.g., [11,30,42,46]), our model avoids making explicit or implicit assumptions about the relationships between ground-truth values (or grades) and the quality of peer assessments. However, it is still flexible enough to learn such correlations from assessment data if it exists. Our experiments below have shown that our model outperforms other models with restrictive assumptions regardless of whether their assumptions are present in the data or not.

Graph Convolution Networks
Our learning task is semi-supervised. Given a social-ownershipassessment network = (S, O, A) and a set of ground-truth valuations D = { | ∈ D} for a subset of items D ⊂ I , we aim to predict for ∉ D. More specifically, we aim to learn the function ( | , ) for predicting the ground-truth valuation bŷ = ( | , ). The model parameters are learned from both the observed ground-truth valuations D = { | ∈ D} and socialownership-assessment network . We formulate the function by a modified graph convolution network (GCN) with a logistic head: where (.) is the sigmoid function for converting the linear transformation of the node 's embedding z into its predicted valuations. Here, w and are the weight vector and the bias parameter for the output layer. The node (i.e., item) embedding z is computed with layers of graph convolution network. Let H ( ) be the ( + ) × matrix of -dimensional node embeddings at layer for all users U and items I such that user and item 's vector embeddings are located at the -th and ( + )-th rows, respectively. In Eq. 1, the item 's embedding z is the ( + )-th row of H ( ) with the updating rule of The matrix M is constructed from social-ownership-assessment as the indicator function. 2 The core idea in Eq. 2 is to update the node embeddings at layer + 1, denoted by H ( +1) , from layer 's node embeddings H ( ) . This update includes the multiplication of the layer 's embeddings H ( ) by the normalized matrix D −1 M, then linear transformation by learned weight matrix W ( ) at layer , and finally passing through a non-linear activation function ( ) . The initial embedding matrix H (0) can be node-level features (e.g., textual features for items, user profiles for users, etc.). When the node-level features are absent, the common practice is to initialize the embeddings with the one-hot indicators [16,22,43]. As our GCN is built upon our SOAN model, we refer to this combination as GCN-SOAN.
The updating rule in GCN-SOAN (see Eq. 2) benefits from row normalization of the adjacency matrix similar to many other graph neural networks [41,43,45,47]. As the choice of an effective normalization technique is an application-specific question [6,16], we have decided to normalize our weighted SOAN model by taking an unweighted average, which has been suggested as a solution to address the sensitivity to node degrees for neighborhood normalization [16].
GCN can aggregate the information from multi-hop neighborhoods (e.g., neighbors, neighbors of neighbors, and so on) of SOAN. This overarching aggregation makes GCN-SOAN well-equipped to capture the assessment behaviors and patterns. For example, in our experiments below, we demonstrate how GCN-SOAN learns strategic behaviors in peer assessment and predicts the ground-truth valuations more accurately (compared to other methods).
Learning. Given the social-ownership-assessment network and a small training set of ground-truth valuations D , one can learn GCN-SOAN parameters by minimizing the mean square error for its predictions: where |D| is the number of items in the training dataset, and ( | , ) is the estimated valuation of GCN-SOAN for item . This loss function can be minimized by using gradient-based optimization techniques (e.g., stochastic gradient descent, Adam, etc.). As opposed to many existing peer assessment systems with unsupervised learning approaches (e.g., [8,30,42]), we deliberately have adopted a semi-supervised learning approach for predicting ground-truth assessment. This choice offers many advantages at some cost of access to a small training data. By learning from the training data, GCN-SOAN is well-equipped to mitigate the influence of strategic behaviors, assessment biases, and unreliable assessments in peer assessment systems. Of course, the extent of this mitigation depends on the size of training data. For the deployment, one can control the number of ground-truth valuations in training data to achieve intended prediction accuracy as the elicitation of ground-truth valuations from an expert (e.g., instructor grades an assignment) can be incremental as it needed.

EXPERIMENTS
We evaluate our GCN-SOAN model on various datasets while comparing it against other peer assessment methods. We explain in detail our datasets and experimental methodology, and report our results. Our results demonstrate notable improvements compared to the existing baselines.

Datasets
We run extensive experiments on real-world and synthetic datasets. While the real-world datasets allow us to assess the practical efficacy of our approach, we generate various synthetic data to assess its robustness and reliability in various settings (e.g., strategic assessments, biased assessments, etc.) Real-world dataset. The peer grading datasets of Sajjadi et al. [36] includes peer and self grades of 219 students for exercises (i.e., questions) of four assignments and their ground-truth grades. 3 For each specific assignment, the submissions are group work of 1-3 students, but each student individually has self and peer graded all exercises of two other submissions (in a double-blind setup). We treat all data associated with each assignment as a separate dataset, where all the submitted solutions to its exercises form the item set A and the user set U includes all students who have been part of a submission. We also have scaled peer, self, and ground-truth grades to be in the range of [0, 1]. Table 1 shows the statistics summary of our datasets.
Synthetic datasets. In order to evaluate our proposed approach in different settings (e.g., strategic assessments, biased assessments, etc.), we generate various synthetic datasets. We discuss different models used for the generation of our synthetic data. Ground-truth valuation/grade generation. For all ∈ A, we sample the true valuation from a mixture of two normal distributions ∼ (x; , , ) = 2 =1 N (x; , ), where = ( 1 , 2 ), = ( 1 , 2 ), and = ( 1 , 2 ), with , , and being the mixing probability, mean, and standard deviation of the component . Social network generation. We consider two general methods for creating social networks between users. In Erdős-Rényi (ER) random graph ( , ) model, each pair of users are connected to each other with the connection probability of . For our homophily model ({ }, O, ), we connect two users and if | − | ≤ when = 1 and = 1. This means that we connect two users if the ground-truth valuations of their owned items are close to each other (i.e, at the distance of at most ). This method of social network generation intends to preserve homophily on users' submissions' true grades. Ownership network generation. For all synthetic datasets, we randomly connect each user to just one item (i.e., one-to-one correspondence between users and items). This setup is in favor of existing peer assessment methods (e.g., [11,30,42]), which does not support group ownerships of items. Assessment network (or peer grades) generation. To generate peer assessment for each item ∈ A, we first randomly select a set of users ( ) ⊂ U such that | ( )| = . Then, for each ∈ ( ), we set 's assessment for item , denoted by , using one of these two models. The strategic model sets = 1, if the grader is a friend of the user who owns the item (i.e., . = 1); otherwise comes from a normal distribution with the mean of and standard deviation of . This implies that friends collude to peer grade each others with the highest grade, but would be relatively fair and reliable in assessing a "stranger. " The bias-reliability model draws from the normal distribution N (x;ˆ,ˆ) with the mean = + andˆ= (1 − ), where is the true valuation of item owned by user , and is the maximum possible standard deviation (i.e., unreliability) for peer graders. Here, the bias parameter ∈ [−1, 1] controls the degree of generosity (for > 0) or strictness (for < 0) of the peer grader. 4 The reliability parameter controls the extent that the reliability of the grader is correlated with his/her item's grade (i.e, the peer graders with higher grades are more reliable graders). 5

Baselines
We compare the performance of our GCN-SOAN model with Peer-Rank [42], TunedModels [30], RankwithTA [11], Vancouver [8], Average and Median. Average and Median (resp.) outputs the average and median (resp.) of peer grades of each item as the predicted evaluation for the item. As PeerRank, TunedModels, and RankwithTA treat users and items interchangeably, they could not be directly applied to our real-world data with individual assessments on group submissions. For these methods, we preprocess our real-world dataset by taking the average of the grades provided by a group's members for a particular group submission as the group assessment for the group submission. 6 For the PeerRank and Tuned-Models, we have used the same parameter settings reported by the original papers. The parameters for RankwithTA and Vancouver are selected by grid search with multiple runs, since the parameters for RankwithTA were not reported, and the suggested parameter of Vancouver results in non-competitive performance. 7

Experimental setup
We implement GCN-SOAN based on PyTorch [28], and PyTorch Geometric [12]. 8 For all experiments, we use two GCN-SOAN convolutional layers with an embedding dimension of 64. We use ELU (Exponential Linear Unit) as activation functions of all hidden layers. We train the model for 800 epochs with Adam optimizer [21] and a learning rate of 0.02. We initialize the node embeddings with the vectors of ones. We use Monte Carlo cross-validation [50] with the training-testing splitting ratio of 1:9 (in synthetic data) or 1:4 (in real-world data), implying that just 10% or 20% data is used for training and the rest for testing. To make our results even more robust, we run our model on four random splits and report the average of error over those splits. For each random split, we compute the root mean square error (RMSE) over testing data as the prediction error.

Results
We report our results on both real-world and synthetic datasets to evaluate our model in various assessment settings.
Real-World Datasets. To assess the effectiveness of GCN-SOAN in predicting ground-truth valuations, we compare it against the baseline methods on eight real-world datasets. These datasets differentiate on (i) which assignment dataset is used and (ii) whether both peer and self-grades are used or only peer grades. For GCN-SOAN, we just create an assessment network, thus allowing us to measure how the assessment network alone can improve the predication accuracy. As shown in Table 2, our GCN-SOAN model outperforms others in five datasets and ranked second in remaining ones with a small margin. RankwithTA and TunedModel are the only models that slightly outperform GCN-SOAN for those three datasets. Notably, GCN-SOAN is the only model which consistently outperformed the simple Average benchmark. This observation is consistent with Sajjadi et al. 's findings [36] on the same dataset that the existing machine learning methods (not including ours) could not improve results over simple baselines. However, the conclusion does not hold anymore as our machine learning GCN-SOAN approach could consistently improve over simple baselines. This improvement mainly arises from the expressive power and generalizability of GCN-SOAN (discussed in Section 3).
We also further investigate if the efficacy of GCN-SOAN can be improved by adding additional convolutions layers. Figure 2 shows how the testing error of GCN-SOAN changes by the number of convolutional layers over various datasets. 9 The error first sharply decreases then slightly increases with the number of layers (for all datasets). The optimal error is achieved with two convolution layers. This observation is related to "over-smoothing" issues found in GCN-like models [22,25,34,40,49]. For future work, one can adopt various techniques (e.g., [25,34,40,49]) to mitigate oversmoothing issue of our GCN-SOAN.
We also explore if the addition of the ownership network can further enhance the performance of GCN-SOAN. As shown in Figure 3, surprisingly, the addition of ownership network has slightly improved the GCN-SOAN's error. We hypothesis that this downgrade might be partially blamed on how we have linearly combined the ownership network and assessment network in the updating rule of GCN-SOAN (see Eq. 2). Further algorithmic development to address this shortcoming is an interesting future direction.
Synthetic Data: Bias-Reliability. We run an extensive set of experiments with the bias-reliability peer grade generation model further to assess our GCN-SOAN under various peer assessment settings. For these experiments, we define a default setting for all parameter of synthetic generation methods (e.g., bias parameter 9 The results for peer evaluation datasets were qualitatively similar.  Figure 3: Root mean square error for GCN-SOAN for realworld datasets with and without ownership networks. Average over four runs. , reliability parameter , etc.). For each experiment, we fix all parameters except one; then, by varying that parameter, we aim to understand its impact on the performance of GCN-SOAN and other baselines. Our default setting includes a number of users = 500 and number of items = 500; random one-to-one ownership network; = (0.3, 0.7), = (0.1, 0.1), and = (0.2, 0.8) for the ground-truth generation method; 10 the number of peer grades = 3, 11 = 0.25, bias parameter = 0, and reliability parameter = 0 for assessment network generation; and no social network generation. 12 Figure 4a shows how the prediction error of various methods changes with the number of peer graders while the other parameters are fixed to the default setting. Unsurprisingly, the performance of all methods improves with . GCN-SOAN not only outperforms  Table 2: Root mean square error of various methods over two classes of real-world datasets. The first and second best are shown with dark and light gray backgrounds, respectively. ↑ and ↓ denote better and worse than Average. GCN-SOAN (ours) is the only method that consistently has outperformed Average for all datasets. Results are averaged over three runs.  others for any , but also exhibits significant improvement over others for a relatively small (e.g., ≤ 4). This superiority of GCN-SOAN with minimal number of peer graders is its strength to make peer assessment suitable and practical for different applications, as so many peer assessment requests will put unnecessary stress and burden on users, thus impeding the practicality of the system. Figure 4b illustrates the errors for each model while changing the bias parameter (and keeping other parameters fixed to default). GCN-SOAN performs significantly better than other models for any bias values, including generous ( > 0) and strict graders ( < 0). GCN-SOAN owes this success to its ability to learn students' grading behavior by leveraging a small portion of ground truth grades and assessment network structure. These experiments show that our model could be a great choice for those peer assessment settings where the peer grades are intentionally or unintentionally overestimated/underestimated. Figure 4c reports the errors for various values of reliability parameters . Recall that the controls the extent that the accuracy of each peer in his/her assessments is correlated with his/her item's grade. Our results show that GCN-SOAN is very competitive to other models, even those built based on this correlation assumption (e.g., [30,42]). We observe that only when > 0.8, PeerRank and RankwithTA outperform GCN-SOAN. One might argue that > 0.8 is implausible scenario in practice. However, this result suggests that our model is still competitive choice for settings in which peer assessment accuracy is correlated with peer success.
To study how various ground-truth generation distribution impacts the prediction error of various method, we first change the default biomodal mixture of normal distributions (for ground truth generation) to a normal distribution by setting 1 = 2 and 1 = 2 = 0.15. Then, we only vary the mean of distribution while other parameters are fixed to default. As shown in Figure 4d, GCN-SOAN consistently outperforms others regardless of the underlying ground-truth distribution. Notably, PeerRank and RankwithTA do not perform well when most users own items with low grades.
We also explore how homophily (i.e., users' own items with similar ground-truth valuations are more likely to friend with each other) affects the predictive error of peer assessment methods. For this experiment, we generate social networks with our homophily model while varying its parameters. We set other parameters to default. Table 3 demonstrates that only GCN-SOAN's error decrease with . However, the improvements are relatively small. We believe these results highlight an interesting future direction for devising graph neural network solutions, which can capture homophily more effectively.  Synthetic Data: Strategic Assessment. We study the performance of all peer assessment methods under the strategic model discussed in Section 4.1. For this set of experiments, we define this default setting: number of users = 500 and number items = 500; random one-to-one ownership network; = (0.3, 0.7), = (0.1, 0.1), and = (0.2, 0.8) for the ground-truth generation method; the number of peer grades = 3 and = 0.25 for assessment network generation by the strategic model; and ER random graph model with = 500 and = 0.05 for social network generation.

Model
To study how the connection density of colluding social networks impact the accuracy of peer assent methods, we vary the connection probability while keeping other parameters fixed. Figure 5 show the outstanding performance of our model compare to other benchmarks and illustrate how our model is more resilient to colluding behaviors. This result suggest that GCN-SOAN is welleqiupped to detect conflict-of-interest behaviors and mitigate the possible impact of any strategic behaviors.

CONCLUSION AND FUTURE DIRECTIONS
We represent peer assessment data as a weighted multi-relational graph, which we call social-ownership-assessment network (SOAN).
Our SOAN can easily express many different peer assessment setups (e.g., self assessment, peer assessment of group or individual work, etc.). Leveraging SOAN, we introduce a modified graph convolutional network approach, which learns peer assessment behaviors, to more accurately predict ground-truth valuations. Our extensive experiments demonstrate that GCN-SOAN outperforms state-of-the-art baselines in a variety of settings, including strategic behavior, grading biases, etc.
Our SOAN model provides a solid foundation for the broader investigation of graph neural network approaches for peer assessments. Our GCN-SOAN can be extended to mitigate the oversmoothing effect observed in our experiments, or to include a different set of network weights for each relation type of social, assessment, and ownership. Another promising direction is to assess the effectiveness of GCN-SOAN or its extensions on real-world assessment data, with the presence of social network data.