{'fraction_non_alphanumeric': 0.024721878862793572, 'fraction_numerical': 0.012608158220024722, 'mean_word_length': 4.812140804597701, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Comparison of Two Scoring Methods of Chinese University Students’ Writing in EFL Course Using MultiFacet Rasch Model

Writing is one of the important productive skills for second language learners. Considering the inadequacy of the research on global scoring and descriptor-based checklist scoring in college English writing tests, this paper analyzes the scoring validity of the test at four dimensions: test-takers, rater, task, and scale and bias of rater-subject and rater-dimension. Based on the data of a college English writing test, the study will also employ the MultiFacet Rasch model (MFRM) to conduct a comparative analysis and bias analysis of the two scoring methods. The study showed that rater severity varied in global and descriptor-based ratings and had high internal consistency in descriptor-based scoring. This study can provide a reference for rater training and be used as a benchmark for deciding which scoring criteria to use in the L2 writing scoring for the EFL courses in China.


INTRODUCTION
Writing is a key component of second language testing and one of the most common forms of testing. However, test validity is difficult to be guaranteed because of several factors. In addition to students' writing ability, the difficulty of the writing task, the assessment criteria and scales which the raters apply to evaluate the writing, and the subjective nature of the rating would affect not only the final score but also the corresponding validity of the writing tasks. Therefore, assessors need to collect different types of evidence and use it to demonstrate that the scoring methods are valid.
The divergence emerging from the definitions of validity and the standards to verify validity even worsens the situation. For example, construct validity, based on True Score Theory, examines the extent to which the test takers' performance matches the language proficiency theory envisioned by the test developer [1]. This index focuses on the factor structure within the test by using exploratory and validation factor methods [2,3]. Another type of rating validity is based on Item Response Theory, including rating stability and inter-rater and intra-rater consistency [4,5]. In comparison to the former, rating validity is a more realistic estimating procedure, which requires raters to track various sources of error in writing tests and produce accurate descriptions of the results. The MultiFacet Rasch Model (MFRM), which was developed and adapted by Linacre based on the Rasch model, has been frequently utilized in recent years for evaluating productive, subjective tests such as speaking [6,7,8] and writing [9,10]. The computational logic of the model is to unify the estimates of each dimension in the same measurement panel so that they can be calculated linearly and be compared cross-dimensionally. As a result, the accuracy of the test scores depends on the difference among the ability of the test takers, the difficulty of the task, the severity of the rater, and the rating scale [11].
As a small-scale test, English writing assignments in college courses have gradually received more attention, while there have been a few studies on the validity of the rating scheme of this task. In general, a few studies have investigated the application of the Rasch model to measure writing tasks in college English courses in China [12].
To summarize, this paper aims to delve into the scoring validity of the propositional essay writing exam. The following research questions will be addressed in this study: 1) Whether the different scoring methods affect the validity of scoring results in the English writing test?
2) Is the scoring behavior of the scorers affected by the two different scoring methods, and is their scoring behavior consistent and stable?
In order to provide a comprehensive account of these research questions, this paper will analyze the phenomenon in aspects of subject differentiation, rater severity, and rating scale.

Participants and Writing Prompt
32 first-year students were recruited as the research participants, and their final writing scores in the college English writing courses were selected as the data source. In the writing task, the research subjects were required to write a 120-word essay within 30 minutes on the difficult level equivalent to Chinese College English Test Band 4.

Scoring Criteria and Data Collection
The essays were marked by 10 raters who were all graduate students in applied linguistics. The global scoring scale applied in the study was divided into three dimensions, which examined the quality of writing in terms of content, organization, and language use, respectively. The descriptor-based checklist was composed of 30 descriptors in 5 dimensions, namely, mechanics, vocabulary, content, organization, and language use.
Both scales contain five levels, ranging from1 to 5, with a 1-point scoring interval. This is intended to eliminate over-centered scoring. To maintain impartiality in grading, the information of the test takers was excluded from the essay, and all raters received rating training prior to scoring. Also, all raters employed two different scoring methods to assess the same writing. Table 1 and Table 2 demonstrate the overall situation of the global scoring model and the descriptor-based scoring model, respectively.

Comparison analysis of the global and the descriptor-based scoring
In Table I and Table II, the span of ability distribution (column 2) is substantial (approximately 3.2 logits), indicating that the global scoring can clearly identify and distinguish subjects' writing abilities. Table I and II is basically normally distributed, with rater 5 being the strictest and rater 4 the loosest in global scoring. There is a high level of inter-rater agreement in both rating methods.

The severity of raters (column 3) in
Concerning the level of the rating scale (column 4), Table 1 demonstrates that the subjects had the most difficulty in obtaining high scores in language use as the raters are stricter in assessing this item (about 0.2 logits), while in descriptor-based scoring, vocabulary (approximately 0.3 logits) is the most difficult.
The boundaries among each of the five items are relatively evident in both methods; For organization in global scoring, band 4 is not easily distinguishable from the proximity score; whereas for language use in descriptor-based scoring, band 1 and band 4 are difficult to differentiate from the proximity score.

Analysis of the rater effects
As seen in Table Ⅲ and Table Ⅳ, the raters are ranked in order of severity level from -1.12 to 1.21 logits, with a range of 3.33 logits. Rater 5 is the most severe, followed by rater 2, 10, 6, 9, 8, 7, 1, and 3, with rater 4 being the most lenient (roughly -0.21 logits). In Table Ⅳ, raters range from -0.71 to 0.97 logits, with a range of 1.68 logits. The most severe is rater 5, while the least severe is rater 2. The information in the table appropriately depicts the severity disparities among the raters.
At the bottom of Table Ⅲ and Ⅳ, the separation coefficient, reliability, and chi-square values are presented, which indicate the actual degree of difference in severity among the raters. The separation coefficients in Table 3 are 4.37 and 4.62 (greater than 2), revealing that the 10 raters could be divided into at least four severity categories with considerable disparities in severity amongst raters. However, the chi-square value of 185.4 for d.f. 9 is significant at a significance value of 0.00 and a reliability of 0.95.
There are two types of rater reliability, including inter-rater reliability referring to the agreement among the raters and intra-rater reliability referring to selfconsistency within one rater. In Table Ⅳ, with the exception of rater 2, all remaining nine raters reached a high level of internal consistency. Infit MnSq values imply that rater variation is likely to be lower than the model predicts as rater ratings tend to be clustered, centered, and overfitted.
In Table Ⅲ, only rater 9 and rater 2 did not fall within the interval of 0.5 to 1.5 in ratings. Rater 9 is less than 0.5, indicating that the rating is convergent, while rater 2 is greater than 1.5, indicating that the rater has poor internal consistency. This may be due to the rater's inexperience with scoring and the insufficient awareness of the scoring criteria. Also, the scoring process was divided into two sessions, which may result in changes in their criteria. However, the majority of the raters were reliable with high degrees of self-consistency.

Analysis of the examinee effects
With regards to the subject-level analysis in Table V, the separation coefficient was 3.99, and the separation reliability was 0.94, suggesting those established marking criteria can distinguish the participants' writing skills considerably. With all items being fittingly less than 1.5, the raters were proven to be consistent in evaluating these essays; however, the overfitting items of rater 5 demonstrated a convergence or halo effect when the rater reviewed the essays. Also, a reliable model fit can be observed with a merely 0.28 error value. However, because of the absence of overfitting figures, Table Ⅵ displayed no neutralization or halo effect when the scorers reviewed this essay. The error of the model fit was 0.14, which can also be considered a trustworthy model fit.

Analysis of the two rating scales
The model analysis provides probability plots of the scores for the different dimensions of the two scoring methods.
In global scoring, in content rating, the peaks of band 1 are not obvious, while organization and language use have distinct coverage regions and separate peaks, suggesting a scale of 1 to 5, it is difficult for the raters to separate band 1 from band 2, whereas the other ratings are easier to be provided.
In descriptor-based scoring, in general, the probability curves for the five dimensions exhibit significant peaks for scores from 1 to 5, indicating that the raters are able to capture the differences in these scores. To be more specific, in mechanics and language use, raters are able to grasp the differences between these bands well. In vocabulary, it is difficult for the raters to separate the band 2 from the neighbor band 4, while the other scores are easier to judge. In content and organization, all scores have sharp peaks except for the band 2, which is not obvious enough.

Bias in the rater-subject interaction
There were 320 experimental bias items, and Table 7 shows that there were 21 significant biases (11 of which were too severe and 10 of which were too generous), accounting for 6.5% , slightly above the acceptable range of 5%. The raters with significant bias were rater 1 (2 times too severe and 4 times too generous) and rater 2 (3 times too severe and 3 times too generous). These raters need focused training. The top raters were rater 3, 4, 5, 8, and 9, who scored very consistently with no bias.
In addition, according to the results of the ratersubject bias analysis, the rater deviated the most from subject 31 (3 serious bias), and the scores given by the rater ranged from 0 to 7. The topic of the essay is The challenge of living in a big city, but subject 31's essay only covers telephone calls, which is a deviation from the topic, and is probably a copy from the reading comprehension text. It can be found that the content, organization and language use of the essay are relatively neat, except for the main idea which does not meet the requirements of the essay. The relatively large scoring errors may be due to the influence of each dimension.

Bias in the rater-dimension interaction
There were 30 experimental bias items, among which 5 significant biases, accounting for 16.6%, were above the acceptable range. Table 8 shows the results of the bias analysis between the scoring dimensions and rater. There were five significant biases between the raters on three dimensions, and the bias mainly lied in language and content, of which three were too generous and two were too severe. Raters 2 and 7 showed more biased on the scoring, indicating that both raters lacked a deep understanding of the scoring criteria. This information suggests that some of the raters had an unstable grasp of severity in evaluating writings.

Comparison of bias analysis of two rating methods
A longitudinal comparison of rater performance revealed that rater 2 and 7 were prone to bias in both overall and checklist-based scale, and that rater 2 was overly severe in both rating methods, and Rater 7 was more severe when using checklist-based scale than overall rating. Rater 1, who had the most severe bias in overall scoring, had no bias in checklist-based scoring, indicating that rater 1 had less internal consistency in overall scoring. Perhaps because overall scoring always preceded checklist-based scoring, and rater 1 may have been more familiar with the content of the subject's writing when scoring was done for the second time and had a better understanding of the scoring process. The bias of rater 2 indicated that she lacked a stable standard in the marking process, which shows the importance of rater training, which should make the scorers clear about the scoring descriptors and maintain internal consistency.
The number of biases in dimensions is significantly smaller for overall ratings than for using checklist-based scale, which is caused by the small number of scoring dimensions on the one hand and the granularity of descriptors on the other. For fine-grained scales, it is important to clarify the dimensions of the descriptors in order to achieve both intra-rater and inter-rater consistency. Both checklist-based and overall scores showed large bias in content. This suggests that a clearer delineation of the descriptors of content was not achieved before scoring, and that individual raters had different criteria for the descriptors, which resulted in bias in content dimension.

CONCLUSION
MFRM provides a wealth of information to enable testers to gain a penetrating and accurate insight into the entire marking process and the dimensions of testing. With the help of the MFRM, the following findings in terms of rater impact and scoring criteria have emerged from this study. First, the severity levels of raters varied significantly, with rater 5 performing the most severely in both global and descriptor-based scoring. Second, rater 9 and 2 showed overfitting and underfitting in both global and descriptor-based scoring when they examined different writing assignments, indicating that rater 9 had a substantial halo effect while rater 2 had poor internal consistency. Also, when comparing the two distinct rating methods used by the same rater, the absolute values of Z for rater 7 and rater 5 were greater than those for rater 2 in descriptor-based rating and were less than those for rater 2 in the global scoring, indicating that these two raters were more severe when using the descriptor-based checklist and more lenient when using the global scoring. For both rater 9 and 2, they had absolute values of Z greater than 2 in both scoring methods, indicating that they applied the scales excessively and severely for both scoring systems. There is also a substantial variation in the severity levels of the raters but no significant differences in the difficulty of the global scoring dimensions.
Overall, the descriptor-based scoring results fit the model satisfactorily, demonstrating the high validity of the criterion and the reliability of the scoring results. However, due to the limitations of the experimental conditions, the number and representativeness of the research participants in this investigation were insufficient, the majority of whom were new and inexperienced in writing scoring so that they may exert a negative impact on the quality of scoring. On the one hand, the university should provide targeted and specialized training for the scorers, and on the other hand, audio thinking can be employed to analyze the scores in depth in order to reveal the deeper reasons and further improve the scoring criteria.