{'fraction_non_alphanumeric': 0.025818521983161834, 'fraction_numerical': 0.007670720299345182, 'mean_word_length': 5.198685736374179, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Severity-Leniency in Writing Assessment and Its Causal Factors

Objectivity in a writing assessment is a highly important matter because it determines the validity and reliability of the writing assessment itself. However, subjectivity in the writing assessment is inevitable. Rater subjectivity in the assessment can lead to assessment difference or variability (severity-leniency) eventually reducing the validity and reliability of the assessment. Therefore, the concept of rater variability, severity-leniency, and its causal factors are required to perceive so that efforts to minimize assessment variability, for instance, severity-leniency, can be implemented. The research aimed at providing a brief description of rater variability, namely severityleniency, and the causal factors, namely rater’s background, criteria, method, and assessment scale. The method used in this research was literature study which source was originated from the articles of scientific journals and books related to the research topic, namely rater variability (severity-leniency) and its causal factors. The result of the discussion in this paper showed that the selection of experienced rater, coherent and cohesive assessment criteria, method appropriate to the aim of the research, and brief assessment scale were considered to be able to minimize the variability (severity-leniency) of the rater.


Introduction
Writing is a complex process necessary to expand the learning, thinking, and communicating with other people [1]. Even in Indonesia, producing academic writing and publishing are the necessity as the graduation requirement to fulfill by bachelor, master, and doctoral students [2]. Based on that policy, it is considered that writing is a highly necessary matter because it influences the future and career of many people.
However, there are problems related to the writing activity, particularly in a writing assessment. The success of the writing assessment is determined by rater [3].
The assessment involving rater generally had disadvantages, namely subjectivity [4].  assessment. The statements of the experts could be proven by the research related to the variability of writing assessment involving rater. One of them was the research conducted by Schaefer [10].
In his research, Schaefer [11] involved 40 native speakers of English to evaluate 40 essays written by the students of English program in Japan. Each rater evaluated 40 essays by using analytical assessment scale with five categories (content, organization, style and expression quality, language usage, and fluency). The result of his study using Many Facets Rasch Measurement (MFRM) showed that content or organization category had quite a tight score (severity) and the language usage category had quite a loose score (leniency). He also found that the raters evaluated the great-skilled writers more tightly (severe) than those with the low skill (lenient).
Severity-leniency assessment referred to the rater's tendency to consistently give a lower or higher score than the score it is supposed to be [12]. This definition is in line with the opinion of other experts stating that severity-leniency is the rater's tendency in conducting an assessment which advantages or harms the people being evaluated [13]. The definition stated by Engelhard [14] is that the rater's tendency in evaluating higher or lower than how it was supposed to be, is seen to be more appropriate. It was because the measurement of higher and lower stated by Engelhard [15] was clearer and easier to be measured than the assessment term "advantage or disadvantage" as stated by Knoch [16].

The Causal Factors of Rater Variability
The background, mother tongue, experience, and training of the rater were stated as the factors influencing the writing assessment. The factors were believed to be able to influence the accurateness, accuracy, and justness in the writing assessment. The researcher found that the factors related to the rater including the rater's background, assessment method used, assessment criteria, professional experience, and error tolerance [17].
Shohamy [18] studied several factors which could influence assessment, namely the rater's background, training, and assessment scale. According to the experts' opinions and library research, the factors influencing rater variability could be concluded into four categories, namely rater's background (including training and experience), assessment criteria, assessment method, and assessment scale.

The Background of the Rater
Rater's background was the factor believed to be able to cause variability or differences between raters related to severity or tightness in assessment [19]. It was in line with the opinion of Barret [20] stating that the background factor was one of the most important matters because, besides assessment criteria, it was what caused rater variability.
Rater's background included personal background, professional background, and evaluating experience [21]. Personal background generally is determined by age, gender, education, etc. Professional background or rater's occupation can influence the rater's reliability. Teacher or tutor, for instance, can be influenced by the instructional aims and particular components in the writing assessment. On the other hand, novice rater could evaluate based on their personal view of the writing. Cumming [22] stated that the rater's experience referred to how long the rater had been evaluating or the number of the assessment completed by the rater. Several studies about experienced raters and inexperienced raters showed different interrater consistency [23].
There was much research investigated the difference between experienced/expert rater and novice rater to bring the problem of rater's background [24]. The research of Cumming [25] showed that compared to the novice rater, expert raters tended to use some wider criteria. Wolfe [26] showed that experienced raters tended to focus on more general (holistic) assessment criteria than the novice raters did. Song and Caruso [27] also found that raters having more experience tended to evaluate more softly than those having less experience did when using holistic assessment.

Assessment Criteria
Popham [28] argued that criteria are a tool used in assessing student's competence in a certain field. Brookhart [29] asserted that good assessment criteria are the existence of coherency inter-assessment criteria and the existence of level or stage in assessment criteria. Criteria in writing assessment column should be coherent which means that inter-criteria should be compatible or there is no contradictory statement in criteria.
In writing assessment criteria, there is also a level or stage to assess performance, work, or evaluated writing activity done individually. As assessment guideline which contains assessment criteria, the column is explained as follows. Mertler [30] argued that assessment criteria should have been settled beforehand to make compatibility with the determined assessment goal. Thus, assessment criteria help an individual to IC-HEDS 2019 understand why they get a certain score in one of assessment and what they should be done to increase their performance for the next assessment.
Stevens and Levi [31] explained that assessment criteria as part of specific feedback.
It is mentioned so because this part enables the assessor in giving specific feedback to an evaluated individual. The good assessment criteria can provide detail feedback about parts of the assignment and how good or bad performance which is done. In assessment criteria, there are details of task assessment components which can be known by other individuals. Thus, an evaluated individual can evaluate themselves in what part their weakness and strength performance is. In this assessment criteria, an evaluated individual also gets assessment transparency since the obtained score can be investigated.

Assessment Method
There are two types of assessment methods in writing, namely holistic and analytical.
The holistic method is used to assess the entire processes or products without assessing the components of the products or processes separately [32]. Conversely, the analytical method is used to conduct scoring separately. The product or performance is assessed at first. Then, the assessment score is summed to obtain a total score.
A holistic method focuses on the whole assessment process regardless of the parts.
In holistic scoring, the focus of the assessment is directed at the holistic or overall writing performance rather than on certain aspects of the essay such as content, organization, grammar, punctuation, and so on. It causes such scoring is not acceptable to measure the specific competency in writing skills. However, this assessment method is considered more practical because the assessor does not need to read it repeatedly to give a rating. It makes the holistic assessment process faster and more comprehensive.
This assessment method usually contains a description of the highest level of performance to describe in the comment column for each criterion to describe the achievement of individual performance at that level. Nitko [33] stated that the application of the holistic method is more appropriate if the given task does not have a definite correct answer. The focus of holistic method application lies on the overall quality, skills or content's understanding specifically, and other skills which involve assessment at a unidimensional level [34].
The assessment process by the holistic method is relatively faster than that by the analytical method [35]. It is because the assessor does not have to repeatedly conduct the assessment [36]. Since the key is the overall performance assessment, a holistic IC-HEDS 2019 method is used for summative performance assessment. Expected feedback of the rubric assessment in this type is limited or not excessive. In a holistic assessment, assessment criteria are possible in the form of a combination or a combination of criteria placed on a single descriptive scale [37]. The assessment by a holistic method supports a broader assessment of the quality of the process or product. The assessment format by using the holistic method according to Mertler [38] is shown in Table 1.  Svingby [40] reported that analytic method assessment can be an effective tool. Analyticmethod assessment allows us to separately evaluate on each component, factor, or its part. Each criterion in analytic method assessment is assessed based on a different scale [41]. Analytical-method assessment is employed when the needed responses of the individual are focused [42]. For the procedure of the assessment, assessment result by analytic method initially is in the form of scores, and the total score is eventually summed up.
Analytic method assessment focuses on the scoring of components assessed by calculating the mistakes in detail. The total score is the merger of the scores from each component. The advantage of this scoring technique is that the teacher as the rater is allowed to assess all elements supporting students' writing skills in more detailed. For the students, this scoring method helps them understand the elements that must be concerned in a text. Moreover, the students are allowed to evaluate whether their writing is good or not by using the assessment criteria. As mentioned previously, the use of analytic method assessment causes the process of the assessment slower, especially because the assessment of some skills or characteristics are individually different. Consequently, the rater must check the product or process more than once. Both construction and its implementation can be too timeconsuming. Individual work should be checked in a different time for each particular performance task or assessment criteria [43]. Moreover, the analytic method assessment possibly causes the overlap among the determined criteria.
However, the advantage of using the analytic method is quite significant, that is the special feedback related to individual performance on each assessment criterion.
This feedback is not available in holistic method assessment [44]. Therefore, holistic method assessment allows the rater and the assessed ones to draw the strengths and weaknesses of individual performance assessed [45]. The assessment format of analytic-method according to Stevens and Levi [46] is shown in Table 2.

Assessment Scale
The scale is the achievement level in the form of a score. The scale represents how well or badly the task is done [47]. One of the scales commonly used is arranged by Huba District uses four quality gradations from below basic, basic, proficient, up to advanced.
The difference is also shown from the quality score made for each assessment category.
For organization, it is given maximum quality score 20, elaboration of idea/content is given maximum quality score 30, grammar is given maximum quality score 25, vocabulary is given maximum quality score 20, and mechanism is given maximum quality score 5. Thus the total score that can be obtained by students is 100. Andrade, the quality gradation of unsatisfactory is given quality score 1, fair is given quality score 2, and good is given quality score 3. Thus, the maximum score that can be obtained by students is 18 points with its minimum score of 6 points. The same as the assessment scale by Hamp-Lyons and Andrade, assessment scale employed in San Diego Unified School District also uses numbers to explain the quality obtained by students. Nevertheless, this rubric matches the number of quality gradations used as many as the quality score used, namely 4 points: 1 for below basic, 2 for basic, 3 for proficient, and 4 for advanced.

Discussion
It is not that difficult to minimalize the variability (severity-leniency) and achieve high inter-rater reliability. It surely requires optimum efforts. An important effort to minimize the severity-leniency is by investigating the background of the rater. It is important since the background of the rater is the factor that potentially causes the inter-rater variability or difference related to the severity of their strictness in assessing [52]. Another effort is by assessor training before assessing. Assessor training indeed cannot eliminate rater variability, but the rater can be more consistent and improve the reliability of the assessment [53].
The decision related to the use of holistic or analytic method assessment for writing an assessment to minimalize the severity-leniency of the rater has some possible implications. The most significant one is the consideration of the goal of assessment and how DOI  the assessment result is employed. If the assessment is general and summative, holisticmethod assessment is more suitable. Vice versa, if the goal is formative feedback, the suitable assessment is the analytic method. The time requirements, the nature of the task, and the specific performance criteria observed must also be considered.
In terms of the selection of the rating scale to minimalize severity-leniency of the rater, there is no definite formula to determine the number of levels or length of the scale used in the assessment scale. Generally, experts use three-level-scale up to fivelevel-scale. The more the number of levels is, the more difficult it is to distinguish and explain precisely why an assessment is included in that level. Most experts consider the assessment of three levels as the optimal assessment level of assessment scale [54].

Conclusion
To minimalize the variability of the rater in writing an assessment, considering the causal factors of severity-leniency is a must. Those are the rater's background, the criteria, the method, and the assessment scale. The efforts to choose the experienced rater, coherent and non-overlapping assessment criteria, a method suitable to assessment goal and short assessment scale are believed to be able to minimize the variability (severity-leniency).