{'fraction_non_alphanumeric': 0.03306013082197179, 'fraction_numerical': 0.022026952478524705, 'mean_word_length': 4.982791136256483, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Automated Evaluation for Student Argumentative Writing: A Survey

This paper surveys and organizes research works in an under-studied area, which we call automated evaluation for student argumentative writing. Unlike traditional automated writing evaluation that focuses on holistic essay scoring, this field is more specific: it focuses on evaluating argumentative essays and offers specific feedback, including argumentation structures, argument strength trait score, etc. The focused and detailed evaluation is useful for helping students acquire important argumentation skill. In this paper we organize existing works around tasks, data and methods. We further experiment with BERT on representative datasets, aiming to provide up-to-date baselines for this field.


Introduction
Automated writing evaluation uses computer programs to give evaluative feedback to a piece of written work, which is often used in educational settings (Parra et al., 2019). Most of the work on automated writing evaluation focus on automated essay scoring (AES), which evaluates an essay's quality by assigning scores (Shermis and Burstein, 2013). It is a long-standing research problem, with the first system proposed in 1966 by Page (1966). Since then this area has been active and attracts a lot of research efforts.
A large portion of AES systems are developed for holistic scoring (Ke and Ng, 2019), which outputs a single score to represent essay quality. This kind of system is useful for summative assessment and can greatly reduce manual grading efforts. However, holistic scoring is not sufficient for providing instructional feedback to student learning because a low holistic score does not provide enough information to the students about how to improve. To address this issue, many research works tried to build trait-specific scoring systems. These systems concern scoring particular quality dimensions of an essay, such as grammar (Burstein et al., 2004), word choice (Mathias and Bhattacharyya, 2020), coherence (Somasundaran et al., 2014), etc. Holistic scoring systems, along with trait specific scoring systems, have already been deployed to commercial settings successfully (Burstein et al. 2004, Rudner et al. 2006. However, most of the systems do not distinguish between essay types (e.g. argumentative or narrative essay). It makes sense when the system is trying to evaluate type-agnostic dimensions such as word choice of an essay, but the use of such systems is limited when an user wants to know more about the in-depth traits specific to an essay type, e.g. whether claims and counterclaims are developed fairly in an argumentative essay. The evaluation of type specific traits is important because these traits reflect important type-specific skills -for example, whether claims and counterclaims are developed thoroughly is indicative of critical thinking skills, which can be hard to gauge through narrative essay writing.
In this paper we attempt to organize the research works regarding automated evaluation specifically for student argumentative writing, which requires the students to evaluate controversial claims, collect and judge evidence and establish a position. The contributions of this paper are as follows. First, we categorized and organized current research body regarding tasks and datasets (section 2) and methods (section 3). Additionally, we experimented with BERT (Devlin et al., 2019) models and provided up-to-date baselines to the community (section 4). Finally, we suggested directions (section 5) based on missing elements in current research body which we hope to be filled in the future.

Tasks and Datasets
In this section, we introduce the tasks commonly studied in automated writing evaluation for argumentative writing, together with their benchmark datasets.

Argument Mining
Argument mining (AM) aims to identify and parse the argumentation structure of a piece of text.
(Lawrence and Reed, 2020) For argumentative essays, argumentation structures may have variations but can typically be represented as trees, as illustrated in Figure 1. The root of the tree is a major claim, which expresses the author's main point on the topic. Children of the major claims are claims, which are controversial statements that either argue for or against their parents. Claims could have children nodes, called premises (e.g. P remise1 and P remise3 on Figure 1), which support or attack the corresponding claims. Further, a premise (e.g. P remise3) can also be supported or attacked by other premises (e.g. P remise4 and P remise5), enriching the logical flow of the essay. In order to generate an argumentation structure from text, argument mining models typically decompose the process into four sub-tasks: (1) argument component identification (ACI), which aims to identify the span of argument components (e.g. text with squared brackets on Figure 1); (2) argument component classification (ACC), which classifies argument components into corresponding types (e.g. where a span is a Claim or Premise); 3) argument relation identification (ARI), which aims at linking related argument components on the argument structure; 4) relation labelling (RL), which focuses on identifying relation type (e.g. for and support) between linked components.
Datasets Gurevych, 2014a (S&G2014) and Gurevych, 2017a (S&G2017a) annotated the most widely adopted datasets regarding this direction. S&G2014 annotated 90 persuasive essays from essayforum, a site providing feedback for students who wish to improve their writing. S&G2017a adopted similar annotation scheme and annotated additional 402 essays. Besides, Putra et al. (2021a) annotated 434 essays written by English to Speakers of Other Languages (ESOL) college learners as quasi-trees. They annotated each sentence as either argumentative component (ACs) or non-argumentative component (non-AC). Annotators then identified relations between ACs. Different from S&G2014, their relation labels include directed labels (support, attack or detailing) as well as an undirected label(restatement). They also reordered the sentences to make the essays better-structured. Wambsganss et al. (2020) annotated 1000 student peer-reviews  written in German, indicating whether a text span is claim or premise or not, and the relations between these argumentative components.
Alhindi and Ghosh (2021) annotated on a token level using BIO tagging schema and this dataset can be used for ACI and ACC. Specifically, each token belongs to one of the five classes: the begin token of a claim; a continuous token of a claim; the begin token of a premise; a continuous token of a premise; non-argumentative token.

Component Detection
Besides parsing the complex argumentative structures of essays as in argument mining, a large portion of work in automated writing evaluation also considers the simpler task of detecting whether an essay or a text span in an essay contains a specific component. For example, Stab and Gurevych (2016) assigned binary labels to essays depending on whether an essay has discussed the opposing arguments to author's own standpoint. Falakmasir et al. (2014) labelled essays as whether or not containing a thesis and conclusion statement. (Beigman Klebanov et al., 2017) annotated essays written by college students for criticizing a piece of argument, deciding if each sentence contains a good critique or not. (Ghosh et al., 2020) annotated on whether a sentence contains a valid critique as well, but the essays they used are written by middle school students.
Finally, we remark that component detection is different from the task of argument component classification (ACC) discussed previously in Section 2.1. Specifically, component detection does not require the text span to match the component of interest. For example, Falakmasir et al. (2014) cares about whether an essay contains a thesis and conclusion statement. In their case, the length of text span is much longer than the component of interest.

Quality Assessment
Quality assessment concerns evaluating the quality of an argument. There are many sub-tasks under this category due to the diversity of argumentation quality theories. Persing and Ng (2015) annotated 1000 essays over 10 prompts from International Corpus of Learner English (ICLE) dataset (Granger et al., 2009) on argument strength. They defined argument strength as how well an essay makes an argument for its thesis and convinces its readers. Horbach et al. (2017) collected 2020 German essays written by prospective university students. Their annotators evaluated the text regarding 41 aspects including quality of argumentation. In addition, Stab and Gurevych (2017b) (S&G2017b) adopted Relevance-Acceptability-Sufficiency criteria (Johnson and Blair, 2006) and asked annotators to decide whether a piece of argument is sufficiently supported or not.
Besides, Carlile et al. (2018) annotated detailed persuasiveness and attributes values based on argument trees from S&G2014 and S&G2017a. They defined an argument as a node in the argument tree along with all its children. For each argument, they assigned an overall persuasive score, common attribute values and type specific attribute values. For example, for arguments using a major claim as the root node, annotations include persuasive score, eloquence, specificity, evidence (common attributes) and persuasive strategies (type-specific attribute).

Methods
All the methods to our knowledge are learning based and they can be divided into supervised learning and unsupervised learning. Most of the work use supervised learning, which can be further divided into feature-based approaches and neural approaches. Next, we introduce these approaches in more details.

Feature-based method
For feature-based approaches, off-the-shelf algorithms are typically used for model training on hand-crafted input features. For example, support vector machine (SVM), logistic regression and random forest are typically used for tasks that can be framed as classification (Stab and Gurevych, 2014b, Stab and Gurevych, 2017b, Stab and Gurevych, 2017a, Persing and Ng, 2016, Beigman Klebanov et al., 2017, Wan et al., 2021. Linear regression and support vector machine regression are often used for task that can be framed as regression (Persing and Ng, 2015, Wachsmuth et al., 2016, Persing and Ng, 2015.
Next, we introduce more details of common features used in these methods.
Lexical features aim to capture word-level information and common lexical features include ngram and frequent words. They have been shown effective (Stab and Gurevych, 2014b, Beigman Klebanov et al., 2017, Stab and Gurevych, 2017b. However, they do not perform as well when used in a cross-prompt setting (Beigman Klebanov et al., 2017) where prompts of testing essays are not seen during training. This is intuitive as the actual wording of essays of different prompts would differ significantly.
Syntactic features usually rely on parse trees. Common syntactic features include number of subclauses in a parse tree, Boolean indicator of production rules, part-of-speech tags, etc. Additionally, basic information such as tense of verbs, presence of modal verbs can also serve as syntactic features. (Stab and Gurevych, 2017a) showed that syntactic features are useful for identifying the beginning of an argument component and (Stab and Gurevych, 2017b) suggested that syntactic features are effective for recognizing insufficiently supported arguments.
Structural features generally describe the position and frequency of a piece of text. For example, they include position of a token, punctuation and an argument component. They also include statistics such as number of tokens in an argument component. (Stab and Gurevych, 2017a) reported effectiveness for these features on both ACI and ACC tasks.
Embedding features are based on word vectors that represent words in a continuous space and are supposed to capture more information than simple n-grams. Stab and Gurevych (2017a) summed the word2vec (Mikolov et al., 2013) vectors for each token to represent a component. Putra et al. (2021b) used BERT (Devlin et al., 2019) to extract token embeddings in their work.
Discourse features captures how sentences or clauses are connected together. One kind of discourse features depend on discourse markers directly. The markers, such as "therefore", suggest the relationship between current text span and its adjacent text span. Another kind of discourse features use the output of discourse parsers. For example, Beigman Klebanov et al. (2017) parsed sentences into corresponding discourse roles, and then used these discourse roles as features. Stab and Gurevych (2017a) reported usefulness of discourse features on classifying argument components, indicating a correlation between general discourse relation and argument component type. Beigman Klebanov et al. (2017) found that discourse features remain useful in cross-prompt settings, which is valuable as it's not always possible to collect a lot of data for a single prompt.
Use of pretrained models The use of pretrained language models has been popular among natural language processing community. This is because state-of-the-art models are pretrained on massive text corpus, allowing information learned from a huge amount of text to be used for downstream tasks.
There are two ways to use pretrained models. First, it can be used as feature extractor. For example, Putra et al. (2021b) has used BERT, a bidirectional Transformer based architecture, to extract input embeddings, and then pass the contextualized embeddings to downstream networks. Second, the pretrained language models can be further fine-tuned for the task of interest. Ye and Teufel (2021), Wang et al. (2020), Alhindi and Ghosh (2021) and Ghosh et al. (2020) used this approach. In addition, other than fine-tuning on the task data directly, Alhindi and Ghosh (2021) and Ghosh et al. (2020) experimented with continued pre-training with a large unlabelled domain relevant corpus first, and then fine-tuning with task data.

Address multiple sub-tasks
As mentioned, argument mining mostly concerns four sub-tasks and they are often addressed together. The naive way to solve them simultaneously is to model each sub-task separately in a pipeline fashion. This introduces at least two issues: first, it does not enforce any constraints between different tasks; second, the errors made early on in the pipeline could propagate.
One way to address these is to use integer linear programming (ILP) (Roth and Yih, 2004) for joint inference. Stab and Gurevych (2017a) and Persing and Ng (2016) adopted this approach and Persing and Ng (2016) further proposed an ILP objective that directly optimizes F score.
Another way to address these issues is to build a joint model that can model all these tasks at the same time. Eger et al. (2017) proposed two different frameworks for joint modelling the full argument mining task and has been influential. Figure 2 showed an illustration of these two formulations. They first framed the task as sequence tagging. For example, they used S&G2017a data for experiments and in this case, each token's label space would include 1) whether this token is a begin or continue of an argument component or it is non-argumentative); 2) the type of the component to which the token belongs; 3) distance between the corresponding component and the component it relates to; 4) the relation type between the two related components. This way they can use offthe-shelf taggers to solve all four sub-tasks at once. They also framed the task as dependency parsing. When framed as dependency parsing, the text is represented as directed trees where each token has a labelled head so that argument component relation information can be encoded. They further labelled these edges with tokens' component types and relation types so that argument trees can be converted to quasi-dependency trees. Finally, they adapted a joint neural model designed for entity detection and relation extraction (Miwa and Bansal, 2016). In this case, they modelled argument components as entities and argument relations as semantic relations.

Unsupervised learning
Aside from the supervised learning work described above, Persing and Ng (2020) proposed an unsupervised method for argument mining. The key for their work is to use heuristics for bootstrapping a small set of labels and then train the model in a self-training fashion.

Experiments
The use of Transformer-based models has been dominating in other natural language processing tasks but is still in its infant stage in this field. Therefore, we experiment with vanilla BERT models on three representative datasets, hoping to facilitate research to that end. The three datasets cover argumentation structure parsing and argument quality assessment. Specifically, we used S&G2017a dataset, which is the benchmark for parsing essay argumentation structure. Besides, we used S&G2017b dataset, which assess argument quality from a logic aspect (Wachsmuth et al., 2017) by annotating whether a piece of argument is sufficiently supported. We also used P&N2015 dataset which assess argument quality from a rhetoric aspect (Wachsmuth et al., 2017)    for these datasets can be found in Table 3, 4, 5 and 6.

Implementation details
S&G2017a For S&G2017a dataset, we followed Eger et al. (2017)    everything else the same as S&G2017a.
P&N2015 For P&N2015 dataset, we used 800 essays for training, 100 essays for validation and 100 essays for testing. Additionally, we ran crossprompt experiments by randomly selecting 247 essays from prompt 1 for training, 31 essays for validation, and rest of prompt 1 essays as well as essays from other prompts for testing. We also used one dropout layer and one linear layer on top of the base BERT model. For objective function, we used mean square loss. Besides, we set patience to 5 and dropout rate to 0.1 and kept everything else the same as S&G2017a.

Metrics and comparison methods
S&G2017a Similar to most work, we followed Persing and Ng (2016) for evaluation. They defined F1 score as F1 = 2T P 2T P +F P +F N , where TP stands for true positive, FP stands for false positive and FN stands for false negative. They also define 'level α matching' (Eger et al., 2017): for α% level match, the predicted component span and ground truth share at least α% of their tokens. For comparison methods, we chose the LSMT-ER model from Eger et al. (2017), which is a common baseline. In addition, we compared with the BiPAM model from Ye and Teufel (2021), which is a BERTenhanced biaffine dependency parser (Dozat and Manning, 2018).
S&G2017b We used macro F1 and accuracy scores for evaluation, following Stab and Gurevych (2017b). As for comparison methods, we chose the best performing CNN model in their work and human upper bound.
P&N2015 We used mean absolute error (MAE) and mean square error (MSE) for evaluation. We compared the baseline with a model developed by Wachsmuth et al. (2016), which is the best performing model to our knowledge.

Results and discussion
S&G2017a The experiment results for this argument mining task are shown in Table 7. The vanilla BERT did not outperform state-of-the-art methods but are close. The BiPAM model, which is a BERTenhanced dependency parser, performs the best. These demonstrate the power of Transformer-based models for the argument mining task. Additionally, LSTM-ER has 1) designed separate module for handling components and relations; 2) encoded explicit syntax information through syntactic parser. BiPAM has carefully designed the argument relation representation so that it can benefit from state-of-the-art dependency parsers. These considerations can be used for further unlocking the potential of Transformer-based models.
S&G2017b The experiment results for sufficiency recognition are shown in Table 8. We can see that the vanilla BERT model outperforms the previous best performing CNN model by a large margin. This is expected as the BERT model has been pretrained on a huge amount of text. It is surprising that the vanilla BERT model already achieved near-human performance. Therefore, we foresee that Transformer-based model can equal or even surpass human upper bound on sufficiency recognition in the near future.
P&N2015 The experiment results for argument strength scoring are shown in Table 9. On this dataset, the vanilla BERT model did not outper-form previous methods. There are two possible reasons for explaining it. First, the average token in an single essay exceed the maximum length(512) supported by vanilla BERT model. This results in part of the essay being truncated. From the data statistics, we can know that at least 25% of the essay is being truncated, resulting in large information loss. Second, both Persing and Ng (2015) and Wachsmuth et al. (2016)  To further gauge the generalization ability of Transformer-based models, we ran the vanilla BERT in a cross-prompt setting and the results are shown in 10. Being able to generalize across prompt is valuable in practice because it is expensive to collect data for each new prompt. Recall that the model is trained and validated on essays of prompt 1. From 3, we can see that the MSE and MAE remains similar across prompts except for prompt 3 and prompt 5. We took a look at these two prompts and found that they are much more abstract and provide less concrete context than the other prompts. 1 We hypothesize it to be the performance drop on prompt 3 and prompt 5 and thus believe that BERT can generalize reasonably well.
Overall, the vanilla BERT models perform comparably to previous methods across datasets. We believe this can be a good baseline model for future research in the community.   Figure 3: Line chart based on Table 10 5 Conclusion and Future Directions In this paper we have organize existing works around tasks, data and methods regarding automated evaluation for student argumentative writing. We provided a baseline to the community by experimenting with BERT models on benchmark datasets. Now the question would be: what is important for the future? Towards this end, we identified several directions we deem interesting. First of all, there is a lack of emphasis on language diversity in current datasets. This concerns two aspect: language of the essays and language backgrounds of the authors. For language of the essays, most datasets are annotated essays written in English. Horbach et al. (2017) and Wambsganss et al. (2020) annotated essays written in German and these two datasets are the only ones not written in English to our knowledge. Intuitively speaking, characteristics such as overall argumentation structure would vary across essays written in different languages. Besides, annotators for essays written in different languages may have different criteria when assessing the rhetoric quality of an essay. As for authors' language backgrounds, current datasets either do not take into account authors' language levels or use essays written by proficient writers (e.g. college-level writing). 2 Putra et al. (2021a) and Alhindi and Ghosh (2021) are the only 2 Although P&N2015 dataset annotated English learner's texts, most of the authors' native languages belong to Indo-European languages and have received at least 6 or more years' English education accessible datasets that target authors whose interested language skills are not proficient yet, to our knowledge. Specifically, Putra et al. (2021a) annotated essays written by English learners from various Asian countries. They discarded already well-written essays and only reserved the ones with intermediate quality. Alhindi and Ghosh (2021) annotated argumentative essays by middle school students and found these essays less structured and thus more challenging. We believe collecting more diverse datasets would be valuable because it will not only expand the impact of argumentative writing support systems, but also pose more challenging research problems.
Second, adoption of Transformer-based models is still in the infant stage despite some recent works have used Transformer-based architectures. As described in 4, we have built the simplest form of BERT models and demonstrated that they have comparable performance to previous state-of-theart methods. In addition, Ghosh et al. (2020) and Alhindi and Ghosh (2021) showed that performance of BERT can be greatly improved by continued pre-training on unlabelled essay datasets or architectural design that takes account of data characteristics. Therefore, we believe there is still huge potential for Transformer-based models.
Third, there is no research regarding generalization in this field. We are aware of relevant research works in the AES area (Jin et al. 2018, Cao et al. 2020, Ridley et al. 2021 but not for systems that care about argument-related attributes. Not being able to generalize across prompts remains a major bottleneck in AES (Woods et al., 2017) and we believe this can also be a main obstacle for deploying automated evaluation systems specifically for argumentative essays. This is because it is costly to collect data for every prompt. At the same time, learners usually need to practice writing over a large amount of prompts and get feedback before seeing significant improvement.