{'fraction_non_alphanumeric': 0.025620145326985717, 'fraction_numerical': 0.011525933350037584, 'mean_word_length': 4.961538461538462, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Indexing student essays paragraphs using LSA over an integrated ontological space

A full understanding of text is out of reach of current human language technology. However, a shallow Natural Language Processing (NLP) approach can be used to provide automated help in the evaluation of essays. The main idea of this paper is that Latent Semantic Indexing (LSA) can be used in conjunction with ontologies and First order Logic (FOL) to locate segments relevant to a question in a student essay. Our test bed, in a first instance, is a set of ontologies such the AKT reference ontology (describing academic life), Newspaper and a Koala ontology (concerning koalas' habitat).


Introduction
This paper describes a novel methodology aiming to support evaluators during the essay marking process. This approach allows measuring semantic similarity between structured (i.e. ontology and binary relations derived from the essay question) and unstructured (i.e. text processed as a bag of words) information by means of Latent Semantic Analysis (LSA) and the cosine similarity measure (Deewerster et al., 1990).
Previous studies (Foltz et al., 1998;Wiemer-Hastings and Graesser, 1999) have used LSA to measure text coherence and comprehension by comparing units of text (i.e. sentences, terms or paragraphs) to determine how semantically related they are. The work presented in this paper is based on the use of "pseudo" documents: these are temporary documents containing a description of knowledge entities extracted from available domain ontologies (i.e. ontological relations). Both pseudo documents and paragraphs in student essays are represented as vectors. Essay paragraphs are indexed according to a measure of semantic similarity (called cosine similarity). The ontological space acts as a mediated schema, a set of virtual relations among knowledge entities related by their degree of similarity. A new knowledge entity can be added in this space and automatically a similarity measure is calculated for all the entities within the space.

Motivation and Context
The main motivation for this work derives from a need for semantics in essay evaluation, whether by a tutor or by the student author in the process of writing. Page (Page, 1968) makes a useful distinction between marking for syntax (i.e. linguistic style) and for content (subject matter) which we will use in our outline. Based on this distinction, four main approaches to essay assessment have been reported (Williams, 2001). Early systems such as PEG (Page, 1966) relied mainly on syntactic and linguistic features and required a sample of the essays to be marked by a number of human judges. E-rater (Burstein et al., 1998) uses a combination of statistical and natural language processing techniques for the purpose of extracting linguistic features of the essays to be graded. Again, the essays are evaluated against a set of human-graded essays acting as a benchmark. In the LSA method of essay grading, an LSA space is constructed based on domain specific material and the student essays. LSA grading performance is about as reliable as human graders (Foltz, 1996). Text categorisation (Larkey, 1998) also requires a database of graded essays, so that new essays can be categorised in relation to them.
In short, the approaches seen so far 1 have either concentrated on syntactic and linguistic features or used domain knowledge in the form of keywords and documents about the domain. What we are proposing in this paper is that a further distinction should be made between using implicit (keywords, documents) and explicit content representations (see Fig.1, our contribution is marked in bold). We then argue the case for adding explicit domain knowledge in the form of domain ontologies. In particular, we merge ontologies, LSA and FOL. An advantage of this approach is that it does not require a corpus of graded essays, except for validation. This feature enables tutors (or students in need of feedback) to evaluate essays on particular topics even when there are no pre-scored essay examples available. Effectively, this capability may reduce the overall time required to prepare a reliable evaluation scheme for a new essay question.

LSA and the Cosine Similarity
In the vector space model, a term-to-document matrix is built in which the entries are weighted frequencies of pre-processed terms occurring in a collection of documents. Dimension reduction methods (such as LSA), when applied to the semantic vector space model, improve information retrieval, information filtering and word sense disambiguation. The reduction in dimensions reduces the noise in text categorisation, reduces the computational complexity of cluster creation, and produces the best statistical approximation to the original vector space model. Likelihood curves characterise with a quantity the level of significance of the reduced model dimensions. Also, the significance of each dimension follows a Zipf distribution (Li, 1992) indicating that the reduced model dimensions represent latent concepts (Ding, 1999). The dimensions in the reduced vector space model can be compared measuring semantic similarity between each of them by means of the cosine similarity. The cosine of the angle between two vectors is defined as the inner product between the vectors v and w divided by the product of the length of the two vectors.

Indexing Essays Paragraphs
An index of relations within the ontologies related to the semantic space is obtained for each binary relation derived from the essay question. Then a subset containing the higher ranked relations is selected and the similarity between each of the relations in the subset and all the documents containing essays paragraphs is also calculated by applying LSA. Finally, an average similarity value is obtained for the paragraph over the number of relations in the subset.

An Ontology Integration Method to Build the Semantic Space
A collection of "pseudo" documents is created for each of the classes within the ontologies describing the domains tackled in the essay. The ontologies are described quantitatively using probabilistic knowledge (Florescu et al., 1997).
Each of these documents contains information (name, properties and relations) about a class. The documents are represented by a vector space model (Salton et al., 1971) where each column in the term-to-document matrix represents the ontological classes and the rows represent terms occurring in the pseudo documents describing those knowledge entities.
Relations within the available ontologies are also represented by a vector space model where the columns in the term-to-document matrix are a combination of two or more vectors from the termto-document matrix representing classes. Each column represents the relation held between the combined classes. A new column representing the binary relation derived from the essay question is added to the new matrix: this new column contains the weighted frequencies of terms appearing as arguments within the relation. For each essay question, one or more binary relations are derived through parsing. For instance: given the query "Do koalas live in the jungle?" the binary relation is live_in (koala, jungle). In the case of this example, the vector representing the question contains a frequency of one in the rows corresponding to the terms koala and jungle.
LSA is applied to the term-to-document matrix representing the ontological relations, the vector space model is reduced and the cosine similarity is calculated to obtain the semantic similarity between the vectors of the reduced space model. For each column, a ranking of similarity with the rest of the columns will be obtained.

Weighting Scheme
Given the term-to-document matrix containing a frequency f ij , the occurrence of a term in all the pseudo documents j is weighted to obtain matrix. The entries of the matrix are defined as, where, l ij is the local weight for term i in the pseudo document j, g j is the global weight for term i in the collection and d ij is a normalisation factor. Then, as defined by Guo and Berry (Guo and Berry, 2003),

Experiments on semantic similarity
In order to evaluate how well LSA captures similarity, this section will describe three preliminary experiments for measuring semantic similarity between knowledge entities (i.e. binary relations and classes) of three different ontologies, the Aktive Portal Ontology (APO), the Koala Ontology (KO) and the Newspaper Ontology (NO).

Experiment 1
The aim of this experiment is to evaluate how well LSA captures similarity between classes that belong to different ontologies. Eight classes have been selected randomly from within three ontologies and described in "pseudo" documents. The words included in each of the documents correspond to the names of the classes and slots related to the class described. The terms have been stemmed and stop words deleted before applying LSA to the term-to-document matrix built using the weighted frequencies of the term occurring within the eight documents describing the classes. Terms have been weighted according to the weighting scheme presented in section 2.1.1 with d j =1, the only difference being that terms corresponding to classes names have been multiplied by two. The similarity measures for the eight classes are obtained (See Table 1) after applying LSA with a rank of two and the cosine similarity measure to the term-to-document matrix.
The results from this experiment show that, in terms of the cosine similarity measure, the class "Researcher" appears to be very similar to the class "Student" in a different ontology. The same results also show that the two classes "Newspaper" belonging to two different ontologies are very similar to each other.

Experiment 2
The aim of this experiment is to evaluate the ability of LSA to measure similarity between a predicate argument and different classes. The query is represented as an added column in the term-to-document matrix which already contains as columns the same documents representing the eight classes used in the first experiment. The column representing the query argument contains only one term corresponding to the name of one of the classes within the ontologies used in this experiment. The frequency of this term is the entry in the added column with a frequency of one multiplied by two as all the other terms representing names of classes. The results for the cosine similarity measure between the eight classes plus the query containing the term "student", "newspaper" and "animal" after applying LSA with a rank of four (see Table 2) indicate that the most similar classes for the query containing the term "student" are the following classes: "Student" from KO, "Researcher" from APO, and "Parent" from KO.

Table 2 -Semantic similarity between arguments and classes belonging to different ontologies
For the query containing the term "newspaper" the results shows that the most similar classes are "Newspaper" from APO, "Newspaper" from NO and "Standard Advertising" also from NO. Finally, for the query containing the term "animal", the most similar classes in order of similarity closeness are "Parent" from KO and "Koala" also from KO.
The results of this experiment indicate that LSI may be accurately used as a measure of similarity between a keyword representing a query predicate argument and a set of documents representing classes that belong to a set of different available ontologies.

Experiment 3
The aim of this experiment is to evaluate the cosine similarity measure as a measure of semantic similarity between binary relations derived from a question or query and relations held between two classes. This measure is based on the same methodology and procedures applied to both experiments described above. For this experiment, eighteen classes have been selected arbitrarily from the three available ontologies (see Table 3).
The binary relations held among the selected classes are represented as documents in a term-todocument matrix that is the union of the two pseudo documents describing the related classes. Following the same procedure as in the previous experiment, a new column representing the binary relation derived from a question is added to the matrix, but in this case it contains the terms describing the two arguments of the binary relation.

Table 3 -Ontological Binary Relations (OBR) used in Experiment 3
The cosine similarity between fifteen predicates and the available relations after applying LSA with a rank of four (see Table 4) show that, in eight of the fifteen cases, the similarity value is higher for the relations held between classes than between predicate arguments. In the rest of the cases, the similarity values are very close for two or more relations including the one held between classes that are the same as the predicate arguments. Another interesting observation is that, Question Binary Relation 3 (QBR3) has a cosine value more similar to Ontological Binary Relation 9 (OBR9), OBR3 and OBR4. In the case of QBR5, the cosine value is higher when measuring similarity with OBR11 and OBR12 than, for example, the cosine value when measuring similarity with OBR3 and OBR4. Similar results were obtained for QBR6 where, apart from OBR6, OBR9 has the cosine value closest to one. Other similar results are obtained for QBR11 and QBR12 where OBR5 is closer to a value of one than OBR7, OBR8 and OBR9. QBR 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000

Table 4 -Cosine similarity between the Question Binary Relations (QBR) and the Ontological Binary
Relations (OBR).
The results of this experiment indicate that the presented methodology is able to detect similarity between compact representations (binary relation arguments) and more expanded representations such as the pseudo documents representing the binary relations within the three available ontologies.

Experiments discussion
We expect that using LSA together with the cosine similarity measure, we will be able to pick up semantic similarity between the compacted and expanded representations of the binary relation and paragraphs from student essays. The main difference between our approach and other essay scoring approaches (e.g. The Intelligent Essay Assessor; Laundauer et al., 2000) where the scores are calibrated using LSA with pre-scored essay examples, is that our approach scores paragraphs using LSA and the cosine similarity with ontologies describing the essay domain. The experiment results in the previous sections validate our view showing that the cosine similarity may be used as a reliable score for semantic similarity between knowledge entities belonging to different data sources (i.e. terms, classes and binary relations).

Conclusion and Future Work
This paper introduces the idea of "explicit content" and its use in essay evaluation. The main contribution of the paper is then the idea that ontologies and First Order Logic (FOL) can be used together with LSA to locate segments relevant to a question in a student essay.
Our main interest is to provide help to tutors in grading and to students for feedback purposes. In fact, even outside the realms of grading, we believe that it will help annotate and rank paragraphs more relevant to queries. In our proposal, we went about doing this supplementing the widely-used LSA method with added semantics (ontologies) and First Order Logic (FOL). Our approach therefore attempts to bridge the gap between statistical and semantic approaches.
There is clearly a lot more work needed to make this technology work well enough for large-scale deployment. Further work may include a visualisation service that also provides a visualisation of annotation of segments relevant to the current question types around the lines of the work described in (Moreale and Vargas-Vera, 2003;Moreale and Vargas-Vera, 2004).