{'fraction_non_alphanumeric': 0.028347946115896833, 'fraction_numerical': 0.009585644538738785, 'mean_word_length': 5.396664486592544, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Design and evaluation of an ontology-based tool for generating multiple-choice questions

Purpose – The recent rise in online knowledge repositories and use of formalism for structuring knowledge, such as ontologies, has provided necessary conditions for the emergence of tools for generating knowledge assessment. These tools can be used in a context of interactive computer-assisted assessment (CAA) to provide a costeffective solution for prompt feedback and increased learner’s engagement. The purpose of this paper is to describe and evaluate a tool developed by the authors, which generates test questions from an arbitrary domain ontology, based on sound pedagogical principles encapsulated in Bloom’s taxonomy. Design/methodology/approach – This paper uses design science as a framework for presenting the research. A total of 5,230 questions were generated from 90 different ontologies and 81 randomly selected questions were evaluated by 8 CAA experts. Data were analysed using descriptive statistics and Kruskal– Wallis test for nonparametric analysis of variance. Findings – In total, 69 per cent of generated questions were found to be useable for tests and 33 per cent to be of medium to high difficulty. Significant differences in quality of generated questions were found across different ontologies, strategies for generating distractors and Bloom’s question levels: the questions testing application of knowledge and the questions using semantic strategies were perceived to be of the highest quality. Originality/value – The paper extends the current work in the area of automated test generation in three important directions: it introduces an open-source, web-based tool available to other researchers for experimentation purposes; it recommends practical guidelines for development of similar tools; and it proposes a set of criteria and standard format for future evaluation of similar systems.


Introduction
Objective tests comprise a set of questions with predetermined correct answers. For instance, such a question might ask a user to select an answer from a set of choices, as in a multiple-choice question, to fill in the blank spaces in a paragraph, or to provide a short numeric or textual response. The finite, deterministic nature of the objective tests makes them a good candidate for computer-assisted assessment (CAA), and enables many important elements of a good teaching practice such as encouraging active learning, emphasising time on task, respecting diverse ways of learning and above all, providing immediate feedback to the learner (Chickering and Gamson, 1987).
The impact of assessment feedback on quality and effectiveness of learning has been one of the major topics in education research in the last two decades. The evidence from the research suggests that prompt feedback has significant positive influence on learner's overall experience, including the engagement, the motivation, the quality of learning and ultimately the improved learning performance (Draper 2009;Govindasamy, 2001;Nicol, 2007). Outside the educational sector, diagnostic tests have been used by business and government's certification and accreditation bodies. Some of the large-scale CAA technology providers supply test platforms to millions of enterprise users worldwide.
Despite the benefits, the issues such as the availability of reusable question banks (Littlejohn, 2003) and the time required to write good questions are preventing a wider adoption of this form of assessment (Sidick, Barrett and Doverspike, 1994). According to Collins (2006) professional test-writers spend one hour in average per test item. The most popular form of objective test items, the multiple-choice question or MCQ consists of the question text (the stem), one correct answer (the key) and a small number of incorrect options (the distractors). Unlike single-response questions, multiple-response questions include more than one key. The most time-consuming part in an MCQ construction is selecting plausible distractors which could distinguish between different levels of knowledge (Mitkov et al., 2009). The automated and semiautomated means for generating distractors offer a possible solution for this problem.
The critics of objective tests (Biggs, 1999;Paxton, 2001;McKenna, 2019) highlight the issues such as overemphasis on factual knowledge, question design bias and guessing. These issues could start to be addressed by introducing the tools for generating questions which will incorporate specific pedagogical guidelines for assessing different knowledge levels while reducing the (human) bias in the process of the test design. Despite the issues, the critics agree that well-designed MCQs can be very useful in formative assessment "as a means of providing interactive exploration and instant feedback" (McKenna, 2019).
Generating questions from text corpora has been a subject of research and experimentation in areas such as natural language processing (NLP) and patternmatching. Mitkov et al. (2009) point to the computational difficulties related to the NLP techniques, and conclude that these processing-heavy algorithms, which are based either on the grammar-matching systems or statistically adequate web searches have not yet produced usable systems. The main issue is that these techniques are lacking the "semantic" information for generating plausible distractors.
A recent proliferation of internet-based knowledge repositories, such as Wikipedia, WordNet, Google Knowledge Graphs and others, has created conditions for development of tools and techniques for computer-generated questions based on the semantics of the underlying concepts. This is a result of the availability of the metalanguage used for structuring the knowledge in these repositories. For example, Wikipedia's categories, disambiguation links, headings such as "See also", and more specific headings for the articles from the same categories, are providing necessary semantic information to enable programmable searches and matches.
Unlike Wikipedia's semi-and loosely-structured knowledge, ontologies provide a more formal means for representing descriptive, non-procedural knowledge based on a simple meta-language borrowed from computer science. The knowledge in ontologies is organised in a hierarchy of classes, individuals and their properties, which are used to denote the knowledge domain concepts, their instances and their properties respectively. Since the inception of ontologies as the main format for representing knowledge in the semantic web (Berners-Lee, Hendler, and Lassila, 2001) many domain experts and organisations have engaged in their development, particularly in biomedical sciences (see for example, https://bioportal.bioontology.org/) but also in other fields such as business, linguistic and engineering. In education, ontologies have been used to formalise instructional processes and learning designs (Knight, Gaševic & Richards, 2006), and to support classification of learning materials (Valaski, Reinehr, and Malucelli, 2017). The role of ontologies in designing learning assessment has been less studied and only recently are techniques for ontologybased assessment starting to emerge. In the follow-up section we provide a review of the main work in this area.
The main application area of this paper is automatic generation of MCQ tests from arbitrary domain ontologies. The paper uses Design Science Research (DSR) approach to extends the current research in this area in three important directions: (i) it introduces an open-source, web-based experimental tool for generating questions from ontologies and from question stem templates which are based on Bloom's taxonomy (1956) ; (ii) it recommends practical guidelines for development of similar tools, based on the evaluations of generated questions (iii) it compares the tool with the similar systems and proposes a set of criteria for future evaluation of test generation tools.
The use of (the original) Bloom's taxonomy as opposed to other alternative learning taxonomies such as Gagne (1965) Anderson et al. (2001), or Marzano (2001, to mention a few, is justified as follows: (i) it is widely used by MCQ developers to develop questions that test knowledge recall, comprehension, application, analysis, synthesis and evaluation (Conole and Warburton, 2005;Nicol, 2007); (ii) it is simple, easy to understand, and it offers a variety of synonyms useful for creating question templates for different levels of the taxonomy (Krathwohl, 2002). The revised Bloom's taxonomy (Anderson et al. 2001) overcomes the limitations of the original taxonomy through: (i) separating knowledge and cognitive process dimensions, (ii) relaxing a strict hierarchy requirement by allowing overlapping levels and (iii) making a more explicit distinction between the synthesis (creation) and the evaluation levels (Krathwohl, 2002). However, none of these improvements are directly relevant to the objectives of this paper i.e. (i) the type of knowledge that can be encoded in ontologies is conceptual knowledge (ii) the tool does not impose a strictly linear progression through the levels of the learning taxonomy, and (iii) the question templates do not include synthesis and evaluation levels. Most importantly, the underlying pedagogical framework embedded in the question templates can be changed with the introduction of different templates, thus allowing other learning taxonomies to be used.
The main contributions that the paper makes to the practice is that it enables a rapid production of questions of different difficulties, that can be used in a context of intelligent tutoring systems, virtual-learning environments, MOOCs and large-scale assessment systems.
The paper is organised according to Gregor and Hevner's (2013) recommendations for presenting the DSR research, and

Related Work
In this section, a historical overview of the related work is presented, including a comparison of experimental systems and techniques for computer-generated objective tests (Table 1). Fischer (2001) was the first author to explore the use of concept hierarchies in generating test questions. He examined the ways in automating question generation for a technology course within an adaptive hypermedia learning environment (Multibook). The questions were based on "part-of" and "application-of" relations between the subject domain concepts and the distractors were extracted from the "sibling" concepts. The experimental validation of the results showed that in most cases the students were not able to distinguish between the question generated manually and by the system, but whenever more than one super-concept relation was traversed to generate distractors i.e. when the semantic similarity between correct and wrong answers was decreased, the students were able to identify the difference. The idea of using the distance between the nodes in the knowledge graph (the semantic similarity) to model the difficulty of distractors has been used extensively in the CTT & IRT Bloom and Krathwohl (1956) subsequent research in this area. The OntAWare adaptive learning tool developed by Holohan et al. (2005) supported generation of simple learning objects such as slide shows and objective tests using related and unrelated concepts from an input ontology. The proximity of the concepts in the hierarchy was again proposed for generating challenging questions. OntoAWare focused on adaptivity and personalisation, and less so on the implementation of the question generator. Their work was further extended in 2006 to include dynamic problem generation from the relational databases and in from of database queries. Mitkov, Le, and Karamanis (2006) used NLP techniques combined with WordNet definitions for generating multiple-choice tests, and then applied manual post-editing to improve the quality of the questions. Their starting premises were that the questions should focus on the key concepts, and that the distractors should be as semantically close as possible to the answer. The latter they named as "the distractor selection premise". Their system identified important terms in the text and transformed the comprising sentences into question stems using three rules: find subject, find type ("kind-of" question), and find object. They then retrieved the hyponyms of the correct answer from WordNet to construct the distractors. They were the first authors to perform a comprehensive evaluation of the generated questions (for details see Table  7) which showed that the semi-computerised construction of questions is at least three times more efficient than purely manual production. They also validated the "distractor selection premise" using methods from the Classical Test Theory (CTT). The item difficulty, measured as a percentage of students who answered the question correctly, was lower than for the manually generated tests, while the discriminatory power was higher. In the follow-up work (2009) they performed an empirical evaluation of various similarity measures for generating distractors, but did not find any statistically significant differences.
The work presented here builds on the work of Papasalouros, Kotis, and Kanaris (2008). Their focus was on distractor generation, given a correct answer from an arbitrary domain ontology. They refined the "distractor selection premise" by introducing eleven ontology-based strategies for constructing distractors. The strategies were based on the basic meta-ontology relations between classes, individuals and properties. For example, if the correct answer is "a is an instance of A", one strategy was to offer the instance b of the class B as a distractor, where B was a specialisation (hyponym, "subclass" or "kind") of A. They did not consider stem generation, and used a generic text ("Choose the correct sentence:") instead. After evaluating a selection of questions generated from five purposefully developed ontologies, they concluded that the input ontologies should adhere to certain conventions to generate syntactically correct questions; for example, property name must be written as a verb or a verb-like phrase. They also found that the propertybased strategies, compared to class-and relation-based, may produce more but less syntactically correct questions. The main limitations of their work are related to the lack of pedagogical foundation and the use of purposely-constructed ontology for the evaluation. They later (2011) extended the work to include multimedia ontologies.
In the previous work the authors (2009) optimised the strategies for generating distractors introduced by Papasalouros, Kotis, & Kanaris (2008) and implemented a test generator as a plugin for the Protégé ontology editor, using simple question types such as "what-is", "is-a", "is-not-a"; they also introduced the question difficulty as a configurable parameter which was implemented according to the "distractor selection premise". That work was extended in 2011, by adding new strategies that made use of ontology annotations, and by introducing templates for generating questions based on Bloom's taxonomy of learning objectives (Bloom and Krathwohl, 1956).
Alsubait, Parsia, and Sattler (2012) focused on generating analogy questions where the stem is a pair of ontology concepts, and the goal is to identify the most analogous pair of words amongst the given options. Their algorithm was based on the "distractor selection premise" and they evaluated it using an automatic "solver" method. However, the choice of input ontologies did not allow for a realistic assessment as the ontologies were either very complex (Gene ontology) or very simple (Pizza, People and Pets).
Al-Yahya (2014) considered simple questions of "kind-of" and "fill-in-the-blank" types, but with no specific strategies for generating distractors. The system was tested on randomly selected questions from two ontologies. The results of the evaluation indicated that the main problem was the quality of distractors. Vinu and Kumar (2015) made use of axiomatised knowledge on the concepts from an ontology. They introduced a measure for question difficulty based on the "distractor selection premise". For the experimentation purposes, they considered four ontologies and restricted the evaluation to two question types: "Choose correct answer" and analogy questions. They found that the difficulty of generated questions judged by the experts did not always coincide with the proposed difficulty. The difference was due to the clues in the correct answers. The problem of generating syntactically correct question items was only partially tackled.
Most recently Demaidi, Gaber, and Filer (2017) used the strategies developed by the authors (2011) to evaluate the generated questions using the statistical analysis of the test performance results; they used the methods provided by the CTT and Item Response Theory (IRT), to analyse the results of three tests. Unlike in previous approaches, a sample of questions were selected and syntactically checked before the evaluation. Their results showed that the test could effectively discriminate between high and low performing students. They also showed that the strategies for generating discriminators and the Bloom's taxonomy level (1956) both affect the question difficulty and the discriminatory power: the questions at the knowledge level were easier than the other types, but no significant difference in difficulty was found in the questions testing application, comprehension and analysis. They did not consider the syntactic correctness, as all questions included in the evaluation were previously edited.
While all the reviewed approaches have some important novel elements, only a few provide empirical evidence on the question utility and the difficulty, and none on the relevance and textual quality of the questions. In this paper, we address this gap by providing an evaluation of generated questions according to all four criteria, and comparing the evaluations with those of other related systems discussed in Section 2. To simplify the presentation, we use the abbreviation "Onto2MCQ" to denote a class of systems or approaches for generating MCQs from arbitrary domain ontologies. Following the guidance by Thuan et al. (2019) for construction of design science research questions, we formulate the following research questions that this paper will try to answer: RQ1. Can an Onto2MCQ be implemented that uses a combination of syntactic and semantic question generation strategies and question templates based on a specific learning taxonomy? RQ2. What evaluation measures can be used to assess Onto2MCQs ?
RQ3. In what ways can Onto2MCQ be improved?

Design Science Research (DSR) is a methodology commonly employed in Information
System research and other design-oriented disciplines. The methodology involves construction of socio-technical artefacts, such as software, algorithms, procedures, principles or methods, and extraction of the knowledge resulting from the novelty of the artefact or its development process. So, in DSR design becomes a research method for the knowledge construction. The method follows the stages of a standard design process: (i) awareness of the problem (ii) suggestions for the problem solution drawn from the existing knowledge base and functional requirements (iii) development of the artefact (iv) evaluation of the artefact, where the last three stages can be iterated. In this process, the knowledge flows in two directions: from the existing knowledge base to the "suggestions", and from the "development" and "evaluation" to the knowledge base.
This paper uses the "pragmatic-design" variant of the DSR (Gregor and Hevner, 2013) where the emphasis is not on theoretical contribution but on development and evaluation of a technical artefact. The artefact under the constructions is a web-based Onto2MCQ. The research covers the first two levels in the DSR contribution type hierarchy (Gregor and Hevner, 2013), as it not only describes and evaluates a software tool (RQ1&RQ2) , but it also identifies the relevant technological rules and design principles that can be used for similar systems (RQ3).
The evaluation of the artifact was based on a mixed-method comprising quantitative and qualitative data obtained from the CAA subject experts combined with the usage data obtained from the server platform and the evidence of the impact in the field. The quantitative data were analysed using descriptive statistics and nonparametric tests; and the qualitative data were collected through semi-structured interviews with the evaluators who represented the two ends of the evaluation spectrum (positive and negative opinions).

The Onto2MCQ tool
The Onto2MCQ experimental tool (http://www.opensemcq.org) was developed as a standalone web application using the open-source and free development tools such as Java, PHP, UserCake, MySQL, Apache Jena, Apache PDFBox. The code for the question generation engine was made available as an open source in the GoogleCode repository. The tool, publicly hosted in one of the authors institutions, provides a fullyfunctional standalone web-based experimental system for generating MCQs (singleresponse format) from ontologies which (i) uses a combination of semantics and annotations-based strategies for generating distractors (ii) employs templates for generating questions stems corresponding to different knowledge levels in Bloom's taxonomy (1956), including factual knowledge, comprehensions, application and analysis.
The inclusion of annotations i.e. textual descriptions of the ontology concepts, individuals, and properties to the algorithm was justified by an increasing trend of adding annotations to the ontologies by automatic or semi-automatic means (Reeve and Han, 2005); this has resulted in more ontologies being equipped with textual description. The annotations have enabled new question templates to be used in the process of test generation (e.g. "What is the paragraph describing …?", "Fill-in the missing words in the text …", "What is the correct definition of …?") and consequently, production of more diverse tests.
Other question templates used for generating tests were linked to the strategies previously defined in Papasalouros, Kotis, & Kanaris (2008), and the authors (2011) as illustrated in the example in Figure 1. In the example, the distractors are selected from the instances of the "parent" class (Payment method) and the "sibling" class (MoneyTransfer).
According to the classification from Papasalouros, Kanaris, and Kotis (2008), strategies 3 and 7 are property and sub-concept strategies, while the rest are class-based strategies. The techniques used for generating syntactically similar items include common string-similarity algorithms provided by the open-source Java library SecondString (Cohen, Ravikumar and Fienber, 2003).
Linking the question templates to the Bloom's knowledge levels (1956) has enabled qualifying the difficulty of the questions according to the increasing educational objectives: factual knowledge (what is the correct definition of…?), understanding (which of the following response pairs relates in the same way as … ?), application (which of the following examples demonstrates …?), analysis (which of the following concepts is a generalisation of …?). Various strategies using distractor-selectionpremise were employed for generating questions at different levels. The screenshots in Figure 2 illustrate the views for creating and viewing the generated tests.

Evaluation
The Onto2MCQ tool was evaluated using the criteria such as quality, difficulty, validity and utility of the generated questions. In this paper, the focus is on the evaluation of the test generation engine, rather than usability of the user interface or efficiency of the tool, which will be considered in the follow-up work.

Evaluation Criteria
The evaluation criteria were chosen based on the recommendations from Gregor and Hevner (2013): (i) the quality criteria assess the quality of the generated test-items,

Figure 2. Creating test and viewing test results
including the syntactic correctness, the choice of distractors and the knowledge level i.e. the question difficulty; (ii) the validity means that the tool works and does what it is meant to do. In the context of this paper the validity was assessed through the relevancy of the question to the underlying ontology concepts (iii) the utility criteria assess whether the achievement of the goals has value outside of the development environment, i.e. if the question could be used without any changes in a real assessment. A CAA expert from one of the author's institution reviewed an interim version of the evaluation questionnaire, and further improvements were made based on the recommendations. The final version of the evaluation questionnaire is included in the Appendix A. To facilitate the evaluation process, the tool was extended with the evaluation capabilities. Figure 3 shows an example of a generated question and its evaluation. A few more examples of generated questions across various evaluation ratings are shown in Appendix B.

Evaluation Ontologies
The selection of the test ontologies from the initial set of 90 ontologies used for generating tests was based on the following criteria: academic subject knowledge domain, different subject domains covering both STEM and non-STEM disciplines, inclusion of annotations, inclusion of other ontology components (concepts, individuals, properties), non-flat concept hierarchy (at least 3 levels), contains at least 10 concepts.

Evaluation Questions
The test generation parameters were configured as follows: • Maximum number of questions per question type: 10 • Maximum number of distractors: 4 • Type of question in Bloom's taxonomy: knowledge, understanding, application, analysis.
• Strategies for distractors (seven strategies described in Section 4).
One test was generated for each of the four ontologies. The total number of questions available for the evaluation was 81, ranging from 19 (biochemistry) to 27 (economy). Thirty-eight questions (47%) were testing factual knowledge while the remaining 43 (53%) were questions related to the higher educational objectives. As expected, the number of generated questions for "comprehension" and "application" levels were directly proportional to the number of "properties" and "individuals" in the corresponding ontologies. Similarly, and as it was the case in the Music ontology, the lack of associations between the classes and individuals have resulted in no "application" questions been generated. Naturally, the structure of the ontology and the number of the associations between different ontology components has determined the diversity and the size of the generated tests.
Regarding the evaluation ontologies, Economy ontology produced more diverse and challenging questions, as 17(63%) of its questions are testing higher educational objectives. Biochemistry and Law ontologies generated less diverse tests (only two types of questions) and less challenging questions: 53% of question in both cases are testing only factual knowledge.
The summary of the counts and percentages of the test questions is shown in Table  3.

Participants
Following an email invitation to fifteen colleagues from one of the authors' University, eight completed the evaluation. All participants were university lecturers who had expertise and interest in the CAA application area. Each participant was asked to evaluate one test that was the closest to their subject expertise and one additional test outside of their expertise area. All participants completed the evaluation of at least one test and one participant completed two evaluations. Two evaluators completed 34% and 45% of Economy and Music questions respectively while other tests were fully completed.
The academic subject areas of participants included business (2), law (1), IT & computer science (3), and life sciences (2). The age of participants was between 25 and 58, and the male to female ratio was 3:5.
The total number of evaluations was N=138, ranging between 14 (music) to 71 (economy). Fifty-nine evaluations (43%) were related to the questions testing factual knowledge while the remaining 79 (57%) were evaluations related to the higher educational objectives. The summary of the counts and percentages of evaluations is shown in Table 4.

Evaluation Results
The frequencies of the responses (0-4) across four evaluation criteria are shown in Table 5. The median values for all four criteria was 1, indicating the following tendencies in data: • The questions were easy to understand, despite syntactic or grammatical mistakes • The difficulty of the questions was low i.e. mainly testing factual knowledge • The questions were not directly useable, but they could serve as a basis for useable question • The questions relate to some extent to the concepts they are testing.    Table 6 includes the comparison of the medians and the corresponding inter-quartile ranges of the responses across different tests, levels in the Bloom's taxonomy (1956), strategies for generating distractors, subject areas and the gender of the evaluators. Kruskal-Wallis test was used for group comparisons; the test compares the medians of the groups, and it is adequate for comparing more than 2 groups when parametric assumptions are not met. The use of the non-parametric test is justified because the data considered here is not interval data. The resulting p-values are shown in the last row for each of the groups of the evaluations indicating significant differences in quality of questions across different ontologies, question types, strategies and evaluators. The evaluation was followed-up with unstructured interviews with two evaluators from the law and business subject areas, who assigned the highest and the lowest marks to the questions respectively. The interviews were focused around the topics of usability of the tool in the specific subject areas and the further improvements. The answers revealed the differences in the type of knowledge and assessment in the two domains. While the multiple-choice and concept-oriented questions were commonly used in Law subjects, they are less common in Marketing subjects where "thegeneralisation-of" questions where found particularly unsuitable. Improvements were suggested in the areas of clarity of questions, availability of the ontology views and the necessity of providing the explanation on why the answer is wrong. 5.5 Impact Since the early prototype was introduced in 2013 the tool was used by more than 1000 external visitors from more than 10 different countries with the average session duration of 5 minutes and an average bounce rate ( Figure 4); during that time, more than 90 different ontologies were uploaded to create 251 tests with 5230 questions. It should be noted that no advertisement has been done except through the published papers and conferences. Hence, the external visitors are probably researchers in the field resulting in a noticeable impact in the research community. The prototype and the method have been referenced in 50 publications, including books, journal and conference papers and the tool has been used for providing experimental support and data for work on 2 ongoing PhD dissertations.

Findings and Discussion
In this section, the main findings from the evaluation are discussed, the tool is compared to the similar systems across a range of evaluation criteria and areas of future research are recommended. The analysis of the results indicates that the textual quality of the generated questions varied across different types of questions, ontologies and strategies. The items testing the application of knowledge through identification of examples of specific concepts are shown to be easier to understand compared to the other question types; the higher-order question types such as analysis questions, were more difficult to understand. This could be attributed to the use of the syntactic similarity strategies for generating distractors in the former case, and to the artificial formulation of the question text in the latter example, which was driven by the ontology meta-language (e.g. the-generalisation-of questions).
The strategies based only on semantic similarity have generated questions that are perceived to be of a higher textual quality. This might be due to less text and thus, less syntactic errors in these questions.
The difficulty of the questions testing factual knowledge, comprehension and application was assessed to be matching the expected knowledge levels, while the analysis questions were evaluated to be easier than expected.
The myth that objective tests cannot test higher orders skills has resulted in a very few examples of these question types in practice. On contrary, the ontology graphs provide means not only for creating more difficult questions but also for generating question templates from the graph sub-components. The challenge is to make those templates accessible using the specific domain language. This approach could be supported with the machine learning techniques for improving the language quality.
Validity of the questions and the overall evaluation marks were significantly different across different strategies. The strategies using only "distractor selection premise" generated questions of higher validity and higher overall value indicating that overall these questions were valued more by the evaluators.
Not all questions were assessed to be directly useable, and many are requiring syntactic improvements. The text included from the ontology includes annotations, as well as the names of the components (classes, individuals and properties) which are frequently using artificial (programming language) syntax and conventions. A preprocessing of the ontology text would therefore be required for any real-life use of the tool.
The question feedback (true/not true) is not always useful and more substantial feedback is required to support the learners and provide opportunities for "deep learning". With the approach presented in this paper, the feedback is already encoded in the strategy for generating the distractor. Further work is needed in translating these strategies to accessible language.
The quality of the ontologies considered for the evaluation has made an impact on the quality and difficulty of the questions. Therefore, in future work, real life ontologies should be considered as they are more likely to undergo quality assurance. This requires the performance improvements of the tool, or the techniques for splitting the large ontologies into smaller useable components.
Regarding the individual attitudes, the evaluators from the STEM disciplines were more critical of the quality of the questions than their non-STEM colleagues. This could be attributed to the more experience with the multiple-choice questions in the STEM disciplines, and subsequently, higher expectations from the experts from these disciplines.
The extreme difference is evaluations between the law and the business domain experts suggests the differences in the type of knowledge in these domains (rulebased vs. procedural, how-to knowledge) and subsequently different assessment requirements.
The current system is fully automated, and while automation reduces the cost of test development the issues that creates such as some of the questions lacking utility or comprehension, may incur costs that are prohibitive of deployment.
The comparison of the evaluations across different experimental systems, including the authors' is presented in Table 7, showing improvements in percentage of questions evaluated to be useable after small syntactic corrections (60%) and a number of methodology improvements such as: number of expert evaluators (8), variety of the evaluation criteria (4), total number of generated questions for the evaluation (90), number of ontologies used in the evaluation (4) , richness of the underlying ontologies with regard to the variety of components (concepts, individuals, annotations and properties). In addition to that the results demonstrate significant differences in evaluations across different strategies for generating distractors and different question types confirming similar findings by Demaidi et al. (2017).
The table also suggests a standard format (methodology) for reporting future

Conclusions
This paper uses a DSR approach to present and evaluate an experimental web-based Onto2MCQ tool that uses a combination of syntactic and semantic question generation strategies and question templates based on Bloom's taxonomy (RQ1). The Onto2MCQ tool addresses two important problems in the domain of intelligent tutoring systems, namely, providing prompt feedback to the learner, and offering a cost-effective assessment solution for the test provider. The tool was evaluated according to the quality, difficulty, validity and utility of the generated questions and the findings were compared to the related work in this area indicating improvements in many of the evaluation categories (RQ2). The limitations of the tool and of the evaluation method have been recognised; and practical guidelines for improvements and future research areas have been recommended (RQ3).
The recent rise in development of online data vocabularies, increased potential for large-scale experimentation in massive open online courses, and use of machinelearning techniques are opening new possibilities for advances in development of Onto2MCQ systems. Lowering the cost of question creation through increased automation can reduce barriers for a wider adoption of CAA and provide opportunities for higher learner's engagement.
This paper demonstrates that the subject of generating objective tests from ontologies is maturing and that new hybrid approaches and standardised evaluation methods are required for advancing this important area.  A trade-off needs to be considered between (i) using ontology meta-language capabilities for generating questions and (ii) compatibility of the generated question types with the standards in the specific domain. R2

References
The "distractor selection premise" could be used to increase difficulty of generated questions at all levels of the taxonomy.

R3
The ontology selected for generating questions should be pre-processed to improve the language where: (i) the text of annotations should be checked for syntactic correctness and (ii) all ontology components should be provided with user-friendly labels.

R4
The strategies for generating distractors could be translated into natural language explanations that can be used for generating question feedback. R5 A trade-off between generic Ont2MCQ vs. subject-specific Onto2MCQ needs to be considered. R6 A trade-off between full automation and the cost of language improvements needs to be considered. R7 The Ont2MCQ evaluation methodology and reporting should be standardised.