{'fraction_non_alphanumeric': 0.024859663191659984, 'fraction_numerical': 0.00040096230954290296, 'mean_word_length': 5.2375, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Assessing High-Order Skills with Partial Knowledge Evaluation: Lessons Learned from Using a Computer-based Proficiency Test of English for Academic Purposes

English proficiency tests (EPT) are required from students enrolled in Master and PhD programs in many non-English speaking countries, as the students must demonstrate their ability to understand and produce technical literature that is predominantly in English. Some universities and funding agencies in Brazil require the student to pass a general-purpose exam, such as TOEFL or IELTS. However, these exams do not evaluate students’ competence in terms of the demands of highly standardized research articles writ ten in English, which will be needed for the student to review the literature of his/her re-search area. In order to overcome this limitation we implemented a web-based, fully automated system for English Proficiency tests, named CAPTEAP, which checks the students’ ability in the use of English for academic purposes. In this paper, we describe how CAPTEAP assesses high-order skills of Bloom’s taxonomy, where students are assessed in their reception and familiarity with the schematic structure of scientific papers. CAPTEAP employs the resources of a case-based reasoning tool to assist non-English speakers in producing scientific papers. It comprises 4 modules, through which the students are evalu-ated with regard to their ability to: i) analyze the structure of a given section of a paper; ii) analyze rela-tionships among clauses signalled by discourse markers; iii) recognize conventions and rhetorical func-tions in English from scientific texts; iv) identify writing strategies for each component of a section. CAPTEAP has been applied in the official proficiency test required from MSc. students at the Computer Science Department at University of Sao Paulo, Brazil. In order to be fully automated, CAPTEAP con-tains multiple-choice, objective questions but allows a refi ned assessment of abilities by using scoring procedures based on the Admissible Probability Measurement (APM). The lessons learned from introducing this fully automated test are presented in connection with the following topics: the web-based system interface, the modules and cognitive levels of the tasks in the tests, and the scoring procedure. As for the EPT interface, we learned that in formal exams users are unlikely to tolerate system faults as the latter may affect their per-formance. Evaluation of interfaces by real users