{'fraction_non_alphanumeric': 0.021532846715328468, 'fraction_numerical': 0.0036496350364963502, 'mean_word_length': 5.63680387409201, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
EXPLORING THE DYNAMICS OF COMPLEX PROBLEM-SOLVING WITH ARTIFICIAL NEURAL NETWORK-BASED ASSESSMENT SYSTEMS1

Assessment of cognitive models developed by students in complex scientific disciplines ideally captures the progressive and dynamic nature of learning. We have developed a computer-based performance assessment system based on the production-system model of problem solving (Newell & Simon, 1972) in order to explore assessment of medical student problem-solving skills. We are able to track student data acquisition within groups of related concepts (concept domains) through a process called Òsearch path mapping,Ó which reveals differences between successfully executed performances and unsuccessful strategies. We were able to identify both changes in the overall strategic approach (i.e., selecting the correct concept domain) and refinements in problem-solving strategies (i.e., more efficient use of concepts within a domain). While search path mapping allows for the assessment of student strategies, it is also time-consuming and potentially subject to reviewer bias. We therefore automated the search path map analysis by using supervised back-propagation artificial neural networks (ANNs), trained with data from previous medical student performances. These artificial neural networks could discriminate strategy improvement as well as hypothesis utilization in problem solving. To provide an indicator of learning, we compared the number of hypotheses utilized by students on practice and examination problems in the same concept domain. We found that medical students used information more efficiently during the examination, resulting in fewer hypotheses generated. Although the overall utility of ANN-based analysis was seen, this supervised network did not identify certain specific cases determined by search path mapping to be more ÒexpertÓ than the training data set. Our ANN also misinterpreted rare cases of irrelevant concept usage. These studies demonstrate the utility of ANN-based assessments as an adjunct to traditional forms of assessment and suggest additional studies needed for this approach to become part of a comprehensive evaluation system. 1 The authors would like to thank Dr. Robert Allen for his critical review and Stanley Chen, John Stallings, and Peter Wang for their expert technical assistance. 2 Molecular Biology Institute, UCLA School of Medicine 3 Department of Microbiology and Immunology, UCLA School of Medicine 4ÊDepartment of Medicine, Division of Clinical Immunology and Allergy, UCLA School of Medicine 5ÊGraduate School of Education and Information Studies/National Center for Research on Evaluation, Standards, and Student Testing CRESST, University of California, Los Angeles