{'fraction_non_alphanumeric': 0.02089735709895513, 'fraction_numerical': 0.004917025199754148, 'mean_word_length': 5.672131147540983, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Automated Analysis of Written Assessments in STEM : Methodological Issues

Understanding genetics is fundamental for biological literacy and is broadly recognized as one of the most difficult disciplines in biology. Although a number of published assessment instruments are available to assess students’ genetics knowledge, it is unclear to what extent these instruments can measure students’ understanding or reasoning. Constructed response (CR) questions, in which students craft responses to questions using their own words, may provide a more authentic assessment of students’ understanding and reasoning. Automated scoring has made it possible to use CR assessments in large enrollment classes. We have created CR assessments in which students are asked to predict the effect of a DNA mutation on the processes of replication, transcription, and translation. We analyzed over 4,000 student responses, collected from introductory and upper-level biology classes across five universities in the U.S., using a computer scoring model, trained with 1,031 human scored responses. Analyses of path maps of students’ responses and written answers revealed students’ alternative conceptions of each process. The results show that students’ understanding of transcription is critical for the overall comprehension of the processes involved in the genetic information flow. Researchers may find our approach useful for evaluating overall students’ performance in multiple-part CR questions. We suggest instructors focus on transcription as a keystone concept to understanding the processes underlying the flow of genetic information.