{'fraction_non_alphanumeric': 0.0339652448657188, 'fraction_numerical': 0.0007898894154818325, 'mean_word_length': 5.431472081218274, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Exploring ‘reverse-tracing’ Questions as a Means of Assessing the Tracing Skill on Computer-based CS 1 Exams

In this paper, we perform a comparative analysis using a within-subjects ‘think-aloud’ protocol of introductory programming students solving tracing problems in both paper-based and computer-based formats. We demonstrate that, on computer-based exams with compiler/interpreter access, students can achieve significantly higher scores on tracing problems than they do on similar paper-based questions, through brute-force execution of the provided code. Furthermore, we characterize the students’ usage of machine execution as they solve computer-based tracing problems. We, then, suggest “reverse-tracing” questions, where a block of code is provided and students must identify an input that will produce a specified output, as a potential alternative means of assessing the same skill as tracing questions on such computer-based exams. Our initial investigation suggests correctly-designed reverse-tracing problems on computer-based exams more closely track a student’s performance on similar questions in a paper-based format. In addition, we find that the thought process while solving tracing and reverse-tracing problems is similar, but not identical.