{'fraction_non_alphanumeric': 0.03031898094536267, 'fraction_numerical': 0.012211811769659964, 'mean_word_length': 5.3853909926058705, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Training Medical Students to Create and Collaboratively Review Multiple-Choice Questions: A Comprehensive Workshop

Introduction Multiple-choice question (MCQ) creation is an infrequently used active-learning strategy. Previous studies demonstrated that medical students find value in the process, but have minimal training, which may limit potential learning benefits. We therefore developed a process for question-creation that required students to complete in-depth training, in addition to collaborative question-writing and editing. Methods We created a question-writing workshop consisting of three components: (1) training in MCQ writing utilizing NBME online modules, a practice MCQ-writing session, and a training session, (2) writing MCQs independently after choosing topics from an institutionally generated blueprint, and (3) reviewing and editing MCQs via an in-person session. To understand students' perceptions, we held two four-student focus groups and recorded/transcribed the data. We iteratively reviewed the transcripts to generate a codebook and corresponding themes. We used the focus group data to generate a survey with Likert-scale questions, which we sent to the remaining 10 students and analyzed using Microsoft Excel. Results Eighteen second-year medical students participated in this workshop. Students perceived that question-writing training (3.7/5.0±0.5) and question writing (3.9/5.0±0.3) benefitted their learning. Students perceived that MCQ writing required concept integration (4.1/5.0±0.6). Students described how question writing allowed them to recognize subtle distinctions between therapies and diagnoses. Each MCQ required about 1.5 hours to write and collaboratively edit. Discussion Our results demonstrated that students perceived question writing to benefit their learning. More importantly, students felt that question writing actively engaged them to integrate content and compare concepts; students' engagement suggests that they learned from this question-writing activity.


Introduction
In preparing for assessments during medical school, students often report using passive study strategies such as reading textbooks or notes. 1 In order to improve students' engagement with and understanding of learned content, there has been increasing focus on incorporating active learning in medical school curricula. [2][3][4] Engaging students in active learning is associated with numerous benefits including increased retention of learned content, deeper understanding of material, and increased motivation to learn. 5,6 Previous reports suggested that many learners have received curricular changes involving active learning positively and with enthusiasm. [2][3][4]7 For example, medical students preferred the use of a flipped-classroom curriculum focused on endometrial hyperplasia and cervical dysplasia compared to traditional lecture-based learning. 8 Other activelearning modules focused on obesity management and quality improvement have also been well received by participants. 9,10 Multiple-choice questions (MCQs) are one of the primary methods used to assess medical students' knowledge, 11,12 as they minimize the logistical challenges and time needed to assess multiple content areas while maintaining high levels of validity and reliability. 12,13 Some medical schools have involved their students in the MCQ-writing process as a way to engage them in active learning, introduce them to a novel study strategy, and increase their appreciation for question writing. 14-17 Health professions' student involvement in question writing has been associated with improved examination performance, 18,19 and students who wrote MCQs reported experiencing improved confidence and engagement with the material as a result of the exercise. [20][21][22][23] The training that students have received prior to questionwriting initiatives has been variable; few have included nationally developed training guidelines 15 or in-person locally developed training sessions. 14 The NBME has published detailed guidelines for creating MCQs for the basic and clinical sciences. 24 In a previous report, we demonstrated that students who participated in question writing felt that assessment generation was beneficial to their learning. 17 Recognizing the variability of training in previously reported question-writing processes, we developed a question-writing workshop that included both self-directed and collaborative in-person training to teach medical students to write standardized MCQs.
The purpose of this workshop was to train medical students how to write clinical vignette-based MCQs, use the skills they acquired to write their own MCQs, and collaboratively edit MCQs with their peers. Medical educators interested in incorporating a novel approach to engage medical students in collaborative active learning and introduce them to a unique study strategy will find this workshop particularly beneficial. We recommend that educators utilize this workshop in the education of preclinical and clinical medical students who have completed or are in the process of completing coursework relevant to the content area(s) chosen for the MCQs.

Context
One medical student and two members of our institution's evaluation and assessment (E&A) team collaborated to create the MCQ-writing workshop. The E&A team members consisted of one individual with a doctorate in public health who served as the associate director for E&A, and one individual with an educational specialist degree who served as the assistant director of E&A. Both E&A team members had previous formal training and experience with MCQ assessment creation. A medical student co-created and facilitated this workshop for two reasons: (1) this project was developed in response to a student-identified need, and (2) incorporating the learner perspective and input at all stages was crucial for successful workshop development and implementation, especially with noncontent-expert learners. The medical student completed the same online training modules as the workshop participants.

Recruitment
We used a convenience sampling strategy of all second-year medical students at the University of Michigan Medical School. We sent an email to all 167 medical students at the end of their first year of medical school, and framed participation as an opportunity to learn about MCQ creation and review relevant course material. We recruited students during the summer between their first and second year of medical school, during which time many students had competing commitments. Therefore, we offered participants up to $100 as compensation to complete all aspects of the program, including the workshop evaluation described below.
Workshop Overview This workshop consists of three primary components (totaling 6 hours), as summarized in the Figure:

MCQ Training
Prior to meeting for an in-person training session, learners completed virtual MCQ-related prework. At our institution, students independently completed two 30-minute online NBME University modules on MCQ-writing best practices, which have subsequently been modified and reduced to one comparable online module. This 1-hour online module is free to use and can be found as the "Online Interactive Item Writing Tutorial" on the NBME item writing services portion of the main NBME website. 25 These modules are derived from the NBME's Constructing Written Test Questions for the Basic and Clinical Sciences. 24 Alternatively, students who preferred to learn by reading could review the chapters from Constructing Written Test Questions for the Basic and Clinical Sciences corresponding to the online module content such as the chapters on technical item flaws (p. [11][12][13][14][15][16][17][18][19], 26 basics for writing one-best-answer items (p. 29-33), 27 and structuring items to fit task competencies (p. 40-45). 28 Next, students independently wrote one practice MCQ. These MCQs were not incorporated into the final MCQ bank, but gave students the opportunity to practice question writing as part of their training. Students suggested edits to their peers' practice MCQs via Google Docs. To facilitate this, we divided students into five groups, including three four-member groups and two threemember groups. We provided students with a few key resources to help them write their practice questions: (1) a question formatting sheet to assist with writing MCQs with a consistent structure (Appendix A: Writing Resource 1), (2)  Subsequently, students attended a 1-hour in-person training session. The student lead reviewed the appropriate question format, common question-writing flaws to avoid in MCQ writing, and the resources available to assist students with MCQ writing (Appendix B). One of the members of the E&A team was available during the session to answer any questions or concerns outside of the realm of the student's expertise. After reviewing this content, students then spent 30 minutes with their group members to practice providing feedback on their group members' practice MCQs. Specifically, one student first explained the edits she had recommended for her group members' questions via Google Docs. All group members then discussed their thoughts about the proposed edits, challenging each other's opinions as they deemed necessary, until they reached a consensus.

MCQ-Writing Process
We created a two-dimensional exam blueprint to guide question creation (see Appendix C: Logistical Resource 3 for blueprint template). 29 Our primary framework was the subject matter-in our case the cardiovascular system-with topics within cardiovascular medicine listed underneath the primary framework. Our secondary framework was question type. The four question types we included for our own blueprint were: (1) diagnosis, (2) pathophysiology or mechanism, (3) treatment, and (4) basic science or physiology. The question types and topics used to generate the blueprint should be tailored to suit both the learners' needs and the characteristics of the desired question bank. Example MCQs and lead-ins for each of these question types is included in Appendix A: Writing Resource 2. Students were randomly assigned a topic and question type from the exam blueprint (see Appendix C: Logistical Resource 1 for an example of the topic and question type combination assigned to each student). In order to minimize overlap between question content, one of the workshop leaders generated a list of suggested subtopics for each topic and students chose their subtopic from this list (see Appendix C: Logistical Resource 2 for example topics and subtopics related to the cardiovascular system). Each student remotely wrote two MCQs and suggested edits via Google Docs to their group members' questions before meeting for an in-person review session (see the MCQ review process below).

MCQ Review Process
After all MCQs within a group had been submitted, students reviewed their group members' questions for structure and content and proposed changes using Google Docs on their personal computers. Subsequently, they met for a 1-hour inperson editing session. We gave students three primary aims for the MCQ review sessions: (1) discuss each group members' proposed edits by explaining the editor's rationale and the question-writer's response, (2) use this discussion to reach a consensus on the best changes to incorporate into the MCQs, and (3) submit their final edited MCQs by the end of the session. A workshop leader attended the in-person editing sessions to address student concerns about the logistics of question formatting and submission.
We provided students with a summary sheet with step-bystep instructions for question writing and question editing with references to the relevant appendices (Appendix A: Writing Resource 4).

Workshop Iterations
Depending on the desired number of MCQs as defined by the blueprint, the writing and editing portions of this workshop may be repeated iteratively (Figure). For our implementation of this activity, students repeated the process of writing and editing three times in accordance with our blueprint. Each student wrote two MCQs per cycle for a total of six MCQs each.

Materials
The following materials were required to complete this workshop: r AV equipment to project the PowerPoint presentation during the training session.

Resources Required
Students complete online modules NBME online modules  r Laptops for each student to use during the in-person editing session.

Workshop Evaluation
We measured students' perceptions of the training, writing, and review processes by survey and focus group. 17 Students were randomly selected in gender-balanced pairs to participate in two focus groups, each with four participants. We selected students in pairs from the same MCQ-writing groups to allow them to address questions pertaining to collaborative elements of the writing process; groups were gender-balanced to represent the study sample. We utilized a semi-structured interview guide (Appendix D: Evaluation Tool 1) to lead the interviews and subsequently audio-recorded and transcribed the focus group data. Two authors reviewed the transcripts iteratively to generate a codebook, challenging each other's categorizations until they reached consensus. The authors subsequently used the codebook and corresponding qualitative data to generate themes. We utilized the themes from the focus groups to create a survey which asked students to evaluate various aspects of the question-writing workshop, including 5-point Likertscale questions (1 = strongly disagree, 5 = strongly agree). Specifically, we utilized the qualitative information pertaining to MCQ training, question writing, and question editing to inform the Likert-scale questions. Additionally, we included the questions we asked during the focus group verbatim as open-ended questions.
An individual with expertise in survey design reviewed the survey and made suggestions to improve clarity. Two students participated in concurrent cognitive interviewing and we modified the survey to better correlate with our intended meaning. We sent the survey (Appendix D: Evaluation Tool 2) via Qualtrics to the 10 participants who had not taken part in the focus groups. Two authors utilized the codebook to categorize the qualitative data from the open-ended survey responses; they iteratively reviewed and drew relationships between the focus group and open-ended survey data to inform the final themes. We analyzed the rating-scale questions using Microsoft Excel 2016 (Microsoft Redmond, WA USA) and expressed the Likert-scale data by mean and standard deviation. For responses to the Likert-like questions, the qualitative descriptive interpretations (i.e., agree, strongly agree) reported below were derived from rounding up students' average quantitative response to the nearest whole number value if the average was within one standard deviation of the nearest whole number. If the nearest lower numerical anchor was within one standard deviation of the average, then the qualitative responses were rounded down to the lower anchor. For example, a response of 3.8±0.5 would be reported that students agreed, whereas a response of 3.8±0.8 would be reported as students neither agreed nor disagreed. We represented descriptive data as a percentage.
The Institutional Review Board at the University of Michigan reviewed and approved the protocol associated with this submission (HUM 00129824).

Learner Characteristics
Of the 20 second-year medical students originally enrolled, 18 (90%) completed the study. The two participants who unenrolled completed the online training modules, but did not attend the inperson training session, write MCQs, or attend the MCQ editing sessions. The participants identified their gender as 50% male and 50% female. On average, the participants were 26 years old (range 22-39 years, n = 18).

Quantitative Results
MCQ training: Students' perceptions of completing all components of MCQ training (i.e., completing online NBME modules, writing one practice MCQ, and attending in-person training sessions) were summarized in the Table. Students agreed that completing question-writing training was beneficial to their learning (3.7±0.5; mean ± standard deviation). Students neither agreed nor disagreed that participating in question-writing training increased their confidence in taking NBME multiplechoice examinations (3.2±0.9). Students similarly neither agreed nor disagreed that completing question-writing training changed their approach for future NBME examinations (3.7±0.9).
Question writing: Students reported spending 50.5 minutes (range 10-90 minutes, n = 10) writing each MCQ. For our implementation of this activity, each student wrote six total questions. They repeated the process of writing and editing two MCQs three times (Figure), in accordance with our blueprint. Students also spent less time on question writing for the first session (69.5 minutes, n = 10), compared to the third session (39.5 minutes, n = 10). Students reported that the 2 hours allotted to write their two MCQs and review their group members' questions was "about the right amount of time" (2.0/3.0±0.5, n = 10) on a scale of 1 (not enough time) to 3 (too much time).
The relative time (expressed as a percentage, totaling 100% across categories) that students spent reviewing various resources in order to write their MCQs were as follows: lecture materials (44%), textbooks (19.5%), online search engines (13%), first aid (12%), Pathoma (7%), DynaMed or UpToDate (4%), and other (1%). The student who selected other reported using the medical literature as an additional resource.
Students' perceptions of writing MCQs were summarized in the Table. Students agreed that question writing was beneficial to their learning (3.9± 0.3) and required integration of multiple concepts (4.1±0.6), but not integration of multiple resources (3.5±1.0). Students neither agreed nor disagreed that the act of writing MCQs made them critically analyze distinctions between therapeutic options (3.9±0.9).
Students reported their preferred study strategies as rereading lecture slides, rewriting notes, and reviewing flashcards. As previously reported, 17 students perceived that MCQ writing required much more time (4.9±0.3), problem solving (4.3±0.9), integration of content (4.1±0.7), and differentiation between diagnoses (4.0±0.8) compared to their preferred study strategies. Additionally, we asked students to rank in order the components of the MCQs that were most valuable to their learning when writing. The majority of students (60%, n = 10) reported that writing the answer choices was most beneficial.
Question editing: The findings related to students' perceptions of question editing are shown in the Table. Students agreed that editing their group members' MCQs online and in-person was beneficial to their learning (3.8 ± 0.6) and that it made them critically assess differences between diagnoses in group members' questions (4.0 ± 0.5). Students also agreed that question editing improved the quality of their own questions (4.0 ±0.7). Students neither agreed nor disagreed that question editing improved their ability to deliver feedback (3.7±0.8).
Students reported that they spent approximately 9 minutes on average editing each of their group members' questions prior to the in-person question editing sessions. Between the question writing (50.5 minutes per question), remote question editing (9 minutes per question), and in-person question editing (30 minutes per question), each additional MCQ required approximately 1.5 hours to collaboratively create.

Qualitative Results
We have reported the qualitative results in a previous manuscript. 17 In summary, the qualitative data showed that students gained a deeper appreciation for the difficulty of question writing through completing MCQ training; they felt that question writing required a significant integration of concepts and content and that creating plausible distractors required subtle differentiation between diagnoses and treatment options. Students recognized that question writing is a time-intensive process that required a significant knowledge base, which may limit its efficiency as a learning tool.

Discussion
We created a comprehensive workshop that rigorously trained medical students on how to write and collaboratively edit clinical vignette-based MCQs. Through participation in this workshop, students learned how to apply information from their medicalschool specific lectures and other commonly utilized information sources to create questions. Students perceived that participating in all aspects of this workshop (including question writing training, writing, and editing) was beneficial to their learning. More importantly, students perceived that participation in this workshop required significant problem solving, content integration, and critical analysis of multiple concepts, which suggested students were actively engaged in the learning process.
Through designing, executing, and analyzing this collaborative question-writing workshop, we have learned a number of important lessons for those interested in incorporating MCQ writing as an active learning strategy into medical school curricula. First, students perceived that incorporating robust question-writing training prior to question writing and question editing was beneficial to their learning. However, it is important to note that students were the least enthusiastic in their responses regarding the MCQ training component of the workshop. Students may have been less engaged in this portion of the workshop due to its technical nature (i.e., discussing many different rules around MCQ writing), the comparatively passive learning (online modules) compared with the remainder of the workshop, and the amount of time required to complete training. Educators should weigh students' relative lack of enthusiasm for this portion of the training and the time required with the benefits of creating consistently formatted MCQs for use by future students. Strategies to provide rigorous training in MCQ writing using more active pedagogies (e.g., interactive online modules, gamification techniques, etc.) may be considered.
Second, in light of multiple demands on medical students' time, it was essential to weigh the ideal number and type of questions students were asked to create with the estimated 1.5-hour additional time requirement for each new MCQ. Students perceived that the amount of time allotted to write each MCQ was appropriate, and this in part may have been due to students becoming more efficient in question writing as they wrote additional questions. Students also reported, however, that question writing took significantly more time than their preferred study strategies. The significant time burden required to write each question may have been one of the most important drivers of students' relative lack of enthusiasm for the training portion of this workshop. For our implementation of this activity, each student wrote six total questions. Tailoring the number of iterations of the question writing and question editing components of this workshop ( Figure) to the institution's specific schedule and needs is crucial for optimizing students' receptivity. Additionally, it was important to clearly communicate time expectations up front, and ideally dedicate protected time for students to engage with this activity.
Third, while the primary purpose of this workshop was to actively engage students in question writing, it had the additional benefit of creating a question bank for potential use by future students. If one of the educator's goals is to use studentgenerated MCQs for purposes other than promoting learning in the question writers (e.g., practice questions for other students, incorporation into high-stakes assessment, etc.), validity evidence would be needed. 30 Faculty or other contentexperts (including potential senior learners) should review questions for accuracy and authenticity. Faculty-vetted studentgenerated questions have been shown to be of similar difficulty and quality compared to questions written by faculty alone; the majority of faculty reviewers felt the review process was worth their time and effort. 16,31 Additionally, administering a student-generated question bank to other students would generate important psychometric data (i.e., discrimination indices, student performance, etc.) and allow for association with other variables (i.e., performance on other assessments).
We recognized limitations in our approach. It was possible that our results represented the views of students involved in our institution's 1-year preclinical curriculum, given that we were unable to involve multiple institutions in our study design. We find this unlikely, however, given the congruence between our findings about the critical thinking and extensive time required to write questions and previous studies. 20,22 Additionally, students who participated in this question-writing workshop voluntarily participated, which may have contributed to their positive perceptions of the activity. Despite their voluntary participation, we were reassured of students' honesty by their comfort with expressing neutral sentiments for numerous questions on the administered survey. Given the logistical and time requirements to run this workshop, it was important to consider how many participants were appropriate to include. In its current form, the workshop may be best suited for relatively small groups of students (15-30 total), which may limit broader incorporation into medical schools' curricula. Focusing students' time on the portion of the activity from which they felt they learned the most (i.e., writing plausible distractors) may help to reduce time and administrative burden while improving student receptivity. Doing so may allow educators to scale this workshop and implement it more broadly. Additionally, focusing students' attention on creating the distractors may improve the efficiency of question writing as a study strategy.
The generalizability of our results was limited by our small sample size and the survey data representing slightly over half of the participants. The survey questions were constructed from focus group discussions with the eight students who did not complete the survey, however, adding additional validity to our findings. Further study to assess a larger pool of students' perspectives of this question-writing workshop will be important to strengthen the quantitative findings, which are significantly underpowered due to a small sample size. Additionally, we did not directly measure learning in the evaluation of this workshop; rather, we assessed student engagement, which is associated with learning. [32][33] To further assess students' learning, it may be worthwhile to incorporate a pre-and posttest in future iterations of this workshop. Educators may also consider analyzing questionwriters' performance on subsequent MCQ examinations for the topics they chose to write about compared to their non question-writing peers; student involvement in question creation has previously been shown to be associated with improved performance on subsequent MCQ examinations. 19 Lastly, although we used cognitive interviewing and expert review to develop and refine the survey, additional validity evidence could be gathered. 34 Our proposed question-writing process engaged students in an active learning activity that they perceived to be beneficial to their learning and required significant conceptual integration. Further study to directly assess how student involvement in this workshop leads to improved performance on subsequent MCQ examinations will be beneficial. Additionally, exploring strategies to involve faculty in the question review and editing process may be beneficial for student learning and question quality.

Ethical Approval
The Institutional Review Board at the University of Michigan reviewed and approved the protocol associated with this submission (HUM 00129824) on June 26th, 2017.

Disclaimer
Some of the materials used or referenced in this workshop are associated with the NBME. The NBME was not involved in the creation of this question-writing workshop and does not endorse use of this workshop for educational purposes. The NBME has granted permission to use all related materials.