{'fraction_non_alphanumeric': 0.021781717966596937, 'fraction_numerical': 0.0055118371683766645, 'mean_word_length': 5.134012219959267, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
The Development and Use of a Multiple-Choice Question (MCQ) Assessment to Foster Deeper Learning: An Exploratory Web-Based Qualitative Investigation

This paper reports on the development and piloting of a new model of multiple-choice question (MCQ) assessment used in two undergraduate degree modules at a tertiary university. The new model was purposefully designed to promote deeper learning closely aligned with the SOLO taxonomy. Students were invited to participate in an exploratory qualitative study exploring their experience of learning using this new assessment. In total, 13 students completed an online open-ended qualitative questionnaire. Data was analyzed thematically. Four themes were generated: (a) empowered choice, (b) iterative reading, (c) forcing comparison, and (d) justified understandings. Findings suggest that the new model MCQ assessment promoted wider and more prolonged engagement with learning materials and fostered critical comparisons resulting in deeper learning. Limitations in study design mean that further research is merited to develop our model of MCQ assessment and enhance our understanding of students' learning experience.


Introduction
In the past twenty years, asynchronous, web-based learning has become a common means of study for learners in higher education (HE). The University of the Highlands and Islands has utilized virtual learning environments (VLEs) such as Blackboard and Brightspace as vehicle for sharing learning materials with students. Embedded within these VLEs are a variety of assessment tools. As with many other HE courses, most assessments are carried out through assignments and essays that are submitted via the VLE to be marked by a course tutor. Such assessments require a substantial investment of tutor time to mark and, with large online class sizes, tutors can find themselves overwhelmed with marking and learners can find themselves waiting a long time for their results.
As a result of these demands, multiple-choice question (MCQ) exams and assessments have been a common assessment format used for online learners. MCQs are set by the course tutor, based on the learning materials within the VLE. Such assessments are highly customizable, with many VLEs offering a variety of options regarding time (both how long the test is available for and how long the learner has to complete it once they have started), presentation (font size and accessibility), and sequencing of question order. An attractive feature of MCQs is that they can be marked automatically by the VLE and the results are instantly available to learners.
However, delivery of traditional MCQ exams and assessments in the digital context can be problematic. For example. if students take the exam remotely, they might complete the assessment whilst having an appropriate textbook handy for reference, thus encouraging a surface or strategic approach to passing the assessment. Indeed, Biggs and Tang (2011) reflect that "MCQs encourage game playing strategies, by both student and teacher" (p. 233). Moreover, when MCQs are delivered online via a VLE, students can become anxious, not just because it is a timed assessment, but because it involves the use of learning technology which some students might be unfamiliar with (Davies, 2015).
MCQs can take various forms with the best known featuring a stem (which comprises a question or a statement) and a set of several potential responses that the student must choose from (Draper, 2009). Other forms include selecting multiple answers from the given choice, selecting the true or false answer, filling in blanks in the question from a given selection, ordering responses, etc. However, critics argue that traditional MCQs foster shallow or strategic learning, rather than encourage deep learning through assessment as part of constructive alignment (Biggs & Tang, 2011). Draper (2009) describes several different versions and adaptations of MCQs designed to foster a deeper learning, but all are essentially the same format: the stimulus material takes the form of a short statement or question followed by potential responses or answers that usually consist of only a couple of words, a sentence, or a number. Alternative approaches to foster deep earning includes asking a student why a response is true rather than which response is true (Draper, 2009).
When considering MCQs as an assessment tool, there are many concerns that must not be overlooked. Both McCoubrie (2004) and Brady (2005) point out how MCQs can be criticized for testing only superficial learning and regurgitation of facts. Biggs and Tang (2011) highlight the 3 problem with the "surface approach" (p. 26) to learning in that learners are invariably assessed on their knowledge of independent facts and a deeper understanding of the topic is not required. When considering Bloom's Taxonomy, such surface or superficial learning is only utilizing the first level of thinking: knowledge (or remembering in Anderson, Krathwohl and Bloom's (2001) revised taxonomy). Biggs and Tang (2011) go on to highlight how a surface approach to learning means that because the assessment is perceived as a task to be got out of the way "with minimal trouble, while appearing to meet course requirements" (p. 24), the learners will focus on memorizing individual facts, without seeing the necessity for a broader or deeper understanding of the topic. Biggs and Tang (2011) use SOLO (Structure of Observed Learning Outcomes) taxonomy to describe learning outcomes. These range from pre-structural answers where the student has essentially "missed the point" (p. 88) through to extended abstract answers where the student has, amongst other things, generalised beyond the presented material. Traditional MCQs which require verbs at the lower order learning outcomes such as memorize or match (Biggs & Tang, 2011) afford no opportunity to operate at the relational level in the qualitative phase of the SOLO taxonomy, requiring higher order verbs such as compare, contrast, or analyze.
In this paper we present a new model MCQ assessment which has been purposefully constructed to fostering deeper learning (described below). We believe that our new model MCQ assessment proffers several potential advantages over traditional MCQs. First, the model encourages students to read around topics that are not solely related to the stem (stimulus) material that they are asked to consider, but which are related to the wider course learning material. In this sense, the new model MCQ assessment is constructively aligned and the assessment itself is a learning tool. Second, students are required to write a brief justification for their selected answer response. Marks are awarded for the justification. The marking criteria for this justification is aligned to the SOLO taxonomy (Biggs & Tang, 2011). The guidance for marking the justifications indicates that higher marks should be awarded for responses that align with multiscriptual or relational levels of the SOLO taxonomy. Additionally, the new model MCQ assessment format eliminates the need for a timed-MCQ examination, thus addressing students' anxiety generated from technical aspects of online MCQ examination (Davies, 2014). Following this, we report on the findings of a pilot qualitative study which explored students' learning using the new model MCQ assessment.

The New MCQ Assessment
Our new model MCQ assessment introduced two elements of choice: (a) choice within the question stem (i.e., selecting the correct answer response), and (b) choice between question stems.

Choice Within Question Stem
MCQ exams normally have several stems which the student must answer. Accepting various permutations of MCQ question formats, MCQs comprise a stem (either a question or a statement) and several possible responses that the student must choose from (Draper, 2009). In our new model MCQ assessment, the stem presented to the students instead took the form of academic 4 articles which acted as stimulus material. The academic articles closely aligned to the learning material that students engaged with during the module. Students were then presented with four statements allied to the article (i.e., possible responses). Students then selected a response which they considered to be most relevant to the article, from a choice of four. A maximum of 10 marks were available for the selection of the response most relevant to the article chosen. The first response was fully relevant, and its selection afforded the student the maximum 10 marks. The next response was less relevant and attracted 5 marks. The next was semi-relevant and attracted 2 marks, while the fourth and final response was not relevant and did not attract any marks. This selection process is why the assessment is multiple choice. The responses were presented in a random order. Students were asked to write a 200-word justification for their choice of response. This justification task was worth a maximum of 20 marks. If a student has selected the least appropriate response, marks can still be awarded for the justification of their choice of selected response.
Marks for the justification of choice of response were awarded at the discretion of the marker, but guidance was issued. The guidance was informed by the SOLO taxonomy (Biggs & Tang, 2011) where more marks were available for justifications of choice of statement that demonstrated more qualitative learning, rather than learning at the quantitative phase of the SOLO taxonomy. The rationale for this is that the more deeply students have engaged with the stimulus material (i.e., the academic article), the more likely students' answers will be at the qualitative phase, described in the SOLO taxonomy. An example of the guidance given to markers when assessing the justification of the choice of statement is given below. More marks are available for evidence of deeper learning. This example relates to the justification of the choice of response associated with an article on biological psychology.
• Reiterating the use of different research methods in different contexts, as per the article will only attract a few marks, e.g. 4 marks. • More marks are available where students might explain the difference between real-world (high ecological validity) vs laboratory-based research (low ecological validity) but do not do this in the context of research into attraction e.g. 8 marks. • Explaining the different approaches to research in the context of attraction and biological psychology more generally will attract 12+ marks. • Explaining research approached and relating this to psychological research in other areas can attract 16+ marks (maximum 20).

Choice Between Question Stem
Rather than simply present students with a single stem (academic article as stimulus material), students were asked to select an academic article and then answer an MCQ associated with it (select an appropriate response). In our exploratory study reported below, one student cohort was offered three academic articles (stems) to choose from, whilst another cohort was offered two academic articles. The new MCQ assessment is summarized below.

5
• Students are presented with several question stems i.e. stimulus materials in the form of topic-specific academic articles closely related to module learning materials • Students are asked to select one of these academic articles to read closely • Students are asked to answer the MCQ which accompanies their chosen academic article • The MCQ comprises four response statements which summarise learning relating to the corresponding article, each response statement ranges from 40 to 100 words • The four possible responses attract marks on a sliding scale with maximum marks available for the most appropriate response statement and no marks available for the least appropriate response statement • Students are asked to select one response statement which they believe to be correct • Students are asked to justify their choice of response in 200 words, which was graded using predetermined marking criteria informed by the SOLO Taxonomy (Biggs & Tang, 2011) Methods

Design
A qualitative study in the form of an open-end questionnaire was used to explore students' experience of learning using the new model MCQ assessment.
Setting, Sample, and Recruitment. The University of the Highlands and Islands is a tertiary university delivering both further and higher education. It is the only tertiary university in Scotland and one of only a handful, globally. The new model MCQ assessment was piloted in two undergraduate modules, Introduction to Psychology and Child and Adolescent Mental Health. The new model MCQ assessment was a mid-term summative assessment comprising 30% of the total module weighting. The assessment was made available to students via the Brightspace VLE at the beginning of the module. The new model MCQ assessment was administered as described above. In the Introductory to Psychology module, students were asked to select their preferred stem from a choice of three stems (academic articles), whilst the Child and Adolescent Mental Health students were offered two stems (academic articles). There were academic reasons for the difference in the number of choices of stimulus material between the two modules related to the learning material for the module.
All students enrolled in the undergraduate Introduction to Psychology and the Child and Adolescent Mental Health modules and who had completed the new model MCQ assessment were invited to participate via UHI's virtual learning platform; Brightspace. All potential participants were sent a copy of the participation information leaflet. Informed consent was obtained electronically with students free to withdraw at any time. In all, 13 students participated.

6
Data Collection. A five-item open-ended questionnaire was developed. The questionnaire was administered electronically using JISC Online Surveys with unlimited text-box responses available to respondents. This data captured students' perceptions about the format of the assessment overall and about their level of engagement with the assessment. Listed below are the five questions from the open-ended questionnaire.
1. Please describe the process you engaged in to complete the MCQ assessment focusing on what you did to make your selection of article to read. 2. Please describe what you did to understand the article you selected. 3. Once you had selected your article, how did you decide which response was the most appropriate? 4. Please describe your approach to writing your justification for your choice. 5. To what extent did the MCQ assessment articles enhance your learning and understanding of the module content?
Data Analysis. Qualitative questionnaire responses were imported into NVivo software (Version 12). Data was analyzed using thematic analysis (Braun and Clark, 2006).
Ethics. Ethical approval was granted by the research ethics committee of the University of the Highlands and Islands, application number OLETHAHB1871 on October 3, 2019. Data have been anonymized to preserve confidentiality.

Results
Data analysis generated four interdependent themes to describe students' experience of the new model MCQ assessment and their learning: (a) empowered choice, (b) iterative reading, (c) forcing comparison, and (d) justified understandings.

Empowered Choice
Students described how the new model MCQ assessment facilitated their learning by offering empowered choice. Students described a range of factors influencing their choice of article including personal relevance, interest, and perceived difficulty. Paradoxically some students chose the academic article which they perceived to be easy, whereas others chose an article they perceived as challenging.
"I chose the article that was mostly interesting to me and also the one that was easier for me to understand language wise."

"[I] chose the one [article] which I related to the most and the one I found most interesting as well as challenging."
Allowing interest to actively drive students' choice of their selected stem (academic article) was perceived as an important facilitator to their learning:

"I read over the choice of articles presented and decided on the article that made most sense to me and that I felt most interested in, my train of thought was that if it's a subject I understand or feel more naturally interested in then I would engage with it a lot easier."
Moreover, some students expressed that empowered choice had enabled them to enjoy the assessment.
" [I] selected the one that most interested me and the one I thought I understood best and I could easier answer the question. I did like the multiple choice as if we had been given only one article to work, chances are that the article could be difficult or uninteresting."

Iterative Reading
Students described a process of iterative reading of the academic articles which functioned to promote initial understandings of the topic.

"I read it and re-read it about five times. I took notes from each section and summarised what my understanding of each section meant."
Understanding the meaning of the text within the academic articles was critical to enhancing understanding. This often necessitated recursive readings of sections which they had found difficult to interpret.

"I read the article once through, picking up on words which I didn't fully understand, so that I could then read it again, aware of what everything meant."
This process of iterative reading additionally encompassed reading beyond the source academic articles (e.g., review of secondary sources, module materials, independent reading, etc.). This enabled students to dig further to explore the topic in more depth and check their learning.

"I think that it was good so that we would be challenged to research the topic in particular and better understand it."
Such iterative readings advanced understanding and helped promote judgement about the subject matter and enabled students to justify their answers.

Forcing Comparisons
Data signals that students were forced to make comparisons while completing the assessment. Students described fragmenting the academic article to facilitate comparisons with the four potential responses within their selected academic article.
"I set aside the multiple-choice selections in a paper format and read the article online and after every paragraph I referred back to the multiple-choice question." Students described a process of positive confirmation by comparing and analyzing the potential responses to their selected academic article. Notably some students chose to match the statement response to the academic article, whereas others retrospectively mapped highlighted elements of article back to the statement responses.
"I went through each statement (or sentence) within each response, went back to the article and chose which one [statement] fitted in best to the article." "I went through the article, highlighting points which supported parts of the statements. By the end of reading through, I had one completely highlighted statement." Others depicted as using an initial discounting or elimination process prior before moving to positive confirmatory selection of the chosen answer response.
"…first eliminating those [answers] that did not seem to fit at all, and I found out that 2 answers could work and then I read the article again to select the right answer. I tried to find in the article the exact / or paraphrased words that was used in the answer." Students reported that such comparisons were "initially difficult as the options were all initially similar" (anonymous student). However, one student reflected that fragmenting the article to force comparisons "help[ed] me practice to break down an article and really consider what was being said". Moreover, students reflected that the assessment had provided an "opportunity to think of the other side of the opinion" (anonymous student) suggesting critical engagement. Others perceived that the new model MCQ assessment had challenged belief and pre-existing assumptions promoting deeper thinking and learning.

Justified Understandings
Data suggests that asking students to justify their choice of MCQ response helped developed students' understanding of both module learning material and the academic articles. Students described, beyond making a choice about their selected response that continued 9 interrogation of the academic article was necessary to justify their answer thus prolonging engagement with the learning material fostering deeper learning.
"I read it through, then re-read with the statements in mind to ensure I had best selected which statement fit most relevantly. Then as I was writing my explanation, I kept skimming through the article." This continued interrogation of the academic article and wider literature moved beyond iterative reading in that students were compelled to apply their understandings by critically justifying their chosen response statement.

"I went about my justification by jotting down my notes of which points in the article supported the chosen statement, and then elaborated on these points in order to get my understanding of the topic across."
Students reflected that having to justify their chosen response "helped cement that I had grasped ideas correctly" whilst the limited word counts available for justification had encouraged students to focus on key understandings of the topic material.

Discussion
The findings from this study strongly suggest that students engage deeply with the academic articles presented to them as part of this new model MCQ assessment and also additional material that they source independently. The stimulus material in the form of academic articles and the additional reading that the students engage with support and are related to the module learning materials. This suggests, therefore, that the new model MCQ assessment that we have presented stimulates wider engagement with the literature.
Traditional MCQ assessments, whether they are used as informal checks on progress or whether they are used as formal assessments, do not encourage students to operate beyond the unistructural level of the quantitative phase of Biggs and Tang's (2011) SOLO taxonomy. The responses from participants, and the themes we have identified here, strongly indicate that students are functioning at the relational level of the qualitative phase of the SOLO taxonomy and are engaging deeply with the module learning material, the stimulus articles, and additional reading.
The theme of empowered choice indicates that students felt they had the opportunity to engage with material that interested them stimulating further engagement. The stimulus material took the form of a number of academic articles and students were tasked to select one of them and a degree of understanding is necessary to make that empowered choice. Some students made their selection of stimulus material based on whether they found the academic article challenging 10 whereas some students sought the one that they perceived to be easy. Either approach to selecting or deselecting stimulus articles requires the student to read all the stimulus articles, in at least some depth. The data does not enable us to judge how deeply students read the stimulus articles if they were looking for the easy option, but they must have engaged in at least some level of analysis in order to make the decision and that again is within the qualitative phase of the SOLO taxonomy (Biggs & Tang, 2011). Further research might yield some insight into this strategy. Some respondents reported enjoying the assessment and this was made possible by providing students with a choice of stimulus article. Again, making the choice based on interest and enjoyment necessitated reading all the stimulus articles and selecting the one that they enjoyed the most. This required an element of comparison and contrast of the stimulus articles and encouraged students to function at the relational level (Biggs & Tang, 2011).
The theme of iterative reading describes how students read the selected stimulus article multiple times to consolidate their understanding of the article. Many went on to read other sources of evidence to validate their initial understanding of the stimulus article. In this way, students were relating their understanding of the stimulus article to independently selected reading. It is apparent that the structure of the new model MCQ assessment encourages the learners to read and re-read the papers in question. Given that an oft-repeated complaint of HE lecturers is that students do not read widely enough or with sufficient depth, it seems that this new model MCQ assessment is beneficial in tackling this complaint. This pursuit of a deeper and cemented understanding of the topic by relating the stimulus article with other independently selected reading illustrates how students are operating at the relational level of the qualitative phase described by the SOLO taxonomy (Biggs & Tang, 2011). A similar strategy was used by respondents to select what they understood to be the most appropriate MCQ response. The iterative reading also helped students develop their justification of their MCQ response.
Data illustrated how students were forced into making comparisons between the stimulus articles and potential MCQ responses. The process of fragmenting and deconstructing the stimulus article into its constituent parts and comparing it to the potential MCQ responses was a strategy that some students used to identify the key elements of the topic under discussion in the stimulus article in order to formulate a deeper understanding. Deconstructing an academic article into its constituent parts requires the learners to consider the key elements of the topic under discussion in order to formulate a stronger holistic understanding Students repeatedly compared the potential responses to the MCQ to the stimulus article or vice versa. This compare and contrast process is firmly in the relational level of complexity in the SOLO taxonomy and is therefore in the qualitative phase. This is certainly a higher level of complexity than the traditional MCQ which rarely moves beyond the unistructural level and is therefore in the quantitative phase (Biggs & Tang, 2011). It is interesting to note that there were two different, but related strategies employed by students: they used comparative analysis, but some chose to start with the MCQ responses, whereas others chose to start with the stimulus material. The data does not support any conclusions as to why this might be or the reasoning behind it, so further research is needed. Challenging students to see other sides of the debate, self-11 reflection, and approaching materials from differing, sometimes contradictory, perspectives are fundamental building blocks of good critical engagement.
Students were asked to justify their selection of MCQ response. The data support the contention that the format of the new model MCQ assessment that we describe here, specifically the requirement for the justification of their selection of MCQ response, fosters a deeper engagement with module learning materials, the stimulus articles, and self-selected further reading. This is not something that traditional MCQs tend to achieve. We have described the way in which we deployed the revised model of MCQ. We chose to use a number of alternative stimulus articles and offering students this choice is a powerful aspect of the model. However, there are numerous different types of journal articles ranging in complexity from the simple to the highly complex. The model offers tutors the opportunity to select the stimulus articles that they consider to be the most appropriate for their students. There are alternatives to journal articles: book chapters, videos, pieces of music, etc. As such, we suggest that this new model MCQ assessment is flexible and adaptable to online, blended, and face-to-face delivery. The assessments can be delivered in exam conditions or can be used as a means of checking progress. Time limits can be added where necessary.
The model that we describe here may be less suitable for subjects such as mathematics where a range of possible answers conceal the correct answer and students are expected to make a calculation in order to identify the correct answer. However, in the same way that we seek a justification from students for their selection of the MCQ response, mathematics students can be asked for the calculation method.

Study Limitations
The data for this study were gathered using an online survey approach. Open, qualitative questions were presented to respondents and the responses were gathered and analysed. However, the nature of online open questions is such that responses are often shorter than those gathered using semi-structured interviews or focus groups. Gaps in our ability to understand exactly how deeply students read the stimulus articles or why students used different comparative analysis strategies could be addressed by using focus groups or semi structured interviews.

Conclusions and Implications for Practice
We have presented a new model of MCQ assessment which we believe encourages students to engage more deeply with module learning materials, the academic articles presented as stimulus material and students' self-sourced material. The data that we report is gathered from students who have completed the new model MCQ assessment and reflects their experiences of deeper engagement and deeper learning with material associated with the module learning outcomes.
The new model MCQ assessment is designed to be flexible and adaptable and we encourage colleagues to use it in different circumstances and for different disciplines. Where encouraging students to engage more deeply with learning material is important, the model we 12 have described here may be beneficial. Further research is merited to develop and enhance the model.