{'fraction_non_alphanumeric': 0.02434854592571264, 'fraction_numerical': 0.010239706305787504, 'mean_word_length': 4.977089383672152, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Use of Feedback-Oriented Online Exercises to Help Physiology Students Construct Well-Organized Answers to Short-Answer Questions

Online feedback-oriented exercises were developed to help undergraduate physiology students practice formulating answers to short-answer questions. Student feedback regarding the learning value of these assignments and the ability of these assignments to improve student outcomes when answering short-answer questions on summative examinations were assessed.


INTRODUCTION
Many science disciplines require both a critical arsenal of factual knowledge and an in-depth conceptual understanding so that students can apply their knowledge and understanding to novel problems and contexts (Crowe et al., 2008;Mynlieff et al., 2014). It is the ability of adult learners to use what is learned during lectures to critically evaluate a novel situation and problem solve that is widely recognized as an important goal of university-level teaching of physiology as Students progressing through basic science programs often find it difficult to make the transition from answering MCQs that assess primarily remembering, understanding of individual pieces of information, essentially a passive activity, to developing well-worded answers that demonstrate their ability to both apply basic science principles and link them in a logical manner, which is an active process (Zoller, 1993;Bailin, 2002;Crowe et al., 2008). Higher-order learning involves the reorganization of new knowledge and understanding as it is transferred from working memory into long-term memory in a way that forges links with that which is already known and understood so that each student's long-term memory can evolve as it integrates new components associated with a more complex understanding of scientific processes (Kirschner, 2002;Kirschner et al., 2006). Referred to as "generative processing" (Wittrock, 1992;Mayer, 2010), this type of higher-order learning can be facilitated (Chi et al., 1994) and assessed by asking students to construct concise, well-written answers to short-answer questions (SAQs). For students with limited experience in this type of evaluation, this can be challenging. Indeed, I have noted the following when marking answers to SAQs provided by students during previous years of summative physiology examinations: inaccurate information, the provision of information that, while correct, is not relevant, the omission of key points that would demonstrate an indepth understanding of relevant concepts, and, finally, an inability to organize the answer logically. I concluded that students needed to practice developing well-organized and accurate answers to SAQs and that, as instructor, I needed to provide that practice before students encounter similarly styled questions on summative examinations (Gagné et al., 1992;Levesque, 2011). Unfortunately, the large sizes of many undergraduate classes make it difficult to offer regular practice in developing explanatory answers and solving problems and, more importantly, to provide students with timely, answer-specific feedback on the written work they have prepared.
It is also important to match course-associated learning activities with summative assessment criteria so that, through practice, students might gain or improve their ability to use HOCS before their work is evaluated via high-stakes summative examinations (Sundberg, 2002;Tanner and Allen, 2004;Bissell and Lemons, 2006;Crowe et al., 2008). Indeed, studies have shown that assessment methodology influences the strategies used by students when preparing to write examinations (Scouller and Prosser, 1994). For example, if it is known that an examination will be exclusively MCQ-based, students will often prepare for it by using surface strategies such as the compilation and memorization of lists of factual information, because they know that assessment methods will be used that address primarily LOCS (Scouller, 1998). However, if students know that the summative assessment has been designed to test HOCS such as analysis, evaluation, and the provision of detailed explanations, then students will use learning strategies that involve the use of HOCS through in-class and out-of-class activities that support these approaches (Scouller, 1998;Crowe et al., 2008).
The purpose of this project was to develop an online tool that would allow students enrolled in a third yearlevel physiology course to practice building, step-by-step, well-organized and comprehensive answers to questions housed within clinical situations or specific physiological scenarios. The development of these interactive exercises was guided by Robert Gagné's nine events of instruction ( Figure 1) so that the ability of this tool to capture student attention and promote interactive, effective learning could be maximized (Gagné et al., 1992). Students will not use an online tool simply because it exists; rather, they must see value in it, including how it links to course objectives and has the potential to improve their summative assessment outcomes (Saunders and Gale, 2012). While it has been more than 20 yr since its original publication, Gagné's list of instructional goals continues to influence instructional design, because it addresses basic principles, such as the need to motivate and engage learners as well as to guide them toward integrating new knowledge with that which is already known so as to enhance understanding, retention, and transfer (Kinzie, 2005;O'Byrne et al, 2008;Carnegie, 2013). The exercises developed for my study asked students to demonstrate their understanding of four important physiological concepts: 1) the regulation of metabolic rate by thyroid hormone, 2) the control of blood sugar by insulin, 3) the influence of altitude on the regulation of erythropoiesis, and 4) the intrinsic regulation of cardiac stroke volume. While the questions used for these online exercises were not identical to questions students subsequently encountered on midterm examinations, they were representative of the level of cognitive ability the students would be expected to exhibit in the short essay component of summative evaluations. During the completion of each of the four online assignments, students were provided with guidance on answer construction and instantaneous feedback regarding the appropriateness of the items they had selected to include in their answers. At the end of the course, students were asked to complete an anonymous survey detailing their perceptions of the effectiveness of the online tool. In addition, student outcomes on SAQ sections of summative exams were compared between populations that had access to the online tool and those who did not and between subgroups of students who showed various levels of participation in the online SAQ assignments.

Exercise Creation
PHS3240 is a full-year, third year-level course in mammalian physiology that is team taught by a number of faculty members within the Department of Cellular and Molecular Medicine at the University of Ottawa. Systems within this course that are taught exclusively by the author include the endocrine and cardiovascular systems. Students learn about the endocrine system during the first part of the course and are assessed on that knowledge in exam 1. Similarly, the cardiovascular system is covered during the next section of the course, with summative assessment of that content during exam 2. Examination scores for this course are typically derived in equal parts from MCQs and SAQs, with each SAQ having a value of five to seven points.
Four interactive exercises were developed for students taking PHS3240, using the action maze software Quandary (Arneil and Holmes, 2009). This freeware can be downloaded for use without charge at www.halfbakedsofware .com/quandary_download.php. Advantages of this software include the ability to send students back to retry a question if an incorrect answer is selected and to provide feedback that individually targets every answer choice, be it correct or incorrect. Furthermore, the scoring function of Quandary allows points to be awarded or deducted, as appropriate, and students to see their scores as they proceed through the exercise. Initial exercise planning involved the use of tables to organize the questions, the answer choices, the answer-specific explanatory feedback, and the marking scheme. Interactive exercises were then created by loading that information into Quandary, defining the navigation to be followed by the student, and finalizing the scoring. The first two exercises addressed the endocrine system and were completed by students in preparation for exam 1. The second pair of exercises targeted the cardiovascular system and was completed as students prepared for exam 2.
Each exercise opened with the presentation of a physiological scenario (e.g., training for a cycling competition at high altitude) or a clinical vignette (e.g., a patient displaying symptoms of Graves disease) accompanied by one or more essay-style questions for which students would normally be expected to compose a well-organized and concise written answer (Table 1A). However, rather than being asked to provide a written answer, a student was instead taken through a series of six to eight MCQs in which he or she developed an outline of his or her written answer by clicking on the "submit" button to select items that should be included in that answer and, by virtue of the choices he or she made, specifying the order in which that information should be presented. Distractors that were provided in the MCQs included incorrect information, information that was correct but did not apply to the question being answered, and information that was correct but should not be presented until later in the answer after the groundwork for that particular piece of information had been laid (Table 1A).
Feedback was provided for each answer choice, either correct or incorrect. If the answer was correct, the feedback provided a short corroborative explanation, sometimes accompanied by additional pieces of information to further consolidate the concept (Table 1B). If one of the distractors was chosen, the feedback explained why that answer was not the best choice, sometimes provided a hint, and always directed the student back to try again by having him or her click on the "Submit" button associated with that feedback ( Table 1C). As an additional effort to reinforce guidance of student thinking (Gagné event 5), the last page of each exercise provided a summary of the correct answers and/or feedback in an effort to show students a worked example of an orderly outline of the elements that should be included in a complete answer to that SAQ (Table 2).
In addition to the inclusion of the summary page, care was taken throughout the process of exercise construction to apply Gagné's nine events of instruction (Figure 1), as detailed in the Reasoning annotations applied to the working document for exercise creation shown in Table 3. The example used is the endocrine SAQ exercise pertaining to the thyroid gland, and shaded Reasoning boxes shown throughout the table identify examples of specific application of Gagné's instructional design principles. For example, the patient presenting with Graves disease was used as stimulus material at the beginning of the exercise to grab student attention (events 1 and 4) and provide an opportunity for application of knowledge with the goal of enhancing retention and transfer (event 9). Furthermore, the broad questions that appeared at the end of each case description informed students of the objectives to be addressed within that exercise (event 2). Finally, as detailed throughout Table 3, each of the individual MCQs with their answer choices and associated feedback repeatedly addressed Gagné's remaining events by stimulating recall, providing practice and learner guidance, and assessment and feedback for every answer selected. Finally, the series of MCQs were constructed in such a way that they guided student thinking as to the logical order in which correct items should appear in their answers (event 5).

Student Populations, Participation, and Feedback
Student enrollment in PHS3240 numbered 144 and 120, respectively, at the beginning of the 2010 and 2011 academic years. There was some attrition over the duration of the course, resulting in 139 and 108 students, respectively, writing exam 2 in the Winter terms of 2011 and 2012. The SAQ exercises were provided online to students using the assignment function of the Blackboard Vista (Blackboard, Washington, DC) course website. Students had 1 wk to complete each exercise and to submit their scores online using the course website assignment drop box. Each correct answer was worth one point, and this resulted in each assignment having a total possible score of six to eight points, depending on the number of questions in that assignment.

Analysis of Student Outcomes
The possible influence of online SAQ assignments on student outcomes was assessed by comparing student SAQ scores achieved during the 2 yr before implementation of the online exercises with those earned during the subsequent 2 yr, when students had access to these assignments. A marking rubric was used for each summative examination SAQ, and these rubrics, like the summary pages viewed by students upon the completion of each online exercise, consisted of short lists of required answer components. The same rubric was used each time a particular SAQ was included in a summative examination. Within each rubric, points were awarded not only for including the required answer elements but also for defining and/or explaining them completely and for applying them correctly to the solution of the SAQ. The statistical comparison of student outcomes was conducted using the independent-samples t test accompanied by Levene's test for equality of variances. Possible While points were not usually deducted for an incorrect answer, on the rare occasion when an answer was selected that was associated with a serious error in fundamental reasoning, a single penalty point was deducted from the student's score to emphasize that error. The software also allowed students to be blocked from choosing the same answer (correct or incorrect) more than once. Each of the four exercises contributed a possible 1% of the overall final mark for this course.
Student feedback was obtained at the end of the first year of implementation of the SAQ assignments using an anonymous survey that consisted of seven Likert-scale questions and a box for open-ended comments and suggestions. The survey collected data on student demographics, self-assessed level of participation in the assignments, perception of exercise value, and interest in having access to additional similar assignments. These data were compiled and analyzed using Microsoft Excel. Table 1. Online appearance of SAQ interactive exercise pertaining to the regulation of hematopoiesis A. The SAQ a G.C. is planning to compete in a 100-km cycling race in the mountains in the middle of July. Because he normally lives at sea level, he has decided that it would be a good idea to travel to the race site 6 wk ahead of time to complete his final training. Use your knowledge of aspects of the regulation of hematopoiesis to explain why this training strategy makes sense. If you did a reticulocyte count on G.C. a week or so after he began training in the mountains, what would you find? Related to this, should G.C. have any concerns about a cycling race of long duration at high altitude in midsummer?
2. You have stated that oxygen partial pressure will be reduced at the higher altitude. How will you relate this to the regulation of erythropoiesis?
Submit a. As G.C. trains at the higher altitude, his muscles will sense reduced oxygen delivery and become larger and stronger.

You have:
points: 1 Submit b. As G.C. trains at the higher altitude, his kidneys will sense reduced oxygen delivery and stimulate erythropoiesis.
Submit c. As G.C. trains at the higher altitude, his heart will sense a reduced hematocrit and increase its release of the hormone erythropoietin.
Submit d. As G.C. trains at the higher altitude, the new red blood cells developing in his bone marrow will switch to producing a type of hemoglobin that can carry a higher number of oxygen molecules.
Submit e. As G.C. trains at higher altitude, his lungs will increase in size to improve ventilation under conditions of reduced oxygen availability.
B. Feedback for selection of the correct answer to MCQ 2 b b. As G.C. trains at the higher altitude, his kidneys will sense reduced oxygen delivery and stimulate erythropoiesis.

Submit
Exactly right. The kidneys are the site of production of the hormone erythropoietin (EPO). When they sense reduced oxygen delivery, they will increase their rate of release of EPO and the higher levels of EPO will stimulate increased rate of production of red blood cells in the bone marrow (erythropoiesis). Please continue to question 3.

You have:
points: 2 C. Feedback for selection of an incorrect answer to MCQ 2 c a. As G.C. trains at the higher altitude, his muscles will sense reduced oxygen delivery and become larger and stronger.

Submit
No. There will be reduced oxygen delivery to his muscles, but that will not stimulate them to become larger. They will compensate in other ways, and you may want to think about that. But this statement does not belong in your answer. It is not correct and it is not related to the regulation of erythropoiesis. Please go back and try again.

You have:
points: 1 a Exercise/altitude-related scenario followed by the second MCQ (MCQ 2), including answer choices and navigation. Student clicks on Submit to select an answer. Note: MCQ 1 established an opening statement to indicate a reduction in oxygen partial pressure at higher altitude, earning 1 point for the student. b Feedback and scoring if the correct answer is chosen. Student clicks on Submit to move to MCQ 3. c Feedback and scoring if an incorrect answer is chosen. Student clicks on Submit to return to MCQ 2 to select an alternate answer.

Student Feedback
Survey data were obtained from 71.9% of students who completed PHS3240 during the 2010-2011 academic year. The survey respondents reported that they had completed either three (7%) or all four (93%) of the exercises. The majority of respondents (86%) were in their third or fourth year of undergraduate study, and while many (78%) reported prior experience with SAQs on 50% or more of their summative exams, for the rest of the students (22%), this was their first attempt at composing written work under conditions of exam-related stress. Figure 2 summarizes student feedback regarding the SAQ exercise content and design. The majority of respondents felt that the online exercises did help them improve their approach to SAQs. Interestingly, there was a rather uniform distribution of students with regard to their perception of the level of difficulty of the exercises. While 58% of students showed partial to complete agreement with the statement that the exercises were sufficiently challenging, the remainder (42%) felt that the exercises could have been constructed to be even more demanding. Students (87%) were especially appreciative of the feedback that was provided for both correct and incorrect answer choices. Finally, the majority of students (78%) indicated they would have welcomed the opportunity to complete even more of these exercises before writing summative examinations.

Student Outcomes
A comparison of student outcomes on SAQs pertaining to the endocrine and cardiovascular systems (exams 1 and 2, respectively) between students writing exams during the 2 yr before introduction of the online SAQ exercises and during the subsequent 2-yr period when they did have access to influences of the level of participation in the SAQ assignments on student MCQ and SAQ outcomes were evaluated using analysis of variance (ANOVA) followed by the Tukey range test to identify statistically significant differences between MCQ mean scores and SAQ mean scores for the three levels of participation. For all statistical evaluations, differences were considered significant at p < 0.05.

Level of Participation
During the 2010-2011 academic year, all of the 144 students enrolled in PHS3240 wrote exam 1. A small number of students dropped the course, resulting in 139 of them (96.5%) remaining to write the subsequent exam (exam 2) during the Winter term. Similarly, 120 (100% of students) completed exam 1 during the 2011-2012 academic year with enrollment dropping to 110 (91.7%) before exam 2. In general, these students showed a high level of participation in the online assignments. Close to 90% (87.5% and 89.2%, respectively, for 2010 and 2011) of students completed the first pair of exercises while studying the endocrine system. However, while 95% of students also completed both of the cardiovascular system-associated exercises during the 2011 Winter term, the level of participation dropped to 82.7% for students given the same assignments during the subsequent academic year. A very small number of students (1.46% and 3.64%, respectively, in each of the two successive academic years) chose not to do any of the exercises at all. In contrast, 84.2% of the 139 students who completed both exams during 2010-2011 participated fully in this aspect of the course, and 74.5% of the 108 students who completed PHS3240 during the second academic year finished all four exercises. Finished! Well done! You can now see the answer summary below and how many points you scored out of a possible 6.
1. You may want to begin by indicating that oxygen is going to be less available at higher altitude. That will set the stage for explaining the compensations that G.C.'s body can make to adjust to this situation before participating in a critical race.

You have:
points: 6 2. The kidneys are the site of production of the hormone erythropoietin (EPO). When they sense reduced oxygen delivery, they will increase their rate of release of EPO, and the higher levels of EPO will stimulate increased rate of production of red blood cells in the bone marrow (erythropoiesis). 3. Higher blood levels of erythropoietin will lead to higher blood levels of red blood cells in an effort to deliver as much oxygen as possible to hardworking muscles under conditions of reduced oxygen availability. 4. Negative feedback will be important. G.C. may benefit from a higher hematocrit when cycling at high altitude, but it would be important for that hematocrit not to climb too high. Hence, when the kidney senses improved oxygen delivery, the rate of EPO release will go back toward baseline levels. 5. An elevated reticulocyte count indicates recent stimulation of erythropoiesis. Reticulocytes are the "almost mature" red blood cells that are released into circulation. Within a couple of days they will become mature erythrocytes. An elevated reticulocyte count, therefore, is indicative of recent stimulation of erythropoiesis. 6. He does need to worry about unnecessary blood clots given that he will have developed mild polycythemia. The slightly increased hematocrit combined with increased water loss when exercising in warm weather makes his blood more viscous, which makes it a little more difficult for it to circulate throughout all blood vessels. There is a slightly increased risk of blood clotting. a This final Web page provides an orderly outline of the key points students should include in their written answer to this question by listing the feedback for each of the six correct answers to the MCQs that formed this exercise.  1 and 4). Description of the patient's symptoms and the identification of her problem as Graves disease stimulates recall of prior learning (event 3) and encourages knowledge application within a clinical context (event 9). Presentation of broad questions at the end of the paragraph informs the learner of the objectives to be addressed in this exercise (event 2).
1. What is your opening statement as you begin to answer this question?
Answer choices Feedback a. This woman needs to be treated for hyperthyroidism.
This is not the best way to start. You should begin by defining Graves disease and describing the physiological basis of its effects on endocrine function. Please go back and try again. b. Graves disease is an autoimmune disease in which the thyroid gland is attacked and destroyed.
This is partly the right way to start, in that you are attempting to define Graves disease, but your information is not completely correct. While Graves disease is an autoimmune disease, it does not result in destruction of the thyroid gland. Please go back and try again. c. Graves disease is characterized by high fever and increased release of stress hormones.
While you have the right idea, in that you should begin your answer by defining Graves disease, your information regarding this endocrine disease is not correct. Graves disease is not an infection (so there is no fever) and it does not target the adrenal glands (so it does not directly cause increased release of stress hormones). Please go back and try again. d. Graves disease is associated with the production of antibodies that mimic TSH.
Yes. This is a good start. You should also use the word "autoimmune" when describing what kind of disease it is and you should spell out what TSH is. Please continue to question 2.
e. Graves disease is a disease that disrupts the ability of the hypothalamus to regulate body temperature.
It is good that you are starting by trying to define Graves disease, but the information you are providing is not correct. Body temperature is increased in a patient with Graves disease, but the effect is not at the level of the hypothalamic thermostat. It targets an endocrine gland. Please go back and try again.

Reasoning:
The question itself provides guidance (event 5) by asking the student to think about the order in which information should be presented. Distractors provide students with information that they should recognize as incorrect (b, c, and e) and address a tendency of some students to struggle with answer organization (a), in this way addressing events 3, 5, and 6. Feedback for the correct answer (d) provides additional guidance (event 5) by suggesting inclusion of the term "autoimmune" in the answer and advising that acronyms must be spelled out in full (a common student error).
2. What are the important pieces of information regarding the thyroid gland that you should include next in your answer? (Choose 2.) Answer choices Feedback a. Thyroid hormone increases oxygen consumption by virtually all cells of the body.
Well done. This is one of the important facts about the thyroid gland that you should include as the next part of your answer. By stimulating increased oxygen consumption by tissue cells in general, thyroid hormone increases heat production by cells, helping us to maintain body temperature. Please go back and continue choosing correct answers to this question until you have two of them. b. Thyroid hormone resets the hypothalamic thermostat to a higher value.
No, this is not one of the functions of thyroid hormone. While the hypothalamus does contain a thermostat, and our body temperature will rise if this thermostat is set to a higher value (e.g., if you have a fever), this is not what is going on in Graves disease. Please go back and try again. c. Thyroid hormone inhibits the body's heat-loss mechanisms.
No, this is not a function of thyroid hormone. While body temperature would rise if we were unable to unload extra heat via peripheral vasodilation and sweating, thyroid hormone does not inhibit these normal heat-loss mechanisms of the body. Please go back and try again. d. Thyroid hormone is released by the anterior pituitary.
No. Thyroid hormone is released by the thyroid gland. The hormone released by the anterior pituitary that regulations thyroid gland function is TSH (thyroid-stimulating hormone). Please go back and try again. e. Thyroid hormone increases the activity of the Na + /K + ATPase in many cells of the body.
Yes. This is an important function of thyroid hormone. It stimulates the Na + /K + ATPase to keep putting Na + and K + back in the right place so that cells continue to be responsive. Running the Na + /K + ATPase requires ATP, so this ties in well with the other important function of thyroid hormone presented in this question: it increases the production of ATP via aerobic metabolism. Please go back and continue choosing answers until you have two that are correct; then proceed to question 3.

Reasoning:
The goal of this question is to demonstrate the importance of including some basic science at the beginning of the answer to show student understanding of thyroid gland function and its relevance to this clinical problem (events 3 and 5). The three distractors provide incorrect information with the feedback stimulating recall of prior learning (event 3). No. Thyroid hormone influences metabolic rate and so hyper-or hyposecretion of this hormone can have influences on body weight. But the rate of thyroid hormone secretion is not regulated by the body's nutritional status. Please go back and try again. b. The production of thyroid hormone requires iodine.
While it is true that iodine is required for the synthesis of thyroid hormone, this is "off-topic" information that should not be included in your answer. It does not relate to Graves disease and it does not relate to the general concept of regulation of thyroid hormone synthesis. Please go back and try again. c. The production and secretion of thyroid hormone is regulated by the hypothalamic-pituitary axis.
Yes. You want to talk about TRH (thyrotropin-releasing hormone) and TSH and how these products of the hypothalamus and anterior pituitary, respectively, regulate the synthesis and secretion of thyroid hormone. You will also want to mention cell surface receptors for TSH on thyroid follicular cells. That will set you up for talking about immune system dysfunction and the production of antibodies that mimic TSH. Please continue to question 4. d. The thyroid gland stores several months' worth of thyroid hormone.
While this is a correct statement, it is extraneous information that does not relate to your answer to a question concerning Graves disease. You want to talk about the hormonal regulation of thyroid hormone synthesis and secretion, not the storage of premade hormone precursor. Please go back and try again. e. Thyroid hormone is secreted during only the cold months of the year.
No. First of all, this statement is not correct. While we do secrete higher levels of thyroid hormone during the colder months, particularly if we are often outside, we also produce and secrete thyroid hormone year-round. Thyroid hormone is needed continually to regulate metabolic rate and to stimulate the functioning of Na + /K + ATPase so that our cells are able to maintain a normal distribution of these cations across the cell membrane and so that our excitable cells remain responsive. Please go back and try again.
Reasoning: Students need to lay the groundwork for subsequent explanation of uncontrolled stimulation of the thyroid gland in Graves disease by first briefly describing the normal regulation of thyroid hormone secretion (events 3 and 5). Two distractors (b and d) provide guidance to students about the importance of being concise and including only relevant information (events 5 and 7) by providing information that is correct but does not pertain to the question being asked.

Where does the concept of negative feedback fit into your answer to this question?
Answer choices Feedback a. In Graves disease, increased body temperature exerts negative feedback to decrease the secretion of thyroid hormone by the thyroid gland.
No. In Graves disease, there is actually hypersecretion, not hyposecretion, of thyroid hormone, and it is the hypersecretion of thyroid hormone that is continually increasing metabolic rate and, therefore, driving body temperature up. Please go back and try again.
b. In Graves disease, there is a loss of negative feedback that normally controls thyroid gland release of thyroid hormone.
Yes. You should talk about the antibodies that are produced by the immune system being able to substitute for TSH at the TSH receptor on thyroid follicle cells. And then talk about how antibody production would not be adversely affected by rising levels of thyroid hormone, whereas TSH production would be. The absence of negative feedback on antibody production would allow the level of stimulation of thyroid follicle cells to continually increase. Well done. Please continue to question 5. c. In Graves disease, there is a loss of negative feedback from the heart on the secretion of thyroid hormone by the thyroid gland.
No. The heart does not produce any hormones that regulate the release of thyroid hormone. While excess thyroid hormone can result in increased heart rate, heart rate does not regulate thyroid hormone secretion. Please go back and try again.
d. In Graves disease, antibodies are produced that bind to thyroid hormone, preventing it from exerting negative feedback to regulate its secretion.
No. Production of antibodies occurs in Graves disease, but these are antibodies that mimic TSH and stimulate thyroid hormone production and secretion. These antibodies do not interact directly with thyroid hormone. Please go back and try again.
e. In Graves disease, nervous agitation inhibits the release of TSH from the anterior pituitary.
No. The nervous agitation is a consequence of the hypersecretion of thyroid hormone by the thyroid gland. While one will see lower levels of TSH in a patient with Graves disease, this is due to negative feedback by high circulating thyroid hormone. It is not caused by nervous agitation. Please go back and try again.
Reasoning: The next logical step is for students to establish the importance of negative feedback in maintaining thyroid hormone levels around a set point. Distractors and their associated feedback help students focus on the role of autoimmune antibodies in Graves disease and their constant and unregulated ability to stimulate thyroid hormone release. This question stimulates recall, provides guidance and feedback, and assesses performance (events 3, 5, 7, and 8).
(Continued) You would actually see lower levels of TSH in a patient with Graves disease, due to excessive stimulation of thyroid hormone secretion by the antibodies that are produced in this autoimmune disease. Thyroid hormone exerts negative feedback on TSH secretion. However, there will still be excessive stimulation of the thyroid gland due to uncontrolled production of TSH-mimicking antibody. This is not a good choice; please go back and try again. b. The antibodies produced in Graves disease have a stimulatory effect on the heart and on the CNS.
No. You need to talk about the effects of these antibodies on thyroid hormone secretion and then link the high levels of thyroid hormone to the symptoms displayed by this patient. Please go back and try again. c. Excessive secretion of TSH has continually boosted metabolism, leading to increased heat production.
You would actually see lower levels of TSH in a patient with Graves disease, due to excessive stimulation of thyroid hormone secretion by the antibodies that are produced in this autoimmune disease. It is the thyroid hormone, not TSH, that has stimulatory effects on metabolism, and thyroid hormone levels will be very high due to the TSH-like antibody. However, the high levels of thyroid hormone will exert strong negative feedback on TSH secretion, so TSH levels will be low. Please go back and try again. d. Uncontrolled release of thyroid hormone continually stimulates metabolism, leading to increased body temperature.
Yes. Because the antibody mimics TSH and because there is no negative feedback to control antibody levels, there is constant stimulation of thyroid hormone release, and the elevated levels of thyroid hormone constantly stimulate aerobic metabolism in body cells. This is one good choice. Please go back and continue choosing answers until you have two that are correct. e. Increased heart rate and nervous agitation are a consequence of overstimulation of β-adrenergic receptors in the heart and CNS.
Yes. The constantly high levels of thyroid hormone will up-regulate β-adrenergic receptors, resulting in excessive stimulation of cardiac muscle cells and CNS neurons by the sympathetic nervous system. Please keep choosing answers for this question until you have two that are correct. Then continue to question 6.
Reasoning: This question allows students to practice applying their new basic science knowledge within a clinical context (event 6). Distractors and the associated feedback have been designed to ensure a thorough exploration of the hypothalamic-pituitary-thyroid axis, in order to enhance retention and transfer (event 9).
6. Finally, what are the treatment options for this woman and why do they have to be so drastic?
Answer choices Feedback a. Administration of an antibody to TSH No, this is not a good choice. Her TSH levels will actually be lower than normal (you should be able to explain why this is the case), and it is not TSH that is causing the hyperthyroidism. Please go back and try again. b. Removal of the thyroid gland Yes. While this seems drastic, there is no way to halt the production of these antibodies; they are not regulated by typical endocrine negative feedback. So the only option is to remove the thyroid gland. You should then talk about how this woman will need thyroid hormone replacement for the rest of her life. Well done. c. Treatment with cortisol to suppress the immune system No. This is a good thought, in that the cortisol should at least reduce the level of TSH-like antibody production by the immune system. But the immune system is important for daily ability to withstand attacks by microorganisms and to police the body for cancerous cells, so one would not want to suppress her immune system for the rest of her life. And there will still be a level of TSH-like antibody production. Please go back and try again. d. Removal of the pituitary gland No. The problem is not at the level of the pituitary gland and, indeed, her pituitary is producing less TSH than normal because of excessive negative feedback. This is not a good choice. Please go back and try again. e. Treatment with an antibody to thyroid hormone While the problem is, indeed, excessive production of thyroid hormone, this is not the approach that is used to treat Graves disease. Likely it would be excessively costly and it would be difficult to give the appropriate amount of antibody to still support normal levels of thyroid hormone. Please go back and try again.
Reasoning: This question, along with the feedback provided for both correct and incorrect answer choices, provides a final opportunity to stress that in Graves disease, thyroid function is no longer under negative feedback regulation (events 3 and 9). a This exercise pertains to the endocrine regulation of metabolic rate by thyroid hormone. The chart includes the SAQ, each of the six MCQs used to help students formulate an answer to that SAQ, the answer choices available for each MCQ, and the feedback for each correct and incorrect MCQ answer. The Reasoning boxes highlight the specific events of Gagné's nine events of instruction that were applied throughout the various steps involved in the creation of this exercise. formative online SAQ assignments revealed no significant effect (p > 0.05) of the exercises on these measurements of student success (Table 4). However, while not significant (p = 0.098), it should be noted that there was a trend for students who had access to the formative exercises to perform better on the endocrine system SAQs in exam 1 (69.0 ± 1.40%) than the students who did not have access to the SAQ assignments (65.3 ± 1.7%). In contrast, student outcomes on the cardiovascular system SAQs (exam 2) were virtually identical (p = 0.907) between the two student populations. When the focus was shifted to only the student populations provided with online SAQ exercises, it was found that the level of participation in formative assignment completion was strongly associated (p < 0.001) with student SAQ scores (Table 5). Student outcomes on the endocrine system (exam 1) and cardiovascular system (exam 2) SAQs were compared with their overall MCQ scores in each of these exams in an effort to isolate an effect of the formative exercises on the ability of students to formulate well-organized, complete, and accurate written answers. With regard to exam 1, the small number of students who completed only one SAQ exercise or completed neither had poorer outcomes (p < 0.05 and p < 0.001, respectively) when answering the endocrine SAQs in exam 1 compared with those students who participated fully in the online SAQ work. With regard to the MCQ outcomes, a significant difference was noted (p < 0.001) only when comparing the group who participated fully in the online work with those who did not participate at all. Similarly, for exam 2 and with regard to both SAQ and MCQ outcomes, significant effects of participation in the online assignments on summative examination scores were noted only when comparing those students who participated completely with those who did not attempt either assignment (p < 0.001 and p < 0.05, respectively). It should be noted when comparing student performance on the SAQ versus the MCQ portions of each of the two exams, while these values were within  Values with identical superscripts are significantly different (p < 0.001). Table 4. Student outcomes on endocrine system (exam 1) or cardiovascular system (exam 2) summative SAQs during the 2 yr before (No SAQ exercises) and the 2 yr after (Access to SAQ exercises) students were provided with online SAQ assignments each exercise contributed only 1% toward the final overall grade. The practice questions directed students to apply HOCS, referred to as generative cognitive processing, as they analyzed specific situations, applied their recently acquired physiological knowledge to an understanding of the problem to be addressed, and justified the concepts they applied or courses of action they suggested (Anderson et al., 2001;Clark and Mayer, 2008). Indeed, a strong case has been made for a role of self-generation of explanations in improving deep learning and understanding by students (Chi et al., 1994), and adult learners have frequently been described as being self-directed and learning best when conditions are task oriented (Knowles & Associates, 1984;Pratt, 1993). An additional consolidation-promoting feature of these exercises was that the final screen seen by students provided a summary of the correct answers to all the questions in that exercise, in this way providing them with a worked example of a good outline to be followed when composing a written answer. The provision of worked examples has been reported to promote learning by allowing a greater proportion of the limited capacity of short-term memory to be focused directly on the learning process itself (Kirschner et al., 2006;Clark and Mayer, 2008). The majority of students reported that they found the SAQ exercises to be helpful, benefited from the feedback, and would like to have access to more of these formative exercises. It therefore did appear that provision of these online formative exercises improved the quality of the learning experience for these students and addressed a need to provide them with sufficient opportunities to participate in activities that would allow them to develop their critical-thinking and problem-solving skills (Crowe et al., 2008;Saunders and Gale, 2012). An important component of higher learning is the development of metacognitive skills-the ability to recognize and reflect on mistakes in order to acquire a deeper level of learning that can be applied to novel situations (Biggs, 1988;Livingston, 1997). The instantaneous feedback provided to students as they worked through the exercises, as well as opportunities to reflect and select a better answer, promoted the attainment of some important metacognitive goals (Livingston, 1997;Mynlieff et al., 2014). While about half of the respondents suggested that the exercises could have been more challenging, it is important to realize that the physiological concepts being addressed in each exercise required similar levels of problem-solving and critical-thinking skills as the SAQs these students would later encounter on their summative exams. For example, the second SAQ formative exercise asked students to identify diabetic ketoacidosis as the cause of symptoms displayed by a patient who arrived at the emergency room and to describe the physiological disruptions responsible for those symptoms, while a SAQ used repeatedly on summative exam 1 asked students to discuss the physiological cause of type 1 diabetes, certain key functions of insulin, and an important test used to monitor the management of a patient with this disease. Both questions targeted content that had been discussed thoroughly during lectures and required students to display an understanding of the key role of insulin in regulating blood sugar levels. However, it is also important to note that the MCQ-based design of the exercises, by taking students stepby-step in the formulation of their answers, would have provided some cues that could trigger information recall. The 10% of each other for those students completing both assignments, student outcomes on the SAQ portion dropped to being lower (p < 0.05) than those on the MCQs by 15-23%, suggesting a stronger influence of participation in the online work on student ability to answer summative SAQs.

DISCUSSION
These interactive exercises allowed students to practice important HOCS associated with the development of well-organized answers to physiology-based questions before writing their summative examinations. While these exercises did still rely on the use of MCQs, these were MCQs that prompted students to take steps in the construction of explanations pertaining to their SAQ answer by planning the answer foundation, evaluating the relevance of various pieces of physiological information, selecting appropriate information to include, and forging links between related concepts, rather than simply selecting a single answer to a fact-based question. Hence, these exercises provided opportunities for students to use HOCS such as analyze, evaluate, apply, and create as they worked their way through the series of MCQs associated with each assignment. The ability of Quandary to support the provision of a limitless number of answer options, to allow one or more correct answer choices, to either reward or penalize students based on the answer selected, and to provide immediate instructive feedback for every possible answer choice allowed the construction of interactive online exercises that guided students toward the logical organization of new knowledge, the creation of appropriate links, and the development of explanatory arguments as they practiced knowledge application within appropriate clinical and real-life contexts (Arneil and Holmes, 2009). While there are conflicting views in the literature regarding the amount of guidance that should be provided to students, ranging from purely constructivist approaches to a defined curriculum supported by didactic lectures, a strong case has been made for the provision of a reasonable level of guidance that is specifically geared toward supporting the cognitive processing needed during that particular learning process (Mayer, 2004;Kirschner et al., 2006). The provision of learner guidance is also a key instructional design principle that forms the fifth of Robert Gagné's nine events of instruction (Gagné et al., 1992). These formative exercises also thoroughly addressed the remaining eight of Robert Gagné's events of instruction by providing feedback-associated opportunities for students to practice both the application of their new knowledge and the organization of their reasoning when doing so. Especially worthy of note was the use of real-life clinical and physiological scenarios to capture attention, motivate learners, and provide contextual relevance. Finally, and with regard to Gagné's seventh and ninth events, the provision of immediate instructional feedback that addressed both the correct answer and the distractors, in contrast to simply an indication of whether the answer chosen is correct or incorrect, has been shown to be important to students and to promote both knowledge retention and transfer (Moreno, 2004;Mason and Rennie, 2008;Guy et al., 2014).
The high level of student participation in these formative assignments demonstrated that students recognized the educational value of the SAQ exercises, despite the fact that sample sizes between the three populations evaluated. Furthermore, when considering the lower average exam scores for students who chose not to complete the online exercise(s), one cannot ignore a very likely inverse association between level of student participation in assignments and motivation to work hard and/or interest in course content when preparing for summative examinations. However, the positive reception of these assignments by students and the beneficial outcomes noted for students who participated fully in assignment completion suggest a value for this online work and that it should be continued and perhaps even expanded.
Finally, why put so much effort into a SAQ portion of an examination and development of student answering skills when that is so much more labor-intensive than computer-graded MCQ examinations? Preparation for an examination that will require students to write, to express their understanding of concepts and their ability to analyze and apply them in context will prompt students to use deep learning approaches (Chi et al., 1994;Scouller, 1998). Furthermore, SAQs have the ability to distinguish between students who can recognize correct answers to MCQs either through memorization of that factual information (surface learning) or a good educated guess after the elimination of some obvious distractors from those students who have acquired an in-depth understanding of that course content and can produce/synthesize their own clear explanations and justifications (Scouller, 1998;Crowe et al., 2008). For example, one study conducted by the author found that a significant proportion of students who selected the correct answer to certain physiology-based MCQs on summative examinations were unable to correctly apply physiological principles to justify that answer choice when subsequently asked to support it with a written explanation (Carnegie, 2006).
In conclusion, online exercises that use a series of feedback-associated MCQs can be used to direct student thinking and provide practice application of HOCS when facing the challenges of teaching large classes by guiding students through the relevant concepts to be explained, linked and applied to the solution of a particular problem. The use of appropriate clinical and physiological scenarios can effectively engage student interest, and the provision of immediate feedback and opportunities to retry questions promotes reflection and an in-depth understanding of physiological concepts, their interrelationships, and their application. Similar approaches would work well for other basic sciences such as biology, chemistry, physics, and pathophysiology, because these online self-testing exercises promote scientific literacy, an important component of university-level education.
exercises were also open-book assignments, and students were prompted to go back and try each question until it was answered correctly. These features, combined with the absence of exam-related stress or time constraints, would have promoted student well-being and achievement as they worked through the questions, leading them to perhaps think that the questions written under these conditions were easier than those encountered during summative examinations (Orr and Foster, 2013). And, most importantly, they were gaining practice in organizing and expressing their physiology-based reasoning.
Interestingly, while most students did report that they found the online assignments improved their approach to SAQs, it was not possible to demonstrate a significant effect of the introduction of these exercises on overall summative SAQ outcomes, although it should be noted that there appeared to be a nonsignificant trend toward improved outcomes for exam 1. It is frequently difficult to show a significant effect of a single modification to teaching strategy on student outcomes when average scores are compared between different years for a number of reasons, many of which apply to the current study (O'Byrne et al., 2008). Class composition changes each year, resulting in student populations with differing levels of motivation, ability, and background knowledge. While the examination questions target similar course objectives, the questions themselves do vary from year to year, and each one may not always be at precisely the same level of difficulty. Student examination schedules fluctuate, allowing the introduction of extra challenges posed by clustering of summative examinations within a short time period and the necessity for some examinations to be written in the evening, when students are tired. And, in this study, average class size increased by more than 50% just before the introduction of the SAQ exercises, and a possible confounding detrimental effect of increased class size on the learning success of some students cannot be ruled out (Lindsay and Paton-Saltzberg, 1987). However, the trend toward improved SAQ outcomes noted for exam 1 is encouraging. One would expect that any benefit gained from being guided in the development of an outline for a SAQ answer and seeing worked examples of answer outlines would be greatest with the first exposure to these exercises and that the effect would become less prominent as students became more accustomed to the process of formulating written answers (Libarkin and Ording, 2012).
In contrast, when focusing on only those students who had access to the interactive SAQ exercises, there was found to be a very strong association between the level of student participation in the online assignments and subsequent summative examination outcomes, particularly with regard to the SAQ portions of the examinations. It should be noted that completion of each exercise not only provided students with practice in problem-solving and answer-organization skills but also prompted them to reflect on course content and to consolidate their understanding of a topic when applying it in context. Hence, both learning outcomes, developing an approach to SAQ answer writing and obtaining practice in the application of new knowledge, may have contributed to the vastly improved student outcomes, especially on the SAQ portion of the examinations. But these data must also be interpreted with some caution. The majority of students completed both sets of exercises, resulting in very different