{'fraction_non_alphanumeric': 0.028038321429595846, 'fraction_numerical': 0.01216189082981958, 'mean_word_length': 5.200160056909123, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 17, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Development of a New Framework to Guide, Assess, and Evaluate Student Reflections in a University Sustainability Course

Many institutions of higher education increasingly place a focus on various forms of experiential education, including personal reflections. While much work has been done in this and related areas, the resources currently available are not sufficient to effectively guide, assess, and evaluate student learning through reflection. Guiding students through the process, assessing their work, and providing an evaluation presents challenges for educators. This article discusses a new framework, a robust rubric, and a guide that students and evaluators can use to support experiential learning through reflection. The framework and resources are based on a grounded investigation of student reflections that were compared to various models from the literature. The framework and resources discussed in the article were developed over a period of five years and with more than 1,600 students. Our purpose here is to describe the development of this framework, to provide a description of the rubric and guide, and to share the lessons learned. This framework and accompanying materials will, we hope, be a useful resource for educators and students wishing to support experiential learning through the use of reflection.


INTRODUCTION
Most people likely understand the role of experience in the learning process in an intuitive way. Learning through experimentation, stumbling upon a great idea while participating in a new activity, or reflecting on the consequences of a mistake are surely universal experiences. That said, it should be noted that experiential learning is not necessarily a direct result of experiential education. Experiential education "is the philosophical process that guides the development of structural and functional learning experiences," while experiential learning "refers to the specific techniques or mechanisms that an individual can implement to acquire knowledge or meet learning goals" (Higher Education Quality Council of Ontario, 2016 p. 18, referencing J. W. Roberts [2012], Beyond Learning by Doing: Theoretical Currents in Experiential Education). Because of its relevance to education, the role of experience in the learning process has long been of interest, and has been addressed by researchers that include, among others, John Dewey, Kurt Lewin, and Jean Piaget (see Higher Education Quality Council of Ontario, 2016; Kolb, 2015). The body of knowledge on this topic is extensive, and covers learning styles, intrinsic versus extrinsic motivation, and surface versus deep learning, as well as tools and techniques for educators, including community-based learning, problem-based learning, and reflective writing. . Development of a new framework to guide, assess, and evaluate student reflections in a university sustainability course. Teaching & Learning Inquiry, 7(1). http://dx.doi.org/10.20343/teachlearninqu.7.1.5

56
The objective of this article is to present the development of a framework and associated resources that can be used to effectively support high-quality reflection through guidance, assessment, and evaluation. The framework, rubric, and guide are the outcome of several years of design and testing as part of a set of university-level interdisciplinary courses on the topic of sustainability. As of April 2018, the framework, guide, and rubric have been used in more than 18 classes and with more than 1,600 students. These materials have now been used by instructors in several faculties at McMaster University and by professionals in other fields. Their feedback has been valuable to refine both the framework and resources, so that they are now in a polished form ready for wider dissemination.

THE ROLE OF REFLECTION IN EXPERIENTIAL LEARNING
It is widely agreed that experience plays an important role in learning (Higher Education Quality Council of Ontario, 2016; Kolb, 2015). According to the Association for Experiential Education (2019), "experiential learning occurs when carefully chosen experiences are supported by reflection, critical analysis, and synthesis." Mezirow and associates explain that "critical thinking is informed by reflection" and use it synonymously with "reflective learning," stating that "[a]lthough it is possible to think without either reflecting or learning, thought that involves critical reflection involves learning" (1990, p. xvii). Thus, while critical thinking is a function of cognition, reflection is a function of metacognition and encouraging it is one of the desired outcomes of reflection (Moon 2006). Anderson and Krathwohl (2001, p. 43) describe the two important aspects of metacognition: "(1) knowledge about cognition and (2) control, monitoring, and regulation of cognitive process." Kuiper and Pesut (2004, p. 384) suggest that "critical thinking is to cognitive skill acquisition as reflective thinking is to metacognitive skills acquisition." This implies that just thinking or having an experience do not necessarily result in learning, but rather critical thinking and reflection support and facilitate the learning process. Although Mezirow (1998) makes the distinction between "reflection" and "critical reflection" in that reflection is looking back on an experience but not necessarily making an assessment of what is being reflected upon, we use the term reflection to imply critical reflection.
The use of reflection in the learning process has been studied in a variety of fields, including health (Mann, Gordon, & MacLeod, 2009), professional practice (Schön, 1987), professional development (Moon, 1999), the arts (Leijen, Lam, Woldschut, & Simons, 2009), and more. Jennifer Moon mentions that "[o]ne of the difficulties of studying the literature on reflection is that it emanates from many different sources" (1999, p. vii), and she highlights some that have attempted to transcend disciplinary boundaries, including Boud, Keogh and Walker (1985), and Mezirow (1990) among others.
Although reflection has been studied from many different perspectives, they all have in common a desire to help learners to better develop knowledge, skills, and abilities. Moon (2006) places emphasis on the role of reflection in learning by identifying some purposes for learning journals, which include "to record experience" (p. 44), "to facilitate learning from experience" (p. 45), "to develop critical thinking skills or the development of a questioning attitude" (p. 46), "to encourage metacognition" (p. 46), "to enhance problem-solving skills" (p. 47), "as a means of assessment in formal education" (p. 47), "to enhance reflective practice" (p. 48), "to enhance creativity" (p. 49), and "as a means of communication between one learning and another" (p. S1). Boud (2001, p. 9) states that journal writing can be a record of events, a form of self-expression, and . Development of a new framework to guide, assess, and evaluate student reflections in a university sustainability course. Teaching & Learning Inquiry, 7(1). http://dx.doi.org/10.20343/teachlearninqu.7.1.5 57 even a form of therapy. He presents journal writing as "a device for working with events and experiences in order to extract meaning from them," in order to "make sense of the world and how we operate within it." He also explains that " [a] s a vehicle for learning, [reflection] can be used in formal courses . . . professional practice or any aspect of informal learning" (p. 9). Schön (1987) believes that "education for reflective practice, though not a sufficient condition for wise or moral practice, is certainly a necessary one" (p. xiii). Mezirow (1998) introduces critical reflection of an assumption and critical self-reflection of an assumption, which can have an impact on one's frame of reference and result in transformational change for the individual.
Moon offers a perspective that "most writers on reflection begin their articles with a preamble that refers to one or two of four writers whose work or models have influenced the manner in which the term is viewed . . . As to which of these writers is chosen usually depends on the angle the writer is taking" (2006, p. 11). For this article, we refer to John Dewey and David Kolb because we discuss the process of reflection based on experience as a form of education and learning. Thanks in large part to the works of Dewey and Kolb, it is widely accepted that experiences form the basis for reflections which in turn can lead to new ideas, new experiences, and learning (see, among others, Boud, 2001;Kolb & Kolb, 2009;Kolb, 2015;Moon, 1999). Through this cyclical process, ideas are formed and re-formed as learning continues (Kolb & Kolb, 2009). In Experiential Learning Theory and the appropriately titled The Experiential Learning Cycle, David Kolb (2015) places the act of reflection as the first step towards drawing meaning from an experience, which helps guide the learner to establish new ideas and to engage in new learning experiences (see the learning cycle, figure 1). Previous work has raised the level of awareness of the role and benefits of experience and reflection within the learning process. Recently, for instance, there have been both theoretical works that develop frameworks for reflection, and explorations of the necessary conditions for quality reflection (Higher Education Quality Council of Ontario, 2016, referencing R. R. Rogers [2001], Reflection in higher education: A concept analysis). Mann, Gordon, and MacLeod (2009) reviewed and synthesized 29 studies of reflective practice in the health professions alone and found several examples of approaches to assess reflective thinking. Highlighting the many ideas that are available to assess learning journals, Moon (2006) provides reference to and a brief description of a handful of examples that can be considered in the development of one's own assessment criteria. Mezirow (1998), after presenting the significance of critical reflection of an assumption and variations on how it is used for different purposes and for different applications in adult education, concludes by stating that "[t]he professional task ahead is to find ways to translate the concept of C [ritical] R[reflection] of A[ssumption] and discourse into curricula or programs, instructional methods, materials development, and evaluation criteria" (p. 197). While there are a number of models of reflection, there is still limited information about how to effectively apply these theories in practice. There is agreement that reflection is best when it is taught and guided by an educator (Moon, 1999;Russell, 2005;Ryan, 2013). Parting from Mezirow's signal of the professional task ahead, significant opportunity exists in the ability for educators to have a deep understanding of reflection and to be able to effectively teach and guide students through the reflective process.

RESEARCH CONTEXT
Education is a priority of governments at various levels and within many jurisdictions. The Canadian federal government, for example, has committed to creating more jobs and greater opportunities for young Canadians. This commitment includes greater use of experiential learning, with an annual budget of $40 million (CAD) to help employers create more co-op placements for students (Liberal Party of Canada, 2015). Likewise, the province of Ontario has emphasized the importance of experiential learning to help develop a highly skilled workforce. A 2016 report, The Premier's Highly Skilled Workforce Expert Panel provides four key recommendations for the province, which included the expansion of experiential learning opportunities. Specifically, the report recommends that "every student has at least one experiential learning opportunity by the time they graduate from post-secondary education" (p. 27). As the discussions at the national and provincial levels were had in consultation with educational institutions, these recommendations are aligned well with the goals of colleges and universities. Progress in this area is notable. According to the Higher Education Quality Council of Ontario (2016), about half of all students now have an opportunity for experiential learning before graduation. In response to a call from the Ministry of Training Colleges and Universities (now the Ministry of Advanced Education and Skills Development), all institutions of higher education in Ontario have prepared a strategic mandate agreement outlining their areas of focus.
McMaster is one of many institutions to include goals of improved learning experience and career preparedness through experiential opportunities (McMaster University, 2014). While commendable, there are inherent difficulties involved in measuring success when it comes to experiential learning. Indeed, as indicated in the agreement, current metrics of success, such as "the percentage of courses that include experiential learning opportunities" (McMaster University 2014, p. 5), are about delivery of opportunities, but not about their impact on learning. To reiterate, providing the opportunity for learning does not guarantee that learning actually happens. Prior to being able to measure such results at the institutional level, instructors must first have effective tools to guide, assess, and evaluate experiential learning at the student level. This research was developed to attend to these needs.
Given the relative scarcity of information on best practices for guiding, assessing, and evaluating learning through experience, the initial stage of the research followed a grounded theoretical approach, and it involved the analysis and coding of student reflections taken from a level-two environmental issues course. The findings from this initial stage, complemented with a review and comparison of the . Development of a new framework to guide, assess, and evaluate student reflections in a university sustainability course. Teaching & Learning Inquiry, 7(1). http://dx.doi.org/10.20343/teachlearninqu.7.1.5 59 literature on the topics of experiential learning and guided reflection, led to the creation of the reflective learning framework.
The reflective learning framework was used in four different academic courses at McMaster from 2013 through 2017 and the winter semester of 2018. The courses selected for testing are all part of McMaster's Sustainable Future Program, or Sustain, courses. This program is an ideal setting for research on experiential learning, given its mission to "inspire in all students a desire for continued learning and inquiry through experiential education" (http://asp.mcmaster.ca/). The program provides opportunities for interdisciplinary, student-led, community-based, and experiential education focused on sustainability, and thus ample opportunity for the use of reflection-based techniques.

METHODS
With the objective to develop a framework for guiding, assessing, and evaluating student reflection that could be effectively used in practice, we employed grounded theory research methodology (see Corbin & Strauss, 1990).
In line with grounded theory approach, the first set of data was a sample of university-level student reflections. Through free-form "open coding" the data were broken down analytically and given conceptual labels. Similar events were grouped to form categories that helped generate questions and inform further analysis. Through the next phase of "axial coding", the categories were tested against the data over and over again. This rigorous and systematic process is integral to the process of grounded theory. Corbin and Strauss (1990, p. 13) describe that "a single incident is not a sufficient basis to discard or verify a hypothesis. To be verified (that is, regarded as increasingly plausible) a hypothesis must be indicated by the data over and over again." The third phase of coding, known as "selective coding," "core" categories were identified and were then compared to existing theoretical models of guided reflection.
Further testing and refinement took place, which led to the development of the first iteration of the reflective learning framework. Following this, and after having identified the need to compare the framework to cognitive processes involved in learning, the categories were compared to Bloom's taxonomy for learning, teaching, and assessing (Anderson & Krathwohl, 2001), upon recommendation from a colleague. Testing and further refinement of the framework took place.
The resulting resources could then be used to facilitate knowledge transfer, and support instructors, teaching assistants, and students in the use of reflection-based techniques. Specifically, the guide provides a concise overview of the justification of each reflection component, and supports the guidance, assessment, and evaluation of student reflections. As stated, the guide is also intended for use by students. In this capacity, it provides direction as they reflect, and can be used for self-assessment and evaluation. The framework, resources, and process for use were continually reviewed by students and educators and were refined based on their inputs. Reviewers have included teaching assistants in the Sustain courses; undergraduate and graduate students at McMaster; academics; a wide range of educators, both within McMaster and elsewhere; and finally, three anonymous reviewers who read an earlier version of this article. Student comments are included here to illustrate the kind of feedback received from them. All student comments were obtained from university-administered course evaluations and were taken from one of four open ended questions: (1) "Please comment on the quality of the TA's in this course"; (2) "Please list aspects of this course that you found valuable and should be continued"; (3) "Please list aspects of this course that might be improved"; and (4) "Additional comments." . Development of a new framework to guide, assess, and evaluate student reflections in a university sustainability course. Teaching & Learning Inquiry, 7(1). http://dx.doi.org/10.20343/teachlearninqu.7.1.5

60
While the process may seem like a "chicken and egg" situation, the sections that follow include an overview of the stages of development of the reflective learning framework and offer additional clarity of how the current version of the framework came to be. A summary of this process appears in table 1 and is followed by a full description of the key stages and then a section on lessons learned and suggestions for use. The framework and associated resources are available at https://asp.mcmaster.ca/resources. Anonymized lifestyle project reflections were given to the first author for exploratory analysis and testing, specifically to determine which components stood out as contributing to a high-quality reflection.

Mid-summer 2013
Literature review and testing using selective coding Using the selected codes from the previous phases, theoretical models from the literature were reviewed and assessed for applicability. The framework by Ryan (2011) was identified as the most applicable to findings from the exploration phase. Ryan's framework was tested against the lifestyle project reflections, but challenges were identified. Building on Ryan's structure, additional components were included to provide greater support to students and instructors. The framework was also used to create a draft evaluation rubric. The evaluation framework was then tested on a sample of the lifestyle project reflections. After some refinement, it became the first version of the framework and was taken forward for consultation and feedback. Version 1.0 development through piloting, formal testing, and consultation Fall 2013 Piloting version 1.0 The framework 1.0 was piloted in a level-three sustainability class of 36 students with good success. However, there was only one reflection, which took place during the exam period, which we learned was not ideal.

Winter 2014
Formal testing of version 1.0 Version 1.0 was further piloted in a level-two sustainability class. A total of 126 students were enrolled in the course and 26 students chose to take part in the initial study, which included three reflections during the course. While the framework was an effective tool, the process for using it to guide and assess reflective writing proved to be challenging.

Winter 2014
Consultation and inspiration for the version 2.0 Through consultation with educators came a recommendation to align the framework with Bloom's taxonomy. During this alignment, some additional revisions were made that also added clarity to the framework. Development of version 2.0 and establishing an effective process for using the framework Early summer 2014 Aligning the framework with Following the recommendations provided, the framework was revised to align with Bloom's taxonomy, which was major evolution, warranting a version 2.0. Whalen Study findings were further assessed and more consultation took place with students, faculty, and staff. At this stage, only minor revisions were made to the framework. However, significant revisions were made to the process for introducing the framework and using it to guide students in the reflective process. Additional testing and refinement of the framework and process for use Fall 2014 to winter 2016 Piloting the new process for using version 2.0 While piloting the new process for using the framework, the first author worked closely with the instructional teams to obtain feedback on introducing the framework, supporting students through the reflective process, providing support along the way, as well as assessing and evaluating the reflection assignments. Fall 2016 and winter 2017 Testing the new process for using version 2.0 Version 2.0 and new process for use was tested in three separate sustainability courses. Of the total 239 students enrolled, 100 agreed to participate in the study. Feedback from the instructional team confirmed success in facilitating the new process for using the framework. Student reflections demonstrate that they have a good understanding of the framework and are able to use it to produce highquality reflections. Summer 2017 Documenting the process The process of developing and using the framework as well as lessons learned were documented in a working manuscript.

Winter 2017
Anonymous reviewer feedback The refined manuscript was submitted for review. With additional feedback from anonymous reviewers and then from members of the instructional teams, the framework was refined and currently stands as version 2.2

DEVELOPMENT OF THE REFLECTIVE LEARNING FRAMEWORK
As described in the preceding section, development of the reflective learning framework was informed by (1) an analysis of university-level student reflections, (2) comparison to previous models for guided reflection, (3) comparison to Bloom's taxonomy for learning, teaching, and assessing, and (4) continuous testing and refinement. Each of these elements is described in detail below.

Exploration through reading student reflections
Prior to the development of the reflective learning framework, an analysis of student reflections was conducted by the first author. During the winter semester of 2012, a total of 350 level-two students enrolled in an undergraduate-level course, Introduction to Environmental Issues and took part in a personal lifestyle challenge, which was based on the lifestyle project of Kirk and Thomas (2003). Through this project, students had the opportunity to learn about their impact on the environment by engaging in a three-week, self-directed, lifestyle change challenge. Students could choose from a list of categories, such as garbage, electricity, or leave the car at home, and kept a journal to record their experiences. Throughout the course, three reflections were submitted for evaluation. A sample of these reflections, which were void of all personally identifiable information, were offered to the first author to support them in gaining a preliminary understanding of student reflections that were loosely guided and . Development of a new framework to guide, assess, and evaluate student reflections in a university sustainability course. Teaching & Learning Inquiry, 7(1). http://dx.doi.org/10.20343/teachlearninqu.7.1.5 62 based on the students' personal experience. These reflections provided the initial data for exploratory analysis, but still without a formal framework for the research.
In order to more effectively guide this initial analysis, we turned to grounded theory. In line with procedures of grounded theory, the goal at this point was to gain a better understanding of the general components that contribute to a high-quality reflection. Identification of these components could then be further refined and eventually used to address two key challenges: (1) how to support students to learn about and practice reflection, and (2) how to effectively assess and evaluate learning demonstrated through reflection. In informal discussions with instructors, it was often stated that "you just know a good reflection when you read it." This is also mentioned in the teaching resources provided by developers of the lifestyle project (see Carleton College 2019), whereby the difficulties of assigning "a numerical grade for something so subjective" are mentioned. Suggested criteria for grading lifestyle project reflections include effort, depth of descriptions, sincerity, and commitment. These challenges are understandable and indicate that a certain level of subjectivity in evaluating reflections is perhaps unavoidable. However, in an academic setting that may include multiple evaluators (concurrently or over time), and where grades hold substantial weight for individual students, instructors face the need to be more prescriptive and intentional in guiding, assessing, and evaluating student reflections. More information on the topic of evaluation can be found in the section on lessons learned.
This initial exploration involved reading reflections and highlighting components that seemed to contribute to a high-quality reflection. Initial trends that emerged included effectively describing the relevant aspects of their experience; thinking deeply about and analysing key aspects of their experience; discussing their initial thoughts and feelings, and how they may have changed; including the impact and/or influence of others; and clearly communicating their learning by providing examples. The outcome of this initial exploration through reading and analyzing a selection of more than 100 student lifestyle project reflections, finding trends, and systematically coding and categorizing the data helped to suggest which components contribute to a high-quality reflection. This process then provided the basis to generate a hypothesis and formulate questions. Three of the main questions that arose were (1) Is it enough to know which components contribute to a high-quality reflection? (2) If we gave students a list of criteria, would they be able to effectively complete a high-quality written reflection? (3) Are there frameworks that already exist that could be assessed and evaluated for use, based on the findings from the initial analysis in this study?
Review, assessment, and testing of theoretical models from the literature At this point, we detected the need for a framework to use in our courses. While we anticipated having to create our own, we were curious to see if there was one that already existed that could be adopted or modified to suit our needs. Armed with the components and findings identified in the preceding stage, the next stage was to conduct a review of the literature up to the date and identify relevant theoretical models to guide, assess, and evaluate reflections. A number of models were identified, compared, and contrasted (Boud, 2001;Grossman, 2009;Kember, McKay, Sinclair, & Wong, 2008;Leijen, Valtna, Leijen, & Pedaste, 2012;Ryan, 2011). The codes and categories identified through grounded theory were compared and contrasted to the above theoretical models.
It was found that some models were impractical for large course settings, lacked concrete descriptions, and/or did not lend themselves well to assessment of knowledge. Because the initial . Development of a new framework to guide, assess, and evaluate student reflections in a university sustainability course. Teaching & Learning Inquiry, 7(1). http://dx.doi.org/10.20343/teachlearninqu.7.1.5 63 exploration focused on identifying components from written work that seemed to contribute to a highquality reflection, the framework that was found to most closely align with the findings from the exploratory analysis was Ryan's (2011) structure for reflective writing in higher education. This structure included four text types as well as a description of the associated elements that should be evident in academic reflection. Table 2 presents Ryan's base structure. It should be noted that Ryan took the base structure further to include specific text structure and linguistic resources. However, Ryan's base structure was of primary value to the development of the reflective learning framework at this point, due to its ability to assess all identified components of a high-quality reflection, the level of detail provided to explain the various components, and its suitability to assess learning.
This framework was tested to determine if it could be effectively used to assess and assign a value to the sample of student lifestyle project reflections. However, its use revealed challenges related to consistency and ease of use. For example, the discussion component of Ryan's framework states that the student "hypothesises about different possible responses, actions and future practices" (2001, p. 104). In practice, these items were often present in reflections, but in various levels of quality. More generally, it was found that blind assessments of the same reflection multiple days apart resulted in different grades: the lack of explicit criteria, it seems, can lead evaluators to create structure through identifying additional criteria to support evaluation, and these criteria could vary by instructor or even the same instructor at different points in time. Thus, while Ryan's framework offered the greatest amount of detail among those considered at the time, it was still not sufficient for an evaluator to reliably and systematically identify evidence of learning within the reflections. Furthermore, if such challenges were faced by an evaluator, they would most certainly be faced by the students as well.
For the above reason, a more robust structure was required to support guidance, assessment, and evaluation, by identifying, providing information about, and describing relevance for individual criteria in the framework. Such structure would ideally support recognition and assessment of each criterion, both by students and evaluators. A new framework was developed from Ryan's framework, by using the core categories that had been previously created, as well as information obtained from other theoretical models from the literature. This framework underwent extensive informal testing before being piloted in an academic course. This testing included assessment and evaluation of a sample of student lifestyle project reflections, the first author writing new sample reflections using the reflective learning framework rubric . Development of a new framework to guide, assess, and evaluate student reflections in a university sustainability course. We also developed the first draft of the guide to the framework to provide more information about each component. We were confident that these resources would be useful for students and educators, but we did not assume that they were a finished product. The framework was still simply a list of components with only limited explanation about each one's importance to learning. Furthermore, at a more practical level, three categories and 10 components were a lot to manage. As an evaluator, it was difficult to recall all 10 components without having to continually reference the framework. In the fall of 2013, version 1.0 was piloted in a level-three sustainability course with 36 students. In this course, students learn about sustainability theory through readings, lectures, and tutorials. Their major project involved working in interdisciplinary teams to tackle a real-world sustainability challenge with the support of a community partner. The students completed one reflection assignment, which was worth 10 percent of their grade and took place during the exam period. Through dialogue with members of the instructional team, we felt like we had good success. However, because students only had one opportunity to reflect and because that opportunity took place after the course was over, we learned that including a reflection assignment so late in the term was a missed opportunity for learning. In the words of one student, "The emphasis on reflection on the course content, particularly for the final project is extremely important and I see great relevance in it! I think greater emphasis should be focused on reflecting periodically during the semester by potentially allocating some time during the tutorial or lecture to give us some time to think and reflect." (Student, Sustain 3A03, fall 2013) This lesson enabled us to revise the course to encourage and support ongoing reflection during the semester and to include two reflections each term, whenever we had the time and teaching resources available to do so. We then launched a formal study with student participants during the level-two sustainability course, which was offered the following term. The major project in this course consists of a self-directed lifestyle challenge where the students aim to reduce their personal impact on the environment and/or enhance their impact on society. A total of 126 students were enrolled in the course and 26 agreed to take part in the study by completing three short surveys and allowing us to use their reflections for research. A key takeaway at this stage was that students experience a level of anxiety about reflection stemming from their uncertainty about how to approach it and/or how it will be evaluated. It was hypothesized that providing more information and guidance to students early on as well as making the process as easy and straight forward as possible would make the process more enjoyable and may also result in higher quality reflections and, therefore, deeper learning. Comments regarding the need for additional guidance included the following: Teach us how to write a structured reflection-this would also help us in other classes (Student, Sustain 3A03, fall 2013) I also liked the reflections, although they were a bit confusing and weren't as reflection-like as other reflections that I had done, in the way that there was kind of a mould we had to follow . . . which seems counterintuitive when were reflecting on our experiences. (Student, Sustain 2A03, winter 2014) [A] class on how to write a reflective piece and a] grant proposal would have been greatly appreciated (Student, Sustain 2A03, winter 2014) Throughout this time, discussion with educators who were experienced with course-based reflections also took place. During these discussions, a recommendation was made to align the reflective learning framework with Bloom's taxonomy, which was then scheduled to take place over the summer months.

Developing version 2.0 and establishing an effective process for using the framework
Aligning the framework with Bloom's taxonomy was a key turning point in the evolution from 1.0 to 2.0. Working with Bloom's taxonomy offered support to ensure all knowledge dimensions (factual, conceptual, procedural, and metacognitive) and cognitive process dimensions (remember, understand, apply, analyze, evaluate, and create) were considered and properly aligned with the framework. Additionally, working within a pre-defined hierarchy and widely accepted framework offered a legitimate way to effectively assess and evaluate learning as well as to more effectively communicate the desired outcomes to the students. Specifically, the major revision for version 2.0 was the creation of two or three specific criteria associated with each component of the framework and its alignment of each criterion with the cognitive processes and knowledge dimensions outlined in Bloom's taxonomy.
This revision helped to remove uncertainty by focusing on cognitive skills demonstrated through reflection. Furthermore, Bloom's taxonomy also addresses the need for institutions to measure and report program-level learning outcomes by enabling instructors to measure and report on courselevel learning outcomes (Higher Education Quality Council of Ontario, 2015). In this respect, Bloom's taxonomy has been considered by many to be the gold standard for developing cognitive process-aligned learning outcomes (Anderson & Krathwohl 2001). It is important to note that one of the criticisms of Bloom's taxonomy is the focus on the cognitive domain, without much attention to the psychomotor and, in particular, the affective domain. Much agreement exists in the importance of affect in learning (Boyle et al., 2007;Kolb & Kolb, 2005;Littledyke, 2008;Sinatra & Pintrich, 2003 Krathwohl (2001, pp. 258-259) address the criticism by stating that, "[the] decision has been justly criticized because it isolates aspects of the same objective-and nearly every cognitive objective has an affective component . . . By intentionally focusing on the cognitive domain, this revision ignores this problem except for the fact, as noted earlier, that the Metacognitive Knowledge category in some respects bridges the cognitive and affective domains." The connection between the cognitive domain and the affective domain can be easily described, in the succinct words of one student: "I found the lifestyle reflection and the reflection assignments really fun. When they are more fun you absorb more information" (Student, Sustain 2S03, winter 2016) In addition, as alluded to above, Bloom's taxonomy, which was first published in 1956, underwent a major revision in the late 1990s (Anderson & Krathwohl 2001). In the earlier version, there were only three knowledge dimensions: factual, conceptual, and procedural. An important aspect of the revised version is the inclusion of the fourth knowledge dimension, metacognitive knowledge. Metacognitive knowledge is the "knowledge about cognition in general as well as awareness of and knowledge about one's own cognition" (Anderson & Krathwohl, 2001, p. 27). As described above, while critical thinking is a function of cognition, reflection is a function of metacognition. This addition to Bloom's taxonomy has been critical for the creation of version 2.0. Given the essentially metacognitive nature of reflection and because it bridges the gap between cognitive and affective domains, appropriate comparisons between Bloom's taxonomy and the reflection data would not have otherwise been possible. Following alignment with Bloom's taxonomy, version 2.0 was further reduced two major categories and eight components. Furthermore, and to allow for scaffolding of reflection skills, each of the eight components were amended with additional levels of depth and detail that would support higher learning through reflection. For example, one component in the version 2.0 encourages students to reference their personal thoughts and feelings. However, by specifying additional criteria connected to Bloom's taxonomy, students are guided to analyze their personal points of view, biases, values, and in tensions, thus further enhancing their reflection. Before aligning the framework with Bloom's taxonomy, many minor revisions were made. However, this alignment called for a distinction from the earlier iteration. As such, we refer to the new version as 2.0, which also came complete with a guide, rubric, and sample evaluated reflection. Once version 2.0 was developed, we also invested time in refining the process for how it was introduced to students and how it was used to help guide them through the process. Attention was given to when and how students were introduced to the framework and to how feedback could be used to help students further enhance their reflection skills going forward. Additional information on this has been included in the section on lessons learned.

Additional testing and refinement of the framework and process for use
Between the fall of 2014 and the winter of 2016, the reflective learning framework and process for use were further refined through experiences using it in three sustainability courses, a level-one, a level-two, and a level-three class. There were lessons learned during this time that led to minor revisions of the framework and its use. We found that two reflection assignments taking place during the course was ideal, as it offered students the opportunity to develop their skills but without putting too much of a strain on educators. A student in the level-one course explained, "I liked how we got to do two reflections and two group assignments this allowed for us to receive feedback and use that feedback to improve on our work. If we did not have two assignments, I feel as if I would not have looked over my first assignment as much. Reviewing your work is a great way to improve yourself academically" (Student, Sustain 1503, fall 2015.
With respect to the process for introduction and guidance, we found that making students aware of the reflection assignment and the framework early in the term is helpful for those students who are eager to learn more or to get a head start. The instructor and/or teaching assistant discuss the assignment and make reference to the framework while reviewing the course syllabus during the first class. The reflective learning framework is also made available to students on the course website. A short formal introduction the framework takes place during tutorial about two weeks before the due date for the first of two reflection assignments. This is found to be effective because the students are conscious that the assignment is approaching.
Once the first reflections are assessed and handed back to students with feedback, a second tutorial on reflection takes place. Students are asked to review the feedback given by their teaching assistant and to come to tutorial prepared with questions. During the tutorial, students engage in an activity to help unpack the framework and enhance their understanding of the reflection assignment. They are given two sheets of paper-one with a sample reflection and the other with a list of the eight reflection components and a short description of each. Students spend about 10 minutes on their own trying to match the reflection components to text in the sample reflection. Basically, we are asking students to test their knowledge by evaluating the sample reflection using the reflective learning framework.
From our experience, there are two components that are most commonly missed in student reflections, cause-and-effect relationships and planning and future practices. Our hypothesis is that cause-and-effect relationships is the component most difficult to understand simply by reading the reflective learning framework and planning and future practices is not often considered because people tend to think of reflection as only looking back rather than looking ahead. For these two components, we intentionally omit them from the sample reflection and instead ask the students to put themselves in the author's position and create a few sentences on how they might satisfy each of those two components. Students then share their answers with a peer before the teaching assistant takes up the activity and facilitates a class discussion. These tutorials are lively with discussion and seem to support students' understanding of the reflective learning framework. In the words of one student, "I also really appreciated that we did the reflection activity in tutorial after we handed in the first reflection. I felt that I got more out of the exercise by already knowing what I wrote about in my reflection and going through the questions, rather than answering the questions and I would not have known how it would help me in the future" (Student, Sustain 1S03, fall 2016). Following the second reflection, students are asked to review feedback provided by their teaching assistant and to seek additional support and/or address any concerns during pre-scheduled office hours, which again take place in advance of the exam period. Once the process for using the reflective learning framework was refined, it was then appropriate to conduct additional formal research with student participation.
The original ethics application was amended with some minor revisions to the study design and the revised framework. Three courses were used for the study, the level-one, level-two, and level-four sustainability class. As discussed, the level-three class only has one reflection assignment, and therefore it did not align with our study design. Of the total 239 students enrolled in the three courses, 100 agreed to participate in the study by providing feedback and making their reflection activity sheets and reflection assignments available for research purposes. . Development of a new framework to guide, assess, and evaluate student reflections in a university sustainability course. Teaching & Learning Inquiry, 7(1). http://dx.doi.org/10.20343/teachlearninqu.7.1.5

68
Through discussion with teaching assistants and instructors, the reflective learning framework and refined process is proving to be an effective tool to guide, assess, and evaluate student reflections. However, further development will surely take place as we continue to learn through our experience and through consulting with others. From our experience during the most recent round of testing, we learned more about the importance of feedback. In particular, we learned about the importance of how written feedback from teaching assistants is articulated, who is providing the feedback, and how quickly feedback is given. Some examples of how students expressed this include the following: It would be much more helpful if the comments on our assignments and reflections came with a rubric and useful comments so that students can understand where they went wrong in their writing. (Student, Sustain 1S03, fall 2016) However, the fact that our written reflections were marked by a different TA was very uncalled for. I am writing a reflection that caters to my TA and their style, but then I get horrible mark from a separate TA. What the heck is this nonsense? Feedback was very poor on my written reflections as well. I think the TA that marked my paper should be more open minded to the work they are marking. (Student, Sustain 1S03, fall 2017) Reflection 1 feedback was given pretty close to Reflection 2 due date. If we receive feedback earlier it would allow me to take the comments into consideration and improve on my subsequent reflection. (Student, Sustain1S03, winter 2018) There are inherent difficulties in marking a large number of written reflections in a short amount of time, which may lead to giving more direct feedback that may come across as negative or terse. Furthermore, we identified a lack of training given to both teaching assistants and instructors, which does not seem to be unique to the department or faculty. Furthermore, reflection assignments are highly personal and additional care must be taken to how feedback is provided. Examples from student feedback include "Reflections were marked unnecessarily harshly, with little feedback to offer an explanation as to why the mark was so low" (1S03, fall 2015), and "I was disappointed in having to write about my feelings and were deemed 'wrong"' (Student, Sustain 2A03, winter 2014). This extends beyond the current scope of this research but has been identified as a necessary topic to address in future. Training is now offered to all teaching assistants at the faculty level, but we have not assessed the training to determine if it addresses our concerns.
In addition to learning about the importance of feedback, we also learned that, while the support provided to students have been welcomed and perceived positively, students do not always make their way through the framework. This can be demonstrated by student feedback: "It would be great to have a sample reflection of someone in the past to look at etc. to have a good idea. Many times I didn't know how to go about writing and what format, although the rubric was incredibly helpful" (Student, Sustain 2S03, winter 2018). The comment demonstrates both the appreciation and usefulness of the guidance provided, but also that they did not read the entire framework. There has been a sample reflection provided in every iteration of the framework since its creation. However, it has always been placed near . Development of a new framework to guide, assess, and evaluate student reflections in a university sustainability course. Teaching & Learning Inquiry, 7(1). http://dx.doi.org/10.20343/teachlearninqu.7.1.5 69 the end of the guide and it is left up to the students to read the framework on their own time. (We discuss this in greater detail in subsequent sections.).
Following submission of this article for external review, valuable feedback provided by the anonymous reviewers led to meaningful enhancements to the framework. While the feedback was valuable, we were encouraged that the recommendations were relatively minor, speaking to the quality of the framework in its current state, 2.2.

THE FRAMEWORK
As described above, version 1.0 was mainly based on the work of Ryan (2011) and further developed, first to address some of the challenges faced when applying it, in terms of consistency and ease of use, and later to align it with Bloom's taxonomy. Bloom's taxonomy helps educators categorize learning objectives, which is important for a variety of reasons, including helping educators to see the objectives from the student's point of view, to see the relationship between knowledge and cognitive processes of the learning objectives, and to see the relationship among objectives and how they are taught and how learning is assessed (Anderson & Krathwohl, 2001). Bloom's taxonomy includes four knowledge dimensions: factual, conceptual, procedural, and metacognitive, and six categories of cognitive processes (see figure 2). The following categories of cognitive processes are listed from those that are "most commonly found in objectives" at the bottom to those that are "less frequently found in objectives" at the top (Anderson & Krathwohl, 2001, p. 30): remember, understand, apply, analyze, evaluate, and create. Anderson and Krathwohl (2001) also refer to those categories further along in the list as having a "higher level of complexity" (p. 34). Under each cognitive process category are two or more cognitive processes. For example, the category "remember" includes two processes, recognizing and recalling (see figure 2).  The current version of the reflective learning framework is broadly divided into two categories with a total of eight reflection components: The organization of the framework is roughly based on the levels of cognition required. Each reflection component is broken down into either two or three specific criteria directly related to a . Development of a new framework to guide, assess, and evaluate student reflections in a university sustainability course. Teaching & Learning Inquiry, 7(1). http://dx.doi.org/10.20343/teachlearninqu.7.1.5 71 cognitive process and knowledge dimension as outlined in Bloom's taxonomy. The reflection category of "recount" includes more lower-level cognitive processes, which correspond to Bloom's taxonomy for remember, understand, and also includes analyze. While Boom's taxonomy also includes the category apply, none of the framework's components aligned with that specific category. The discussion category includes more higher-level cognitive processes categories, which correspond to Bloom's taxonomy for analyze, evaluate, and create. The cognitive process category analyze is included in both the recount and discussion components. It is not to say that recounting requires only lower level cognitive processes, rather it requires more of the lower level cognitive processes in general.
When reviewing the framework, it should be evident that this division of main categories, reflection components, and individual criteria is consistent with the grounded theory process, which we think should support broad application for use by others. However, it is important to note that, during development of the framework, we use Bloom's taxonomy to categorize learning objectives of the reflection activities in four related course all within a particular academic program. The objectives of the reflection activity have been largely influenced by years of study using a grounded theoretical approach. As such, if applied elsewhere, the framework may need to be revised based on the objectives of another educator and/or course. Anderson and Krathwohl describe that Bloom's taxonomy "can be used to categorize objectives, provided that the person or persons doing the categorization make correct inferences. Because inference is involved and because each person may have access to different information, individuals may disagree about the correct classification of an objective" (2001, p. 34). As such, we encourage those looking to use the reflective learning framework for their own instruction to apply their knowledge of their objectives, make their own inferences, and revise as necessary.

LESSONS LEARNED AND SUGGESTIONS FOR USE
The reflective learning framework has been used in the Sustainable Future Program and has gone through multiple revisions and produced two main iterations, and each time the tools were used, new lessons were learned. The Sustainable Future Program is relatively new, having started with one course in the winter 2013 semester. The university and the instructional teams, which include instructors, teaching assistants, and program administrators, are highly focused on continuous improvement at the assignment, course, and program level. Each course offering is an opportunity to proceed with data collection and analysis to identify strengths and areas for improvement. The reflection component is no exception. Learning takes place in each course and revisions in the framework or its use have taken place at least once annually. The main lessons drawn from these experiences have been distilled in the form of suggested use of the framework and accompanying resources. The following recommendations are intended to support the use of the framework.

Tailor the framework to support the specific application
The reflective learning framework has been designed while keeping in mind its general applicability. A primary goal has been to ensure that the categories, components, and criteria are general enough to be used for various applications. That being said, development and piloting of the framework as part of McMaster's Sustainable Future Program has no doubt imparted a certain flavour to it. To be . Development of a new framework to guide, assess, and evaluate student reflections in a university sustainability course. Teaching & Learning Inquiry, 7(1). http://dx.doi.org/10.20343/teachlearninqu.7.1.5 72 effective more broadly, the framework may need modifications to suit the nature, context, and level of study for specific applications.
As an example, it is possible to note that the level of cognitive processing required increases with each reflection component. For instance, "important aspects of the experience" requires a higher level of cognitive processing than "temporal progression", and so on. As such, the associated marks for evaluation are higher for discussion components and lower for recount components. For ease of use, each discussion component was given a value of 6 and each recount component was given a value of 3. Instructors may choose to value each component differently, based on various aspects such as the instructional goals for the course or students' prior knowledge.

Share the framework, guide, and rubric with students
Sharing the framework and resources with students can support self-and peer-evaluation, as well as help to clarify expectations. Posting the framework and resources online and including details within the course syllabus are good opportunities to share these resources with students early on in the course. Findings by Andrade (2001) show how sharing instructional rubrics with students helps them to understand and identify qualities of good work, supports them in producing good work, and is a key part in providing helpful feedback to students.
In The Sustainable Future Program, in addition to sharing the resources with students online, a teaching assistant is instructed to deliver a tutorial on reflection approximately two weeks before the first reflection assignment is due. This timing is close enough so that students have started thinking about their reflection, but still provides ample time to offer early guidance about expectations. This tutorial ensures that students are aware of the framework, provides a review of the framework, and encourages self-and peer-assessment.

Provide opportunities for multiple reflections
Initially, students were only given one opportunity for reflection. The reflection was due at the end of the course during exam period. As a result, students were not able to ask questions, learn from their first experience, or to develop their skills through a second application. Providing students with the opportunity to submit at least two reflections, for instance one mid-term and another at the end of the term, is recommended. Furthermore, ensuring the reflection is due and feedback can be provided before the end of the semester is also recommended to allow students to review and discuss strengths and areas for improvement. Ambrose et al. (2010, p. 141) state that "[t]he full benefits of feedback can only be realized when the feedback adequately directs students' subsequent practice and when students have the capacity to incorporate that feedback into further practice."

Use the framework and guide to support self-and peer-evaluation
Students are often surprised to see their reflection assignment evaluated with such rigor, and they usually have questions about their grade.
This provides an excellent opportunity to encourage students to review the guide and use the rubric to undertake a self-evaluation of their reflection to support the discussion with their instructor about how their grade was awarded. In many instances, once a student conducts a self-evaluation to identify specific areas of concern, they either find the answers on their own or they provide the opportunity for a more effective and focused discussion with the instructor. Moon promotes the use of informal peer and self-assessment on reflection work and states that "[a] learner's ability to assess or evaluate their own work is an important skill to be gained in higher learning" (1999, p. 211).
To support self-and peer-evaluation, and to further assist students in developing their reflection skills, starting in the second pilot, teaching assistants in the Sustain courses are instructed to facilitate a tutorial activity approximately one week after the first reflection marks have been released to students. This has taken the form of an activity sheet where students receive a sample reflection that they must evaluate using the reflective learning framework. To support individual and group learning, students receive two different coloured pens. They work individually with one colour to complete the activity sheet, and then they get into pairs to discuss their responses. Before the tutorial concludes, the teaching assistant reviews correct answers with the class and clarifies questions. Students use the alternate colour pen to make revisions to the activity sheet where necessary. It should be noted that this activity has been piloted in various applications: initially, the activity took place before the first reflection, and later the process was revised so that it took place after the first reflection grades had been returned.
When the activity is facilitated after the first reflection, teaching assistants report that students are highly engaged and they receive positive response from students. A reason for this might be that students already had a previous experience with reflection and also time to reflect on the outcomes of the assignment. Consequently they are more interested and incentivised to learn and improve on the next reflection. Ambrose et al. (2010) describe how "research has shown that adding structure and support-also called instructional scaffolding-to a practice activity in or out of class promotes learning when it helps students practice the target skill and at appropriate level of challenge" and further connect this to "Vygotsky's Zone of Proximal Development, which defines the optimal level of challenge for a student's learning in terms of a task that the student cannot perform successfully on his or her own but could perform successfully with some help from another person or group" (p. 132). In discussing content and timing of feedback, Ambrose et al. recommend providing targeted feedback that is earlier than later and more frequent than less, in general. One primary implication to effective feedback is that it is "provided at a time and frequency when students will be most likely to use it" (p. 143).

Remove uncertainty by focusing on knowledge and skills demonstrated
During the initial pilot of the framework, the instructional team noticed a very large range in reflection grades being awarded whereby some students received failing grades while others received nearly full marks. Upon informal review, the students who received high marks had clearly consulted the framework and spent much more time and effort in completing the reflection assignment. The learning demonstrated through these reflections was of very high quality and included many examples that were found to be surprisingly insightful. However, those reflections that received lower grades were superficial in nature and resembled a record of events that had taken place, rather than an exercise in evaluating and making meaning from their experience.
To gain perspective on this observation, we consulted other members of the instructional team who did not have a deep understanding of the framework specifically, because they were not responsible for teaching or grading the reflection assignments, but were experienced and highly respected in their educational roles. In facilitating this investigation, the instructional team participated in an activity to help increase the accuracy of grades awarded by using the framework. Seven reflections, which included a range in quality, were sent to three members of the instructional team. The reflections were void of grades and comments. The instructional team members were asked to provide a general grade for each . Development of a new framework to guide, assess, and evaluate student reflections in a university sustainability course. Teaching & Learning Inquiry, 7(1). http://dx.doi.org/10.20343/teachlearninqu.7.1.5 74 reflection. The reflection grades provided by the selected team members were all very close to the grades awarded through the use of the framework. However, the higher grades were closer and the lower grades were further apart.
The members of the instructional team did not apply any failing grades to students. However, through using the reflective learning framework, there were students who received failing grades. When prompted, the instructional team members agreed that while the poor reflections deserved a failing grade, they felt badly about giving a poor grade on an assignment that was so personal to the student author and also very subjective. However, as a result, the student was not being provided with valuable feedback or guidance on how to improve. As a result, a personal, value-based challenge surfaced throughout this initial phase of the research, which stemmed from the idea that the framework was developed to evaluate something that has personal significance to the individual student. It was difficult to justify giving a student a poor grade when they had clearly expressed deep and meaningful emotions throughout their reflection. Reflections that were eloquently written and articulated deep emotions about the student's experience were a pleasure to read and also tugged on the heartstrings. However, there were examples where eloquent and heartfelt reflections could be lacking in higher-order thinking. These examples proved to be the most challenging for an evaluator who is trying to both foster and accurately assess for deep and critical reflection.
For those instructors wanting to evaluate student reflections for grades, as we do in the Sustainable Future Program, the reflective learning framework helps to remove subjectivity, offer specific guidance to students on how they can improve, and justify grades awarded. However, it is important to note that there is a case against grading student reflections. The risk of awarding a low grade to well-written, emotional, and positive reflection is that it may stifle the student's desire to continue to learn through reflection.
As an argument against formally evaluated reflections, Kaufman (2013) discusses the type of environment that would support or inhibit reflexivity, supporting "free writing" whereby students don't edit out "bad writing" or "unacceptable thoughts and feelings" (p. 73). Kaufman also supports anonymous reflections, arguing that " [i] f students are worried about how they will be judged based on their written responses, then it is unlikely they will get to the point where they can engage in a sympathetic, yet introspective, analysis of their positions and situations" (2013, pp. 73-74). Additionally, regarding feedback provided to students, Moon states that "[f]eedback on work can be a red slash or a helpful comment that demonstrates empathy with the work and its producer" and that "the quality of feedback is in danger of degradation in the face of burgeoning student numbers" (1999, p. 211).
Given this argument, some instructors may choose to take a different approach by using the framework to guide reflection but refrain from formal evaluation. Relating to the comment by Moon, instructors with large classes and without sufficient resources to guide, assess, and evaluate student reflections, may choose a different model. However, identifying this difficulty sheds light on the challenge faced by instructors when evaluating reflection assignments and provides further value for the use of the framework. This issue was addressed in version 2.0 after aligning the framework to Bloom's taxonomy (using Anderson & Krathwohl, 2001). While it is recommended for instructors to make necessary revisions to support their specific course or purpose, ensuring a focus on knowledge and skills demonstrated to support deep and critical reflection is highly encouraged, both to help remove uncertainty and support student learning and skill development.

SUMMARY AND FUTURE RESEARCH
After multiple years of testing and revisions to the framework, we now feel confident that the reflective learning framework is a robust tool for guiding, assessing, and evaluating reflections, and is ready for broader dissemination. Given the novelty of the framework, it is clear that there are multiple avenues for future research and development. For instance, we understand that reading a formal guide about reflection is not the most attractive method available. Given the wide array of instructional resources available, there is an opportunity to translate the framework and guide into tools that are engaging, interactive, and enjoyable for both students and educators. These resources could additionally be evaluated based on the perspectives of instructors, teaching assistants, and students. By providing a detailed and extensive set of criteria for assessing reflection, we hope that the framework will result in consistent evaluations. The question of inter-rater reliability, however, remains open, and a more systematic study of this issue could help us to understand to what degree we have achieved this goal. Related to the initial question of this research-does reflection as an element of experiential learning inspire a desire for continuous learning and inquiry?-additional research could include a longitudinal study of Sustain students throughout their undergraduate career as well as beyond their graduating from university. This study could compare against a control group of students who had not taken part in courses that had reflection and experience. Through conducting this research and sharing the findings, we hope that educators will be provided with additional resources to guide student reflections and assess them for learning; a method for consistently assessing learning through experience, moving away from metrics on providing opportunities and towards metrics that evaluate learning; and ultimately more opportunities for experiential learning in higher education.