{'fraction_non_alphanumeric': 0.0291762437903166, 'fraction_numerical': 0.008897456810261734, 'mean_word_length': 5.448721013626584, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
How to Architecture Open-ended Problem-solvers? Student Reflections on Scaffolding Experiences in an Open-ended Problem-solving Bioengineering Course Sequence

Through reflection, learning experiences that students most-frequently valued during open-ended, scaffolded problem solving in a bioengineering two-course sequence were identified in this study. Reflective knowledge of this type can inform instructors in orchestrating experiences to scaffold problem solving of this kind and were useful in directly demonstrating students’ development in problem solving. Reflection is necessary for learning and is an important aspect of scaffolding. The literature has made recent calls for the additional use and study of reflection and scaffolding within STEM. This paper is intended to respond to these calls by focusing on students’ reflections about the experiences they most-frequently valued during scaffolded work. Students reflected at multiple points about their scaffolded, problem-solving experiences. These reflections were systematically content-analyzed for the experiences valued by students for their learning and development. These valued experiences can be the focus of instructional and scaffolding efforts for open-ended problem-solving in similar courses.


Introduction
In order to prepare students to solve complex, open-ended, workplace problems, engineering programs must provide experiences that equip students to tackle such problems post-graduation (Jonassen, Strobel, & Lee, 2006). In a junior-level biosignals laboratory, which is the first part of a two-course sequence involved in an ongoing study of open-ended problem solving, students must complete three group-based, open-ended projects, in which they conduct in silico experiments to analyze unknown systems using time-and frequency-domain techniques. For example, one realistic, ill-structured problem involves the analysis of a model of human postural control. This study of scaffolding began several years ago with the ultimate goal of enhancing students' open-ended problem solving, motivated by the instructor's observations of student difficulty and apprehension in solving these types of problems.
As suggested in the literature, when learning to solve difficult problems, students need formal support, or scaffolding, which consists of multiple instructional components (Jonassen, 2011). Scaffolding is offered temporarily to relieve the load while students gain the necessary skills to solve complex problems, with the instructor assisting with parts of the task that the student cannot yet handle. As discussed in the literature, scaffolding takes the form of actions such as apprenticeship-like guidance and just-in-time assistance during problem solving, questioning to drive metacognitive and reflective thinking, and written and verbal instructor feedback (Clark & Mahboobin, 2018). Based on strategies from the scaffolding literature, assignments that guided problem decomposition and asked students to review worked programming examples before class were used. In addition, live questioning during active problem solving was used to monitor student progress; and prompts and questions were embedded within laboratory materials for student self-regulation and reflective/metacognitive thinking. Students submitted systematic progress reports for monitoring, timely feedback, and formal guidance from the instructor before the final written report was due. Students were required to support and justify their solutions and assumptions as part of their progress reports, which were returned by the instructor within 24 hours.
A comparison of project scores across several semesters uncovered statistically-significant improvements in performance associated with the scaffolding that was offered (Clark & Mahboobin, 2018). In addition, qualitative-based post-scaffolding assessment with these same students in a follow-up course (without the formal scaffolding), which uses laboratory experiences to illustrate principles taught in several bioengineering core classes, determined the scaffolding introduced in the biosignals laboratory to be helpful for and applicable to similar laboratory modules in the follow-up course (Clark & Mahboobin, 2018). The preliminary results are important for STEM students and instructors encountering challenges with open-ended problem solving, and they begin to fill a gap recently called out by the STEM scaffolding literature (Clark & Mahboobin, 2018). In the current paper, this study of scaffolding with a focus on student reflections is continued. Via reflection, students identified experiences they valued during scaffolded, open-ended problem solving. Reflection is an additional means to scaffold such problem solving. Students' reflections were systematically analyzed to determine student valuation and potential best practices for scaffolding problem solving of this nature, and this can similarly be done by other instructors for their specific coursework and curriculums involving open-ended problem solving. In addition, a goal was to assess students' development in problem solving via their reflections. Thus, the research question was as follows: Via reflection, what problem-solving experiences did students identify as most valuable, including for future coursework or endeavors, and are these reflections useful in directly demonstrating development in problem solving?

Literature Review
Several theories describe the importance of reflection for learning and/or professional practice (Kolb & Kolb, 2009;Schön, 1987), and there have been recent calls for additional use of reflection in engineering education as well as more scholarly research and publications on reflection (Ambrose, 2013;Turns, Sattler, Yasuhara, Borgford-Parnell, & Atman, 2014). This paper is intended to respond to these recent calls. With Experiential Learning Theory, learning occurs through a combination of doing as well as reflecting on the doing (Kolb & Kolb, 2009). Reflection is also key to professional practice under the Reflective Practitioner Theory as it equips professionals with the skills needed to deal with complex problems in the world (Schön, 1987). Reflection is particularly important for designers, who often practice reflection-in-action (as they design) in response to surprises with complex scenarios, with a reflective back-talk, or conversation, that occurs between the designer and the situation, perhaps assisting in deeper understanding of the design problem (Adams, Turns, & Atman, 2003;Schön, 1987). Adams et al. call for bringing Schön's theory of the Reflective Practitioner more fully into engineering design education.
Likewise, Ambrose has called for engineering curricula with opportunities for reflection to connect thinking and doing, and the development of students' metacognitive abilities to foster self-directed, lifelong learning skills (Ambrose, 2013, p. 16). Regular reflection plays a critical role in the construction of metacognitive knowledge and self-regulatory skills, or planning for, monitoring, and evaluating one's own learning, knowledge, and skills (Schraw, 1998;Steiner & Foote, 2017). Metacognition is thinking about thinking, or knowing about knowing, and a metacognitive student is one who is aware of their learning processes and who can adjust these processes as needed (Steiner & Foote, 2017;Turns et al., 2014). Metacognition is important because it supports the development of lifelong learning skills needed to thrive and excel in the workforce (Ambrose, 2013). Ambrose maintains that students learn by doing, but only when they spend time reflecting on the doing.
Turns and Atman recently began their work in the fall 2017 on an NSF-funded study (Reflection in Engineering Education: Advancing Conversations). They stated in their abstract that …within engineering education, reflection and reflection activities are understudied (Turns and Atman, NSF Award 1733474). However, one discipline that has been able to make frequent use of reflection after active-learning experiences is nursing education, where reflective debriefings are done after human patient simulation to drive clinical judgment, in alignment with Kolb's Experiential Learning Theory (Beyer, 2012;Dreifuerst, 2009;Fanning & Gaba, 2007;Lestander, Lehto, & Engström, 2016). Should engineering education learn from the nursing education discipline in this regard?
Reflection and self-monitoring are particularly important during scaffolded work so students can adjust their performance as needed and internalize new knowledge (Collins et al., 1989;Hogan & Pressley, 1997). Prompting students to reflect can also improve the transferability of the content and skills they learn and their abilitity to solve problems (Ertmer & Glazewski, 2015). A pedagogy and culture of "reflection for learning" has been adopted by others for experienced-based subjects in higher education, whereby students were required to engage in reflective practice, including team-based reflections (Harvey et al., 2016;Miller & Maellaro, 2016).

Method
The full details of the biosignals laboratory component of the two-course sequence were previously described in (Clark & Mahboobin, 2018). A synopsis is provided here for completeness. The laboratory was conducted using elements of both flipped instruction and problem-based-learning (PBL). For several of the laboratory sessions, students were instructed to arrive prepared to actively engage in coding by reviewing material provided by the instructor beforehand, followed by an online quiz, as in a flipped classroom setting. This material contained worked coding examples, one of the scaffolding mechanisms discussed in the literature. The three in silico experiments (i.e., projects) provided the focus for learning as in a PBL setting. Laboratory time was dedicated to problem solving by student teams in support of project completion. During the laboratory, the instructor and several teaching assistants continuously circulated the room providing as-needed coaching to students, observing progress, and probing for issues, as in a flipped or PBL classroom. The students asked many questions as the instructional team circulated and observed. The instructional team prompted students with questions to assess their progress and scaffold their problem-solving efforts. Each team had the opportunity to submit three progress reports (or reviews) per project. These progress reviews prompted each team to explain its proposed solution, provide an argument or justification for it, display graphs and figures, ask questions and raise concerns, and in general report on project progress. The progress reviews were neither mandatory nor graded and were intended as a feedback and communication mechanism between the students and instructor.
In continuing the investigation during the fall 2017 semester, students were asked to reflect on the value of the experiences offered to them in the biosignals laboratory. These could have included scaffolding experiences such as in-class problem solving with instructor guidance and interaction, incremental and progressive technical writing, systematic progress monitoring, and timely written guidance and feedback from the instructor. To obtain students' reflections, both written and verbal (i.e., reflective interview) prompts were used. Asking students to reflect via two methods about their experiences enabled triangulation of the data. The question in Table 1 was embedded within a progress report associated with the third project and was also posed to students during individual interviews between the midpoint and end of the term. This question allowed students to evaluate their experiential learning and discuss knowing about their knowing. Although posed somewhat later in the semester, the students could still use these insights to monitor their learning and adjust their practices if needed as they proceeded towards the end of the term. An additional prompt embedded within two of the progress reports asked students to reflect on any breakthrough or particularly memorable moments related to the successful completion of their first two projects (see Table 2). This question was summative in nature since it was posed after completion of the first two projects; however, insights gained could still be used for self-regulation by the students as they proceeded towards completion of the third and final project. This question was modeled upon questions developed to elicit engineering student reflections via emotional-based prompts that can trigger the recall of learning moments (Walther, Sochacka, & Kellam, 2011). All of the students who were in the biosignals laboratory in the fall 2017 took a follow-up course in the spring 2018, and a subset of them participated in a laboratory module (Balance Module) co-taught by the same instructor. In this follow-up course in bioengineering methods and applications, students were able to physically conduct experiments in five modules of their choosing and were required to produce written reports for each module, similar to what they did in the biosignals laboratory. The assessment analyst interviewed the students in their teams of four, resulting in discussions with 24 students (total class size was 83 of which 24 selected and participated in the Balance Module). The question in Table 3 was posed to students after the completion of their Balance Module. Similar to the first prompt discussed above, this question allowed students to evaluate their experiential learning. • What experiences (if any) in the laboratory portion of the biosignals and systems course are helping you in the present course or in other courses, endeavors, or areas?
• Examples of experiences are problems you were presented with, various types of reporting and reports, feedback, group work, etc.
The instructor and assessment analyst jointly content-analyzed all of these reflections (after the semester was over) using a qualitative coding scheme developed as part of their previous work (Clark & Mahboobin, 2018). This coding scheme is provided in Table 6 and was used to content-analyze all of the student reflections in this paper. Although the responses were double-coded via consensus discussions, the first-time inter-rater reliability for prompt 1 was Cohen's kappa = 0.82, which indicates strong agreement beyond chance (Norusis, 2005). For prompt 2, they achieved Cohen's kappa = 0.68 using the coding scheme, indicating good agreement beyond chance. Finally, for the prompt posed in the follow-up course, the assessment analyst and course instructor were able to achieve a first-time inter-rater reliability of Cohen's kappa κ = 0.75, also suggesting good agreement beyond chance.

Interviews and Written Reflections: The Value of Student's Experiences
Students were prompted to reflect on the value of the experiences they were exposed to in the biosignals laboratory on two occasions -via interview and in writing. Thirty-four students were interviewed, representing 41% of the class. In addition, written responses were received from 7 of the 21 project groups. In content-analyzing the responses, the percentage of interviewees and project groups (i.e., for the written responses) who discussed various categories or themes related to the perceived value of their laboratory experiences could be determined. The results are shown side-by-side in Table 4.
Since these were open-ended questions in which students freely provided responses that came to mind, the percentages do not represent yes versus no responses for a category. Also, a student may have mentioned more than one helpful experience.
These results show that during the interviews as well as the written reflections, the students most frequently discussed the positive value of being able to apply or connect theory, concepts, and content to real-world problems during the laboratory work (APPL CONN). Associated with this was enhanced understanding of bioengineering theory, concepts, or content, including via application. Another experience they consistently discussed frequently was the positive value of teaming as part of laboratory work (TEAM). A theme frequently discussed during the interviews was the value of the computer programming experiences gained while working on the projects (PROG), although this was discussed to a lesser extent in the written reflections. The value of guidance and feedback from the instructor was also discussed most frequently during the written reflections, although this experience was discussed to a lesser extent during the interviews (GUIDE FEED).
Additional experiences frequently mentioned during one of the two prompts were the following: 1) the value of the laboratory projects for other work or job assignments (OTHER WORK JOB); 2) writing and communications skills enhancement (WRITE COMM) via laboratory work; and 3) revisions and modifications to their work (REVISE). The frequencies of occurrence of additional experiences or points of learning or development are also shown in Table  4.

Written Reflections: Breakthrough or Memorable Moments
Seventeen written responses to prompt 2 were received, which was posed on two separate occasions (i.e., after two different projects). Upon content-analyzing (i.e., double-coding) the responses, 13 responses (76%) indicated there were breakthrough or memorable moments in the completion of the particular project, which was a welcome finding. The percentage of responses in which various themes were discussed in relation to the breakthrough or memorable moments were also determined and are shown in Table 5. The same coding scheme as used for prompt 1 about the value of the experiences was used for this prompt. In reviewing the results in Table 5, many of the top (i.e., highest frequency) categories were also most frequently discussed with prompt 1 on the value of their laboratory experiences. These top categories included programming-related work (PROG), application or connection of theory to real-world problems and enhanced understanding of theory via application (APPL CONN), and guidance and feedback from the instructor (GUIDE FEED). In addition, experimentation, methods, data analysis, and/or mathematical modeling (EXPER DATA) was a frequent discussion item with prompt 2. Based on these two prompts, the categories of PROG (programming), APPL CONN (application and connection of content), and GUIDE FEED (instructor guidance and feedback) were the experiences of greatest value to the students as they worked on their open-ended projects with formal scaffolding.

Experiences Reflected upon During Follow-up Course
A total of six four-member groups for a total of 24 students were interviewed in the follow-up course in bioengineering methods using the reflective prompt in Table 3 about prior experiences that were currently helping or supporting them. These interviews were intended to provide evidence of the effectiveness of the scaffolding offered in the biosignals laboratory. The students interviewed were enrolled in the Balance Module in the follow-up course and represented 29% of the class, which was the maximum number that could take the Balance Module given the structure of the follow-up course. Similar to prompt 1 in the biosignals laboratory, the percentage of interviewees who discussed various categories or themes related to the perceived helpfulness or value of the experiences they received previously was determined. The same coding scheme (i.e., Table 6) was used to content-analyze these responses.
The three most frequently-mentioned experiences that were helpful to or applied by students in the follow-up course were the following: programming (PROG, 67%), understanding or appreciation of a defined format for technical paper (FORMAT, 46%), and instructor interaction or provision of guidance, tutoring, and feedback (GUIDE FEED, 42%), as shown in Table 7. Programming activity (PROG) and instructor feedback (GUIDE FEED) were the most-frequently mentioned experiences of value in the prior course. The reported percentages represent the mentions of the experience during the interview as a percentage of the total students interviewed. Other important experiences from the prior course mentioned by students as being currently helpful were the following: 1) value of the experiences for other work or job assignments (OTHER WORK JOB), 2) application or connection of theory, concepts, and content to real-world problems (APPL CONN), and 3) team work (TEAM). These also surfaced frequently during the reflections in the prior course. All other categories were mentioned by fewer than 20% of the interviewees.

Summary
This paper described the use of multi-point reflections for students as a means of scaffolding their problem solving and determining their perceptions of the experiences they valued during open-ended, scaffolded project work in a biosignals laboratory. Students were prompted on two occasions (i.e., once in writing and once via interview) to reflect on the value of the experiences they received during scaffolded problem-solving in the biosignals laboratory.
In addition, the students were prompted on two occasions to reflect on any breakthrough or memorable moments they had in completing their projects successfully. These same students, who continued on to the follow-up course, were then asked to reflect on the experiences from the laboratory that they deemed helpful in the follow-up course or for other courses, endeavors, or areas. The objective of the subsequent analysis was a determination of those experiences students identified as most valuable or impactful later in time.
This work is grounded in both the scaffolding literature and Experiential Learning Theory, which maintains that learning occurs through a combination of doing as well as reflecting on the doing. The scaffolding literature also encourages instructors to prompt students to reflect on their problem-solving as a means of supporting this problem solving. The literature has made recent calls for additional use and scholarly investigation of both scaffolding and reflection within STEM.

Conclusions
High-impact experiences during scaffolded project work, as reflected upon by students, can inform future scaffolding efforts, since similar approaches can be used by other instructors for their problem-solving curriculums. Analyzing the reflections in a structured manner served to demonstrate students' development in complex problem solving and in becoming effective engineers.
A content analysis of the students' reflections demonstrated that they frequently perceived value in being able to apply or connect theory and content from the course to real-world problems encountered during the laboratory (APPL CONN). Other experiences discussed frequently were the value of teaming as part of laboratory work (TEAM), the value of the programming experiences gained while working on laboratory projects (PROG), and the value of the guidance and timely feedback from the instructor (GUIDE FEED). Thus, the reflections demonstrated students' development in problem solving and engineering practices, as desired. With the second reflective prompt about memorable or breakthrough moments, the majority of responses (76%) indicated that such moments did occur during the project work. The same categories were frequently discussed during this second prompt, in particular programming-related work (PROG), application or connection of theory to real-world problems (APPL CONN), and guidance and feedback (GUIDE FEED). In addition, experimentation, methods, data analysis, and/or mathematical modeling (EXPER DATA) was a frequent discussion point with prompt 2 about memorable or breakthrough moments. These categories are all associated with active doing and serve as a motivator and guide to instructors in designing scaffolding experiences.
Promising trends were also seen when the students were interviewed during the follow-up course after they had undergone the scaffolding and initial reflective prompting. In particular, experience with coding and debugging programs (PROG) and an understanding of technical paper formatting/writing (FORMAT) were the top two responses that students reflected upon as being subsequently helpful (i.e., 67% and 46% of interviewees, respectively). These were particular learning objectives with the scaffolded instruction. In addition, receiving feedback from the instructor (GUIDE FEED) and the resulting benefits for other work or job-related activities (OTHER WORK JOB) were also mentioned frequently. Finally, application of theory and content to real-world problems (APPL CONN) and team-work (TEAM) were each noted by approximately 20% of interviewees, indicating additional intended learning outcomes of the scaffolding. By continuing to ask the same group of students to reflect further in the follow-up course on the experiences from the laboratory that were most helpful or applicable at that later point, further development in problem solving was demonstrated via the reflections. Other researchers have asked students to specifically reflect on their STEM problem solving, which has elicited student statements about the importance and value of completing the various steps of the problem solving process as well as using group work during problem solving (Yuriev et al., 2017). In addition, students stated that reflection, and in particular team-based reflection, enhanced their problem-solving skills by driving next-steps planning, suggesting team-role reassignments as needed, and promoting collaboration (Miller & Maellaro, 2016). The authors further stated that team reflective activity can aid problem solving, learning, and future work by prompting re-evaluation of underlying assumptions and inspiring alternative approaches to solving a problem (Miller & Maellaro, 2016).
A lesson learned was that the use of the phrase breakthrough or memorable moments may not have been optimal for the types of projects the students worked on. In the future, this will be re-phrased to moments of conceptual clarity, or the time when the lecture material and project work clicked. The reason for this change is that the biosignals course involves very abstract, difficult concepts, and one of the main goals of the laboratory component is to help clarify these abstract concepts through doing as well as reflecting on the doing.
The experiences uncovered in this research as most valuable, impactful, or helpful by students can be the focus of instructional and scaffolding efforts for STEM students in complex, open-ended problem-solving courses of this nature. Also, a similar reflective and analytical approach can be followed by other instructors and reseachers for their specific problem-solving curriculums to scaffold students in problem solving and identify best practices for supporting and developing students.