{'fraction_non_alphanumeric': 0.029076773566569486, 'fraction_numerical': 0.010962099125364432, 'mean_word_length': 5.147192353643967, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Understanding Children’s Problem- solving Strategies in Solving Game-based Logic Problems

The International Journal of Technology in Education and Science (IJTES) is a peer-reviewed scholarly online journal. This article may be used for research, teaching, and private study purposes. Authors alone are responsible for the contents of their articles. The journal owns the copyright of the articles. The publisher shall not be liable for any loss, actions, claims, proceedings, demand, or costs or damages whatsoever or howsoever caused arising directly or indirectly in connection with or arising out of the use of the research material. All authors are requested to disclose any actual or potential conflict of interest including any financial, personal or other relationships with other people or organizations regarding the submitted work.


Introduction
Problem-solving is the process of overcoming barriers to reach a desired goal state (Mayer & Wittrock, 2006). Problems can vary in terms of their structure: well-defined problems usually have one clearly defined solution (e.g., a simple algebra problem), while ill-defined problems are more open-ended and can have multiple different outcomes (e.g., designing a bridge) (Jonassen, 2000). Problem-solving, therefore, is a diverse task in that approaches to solve a problem might vary depending on the type of the problem and the context in which it is represented.
Learning to solve problems is perhaps the most important skill a child can master, and without it, they cannot properly function in their daily lives or careers (Jonassen, 2004). Given its importance, it is key for educators to understand how children approach and solve problems, to help them become better problem solvers. Our purpose in this study, therefore, was to understand children's approaches to solving game-based problems by examining their performances while solving them.

Background Strategies Used in Solving Problems
Problem-solving involves steps for making sense of the problem (i.e., understanding), planning a solution, implementing the solution, and monitoring and evaluation of the activities for a solution (Polya, 1957;Jonassen, 2000Jonassen, , 2004. Understanding the problem usually involves understanding the patterns and structure that governs the problem. For example, in the context of economics, it could be understanding the supply-demand relationship, or in biology, understanding the relationship between a predator and its prey. To identify the rules or structures governing a problem, one can-especially in interactive environments-explore, observe, and form knowledge of the problem situation (Sonnleitner, Keller, Martin, Latour, & Brunner, 2017). The efficiency of the strategies used to explore the rules can vary in effectiveness, especially in complex problem solving (CPS) contexts where digital assessments are used (e.g., Sonnleitner, 2012). For example, in the case of someone trying to understand how to operate a plane in a flight simulator, pressing all the available buttons at once is not the most effective strategy and certainly not an informative one. Switching one button at a time and understanding what it does is a more informative approach (Kröner, Plass, & Leutner, 2005;Sonnleitner et al., 2017), as manipulating two variables at the same time would confound the results (Croker, 2011). This more effective strategy is akin to the Varying one thing at a time (VOTAT) approach used in hypothesis testing.
VOTAT refers to situations when individuals change one variable at a time to solve a problem. In addition to VOTAT, another often-used strategy is Holding one thing at a time (HOTAT), which refers to changing everything but one variable. Finally, Change All (CA) refers to changing everything and starting from scratch.
In terms of their effectiveness, previous research has identified that VOTAT is the most effective strategy (Croker, 2011;Tschirgi, 1980). However, young children tend to change more than one variable (and sometimes all variables) in problem situations, employing less effective strategies (Croker, 2011;Tschirgi, 1980). It should be noted here that while solving problems and proposing hypotheses are important, figuring out the reasons for the failure of the proposed solution (or troubleshooting) is also crucial in becoming an effective problem solver. A key aspect of troubleshooting is locating the source of the problem or problem finding (Getzels, 1979). Successful problem solving requires effective monitoring and evaluation of the outcomes. If the solution fails, problem solvers who can identify the reasons for the failure and can troubleshoot them have a better chance of solving the problems in later trials than individuals who do not or cannot. Therefore, it is important to effectively execute all necessary program solving strategies to solve and troubleshoot problems.

Game-Based Assessments
In a traditional assessment setting, a student may struggle to stay engaged or might experience testing anxiety, which may impact their performance. Game-based assessments (GBA), or stealth assessments, are those within a game that are virtually undetected to the player (Shute, 2016;Shute & Ke, 2012). This provides researchers and educators with an alternative method of assessing a student's abilities to overcome some of the aforementioned shortcomings of traditional assessments.

International Journal of Technology in Education and Science (IJTES)
247 Assessing students using a GBA allows educators and researchers to analyze their fundamental learning processes (Eseryel, Law, Ifenthaler, Ge, & Miller, 2014). Some additional contributions a GBA offers is that they allow the assessment to occur in a stimulating environment. A stimulating virtual environment is advantageous because a student's metacognitive capabilities increase when tasks are enjoyable (Chatzipanteli, Digelidis, Karatzoglidis, & Dean, 2016). The stimulating gaming environment plays a role in keeping students engaged, which helps to monitor their activity and identify their problem-solving abilities (Liu, Cheng, & Huang, 2011).
GBAs offer simulated embodied experiences, which assist in the development of computational thinking and problem-solving skills (Liu, Cheng, & Huang, 2011). The simulation in GBAs encourages students to think in terms of computation. Students then work to build mental and computational representations to understand how changing one variable may affect another in a problem situation (Eseryel et al., 2014). The use of this process helps familiarize students with the concepts of hypothesis generation, experimentation, and interpretation of their results (Liu, Cheng, & Huang, 2011). Relevant literature supports the notion that GBAs are an effective approach to helping students build on their metacognitive abilities and further develop computational problemsolving skills (Chatzipanteli, Digelidis, Karatzoglidis, & Dean, 20166;Greiff, Wüstenberg, Holt, Goldhammer, & Funke, 2013).

Lightbot: Digital Puzzles to Teach Programming Concepts
Puzzles are types of games, which are rule-governed, played without a competitor, and have one answer (Crawford, 2003;Fullerton, 2008). From a problem-solving standpoint, puzzles are a type of logical problems (Jonassen, 2000): the solver is to discover the method to most efficiently solve the puzzle. Lightbot is a puzzlebased video game where users command a robot to navigate it through maze-like levels. The robot receives commands in the form of directions and action tiles, and users drag these command tiles into an instruction space to execute these commands (see Figure 1). By successfully completing the instruction, users complete the levels of the puzzle and move onto more complex puzzles (Gouws, Bradshaw, & Wentworth, 2013). Lightbot has been previously used as a game-based assessment and learning tool to teach basic computer science or computational concepts (e.g., Gouws et al., 2013). For example, Mathrani et al. (2016) conducted a study to test Lightbot as a platform to teach computing concepts. The researchers found that the game-based learning approach helped with their understanding of basic computer programming and that the cohort had positive feelings toward the use of Lightbot to teach these concepts. It was also found that students in the introductory computer science courses had a more positive experience learning basic programming concepts of abstraction, function, and reuse. The researchers found that the students had a positive experience with Lightbot and performed well on a quiz given to test these basic concepts, scoring higher than the students in the previous semesters (Lopez et al., 2016).

The Current Study
The purpose of the current study was to (a) understand the problem-solving strategies used by students while solving puzzles in the game-based tool, and (b) investigate the problem-finding and troubleshooting strategies the young students used while solving the puzzles. We also aimed to take a critical perspective in our data analysis and provide some insights as to the usefulness of the game-based tool. More specifically, our research questions in this study were: 1. What methods/approaches do the students use to complete each level in the puzzle? a. Do they use one or multiple methods? b. Are there differences in the effectiveness of approaches employed?

Methods
This study is a basic interpretive qualitative study, with observation (i.e., students' screen recordings of their puzzle-solving activity) being the method of data collection. In basic interpretive qualitative studies--or descriptive qualitative (Mayan, 2016)-studies, the researchers look for patterns and themes in the data in order to produce a detailed description of participant activity and the phenomenon observed, framed by the literature that supported the study (Merriam, 2002;Sandelowski, 2000). Basic interpretive and descriptive qualitative studies are popular types of qualitative studies in education (Merriam, 2002).
Observation was chosen as the primary method of data collection in this study, given the constraints of the research setting. Because of the limited time with each participant, interviews were not feasible, so we decided that each participant's screen recordings of their interactions with Lightbot would be the source of data for this study. Observation data overcomes the biases of self-reporting, and it can have higher perceived reliability because of this (Gall, Borg, & Gall, 1996). Observations can move beyond the selective self-perceptions of interviewees and survey-takers (Patton, 2002). Direct observation of subjects describes settings and the activities of the participants in detail, helping the researcher and the reader derive meaning from the participants' actions and behaviors (Patton, 2002).

Participants
The eight (8) participants in this study were all fifth-grade students at a rural, Title 1 Southeastern U.S. elementary school. They were selected from an elective game design class that met once a week for most of a semester. The game design class group was chosen because of their interest in game design and development aligned with the problem-solving activities found in Lightbot, the iPad app and GBA used in this study. Most participants in the game design class were participants in the study, though three students' data were lost due to issues with the screen recording process employed.

Instruments and Procedures
Observations were recorded using iOS screen capture software internally built into an iPad Pro used to play Lightbot. These recordings recorded all on-screen actions of the participants in the game but did not record their faces, movements, or out-of-game behaviors. Using these recorded observations kept the focus on their in-game problem-solving. Participants were given an average of twenty minutes to work with the iPad app Lightbot. The GBA allowed the participants the potential to progress through a total of 11 possible puzzles, each increasing in difficulty and complexity. Participants were allowed to skip puzzles they found them too easy or difficult. Since Lightbot had its own tutorials, researchers did not provide any guidance in the completion of the puzzles. The researcher role was that of an observer as participant: the researchers stayed mostly in their observer role but did have limited interaction with the participants, mostly in an attempt to elicit feedback as to why they made the decisions they did (Glesne, 2016). These attempts to elicit verbal feedback did not result in usable data.

Figure 2. VideoANT Collaborative Video Annotation Tool
The screen recordings of the students were coded using a tool called VideoANT, a free, collaborative video analysis tool created by the University of Minnesota (Figure 2). Using VideoANT, the researchers were able to collaboratively annotate and comment on the screen capture videos of the participants playing Lightbot,

250
identifying important moments and patterns in their in-game play, as well as counting important events (Boudah, 2019) such as the beginning of a puzzle, the strategies used to solve a puzzle, and the troubleshooting strategies employed. The data were coded using a coding scheme, developed based on the aforementioned problem-solving literature and our research questions, wherein the observational data was put into categories based on the frequency of observed activity. To this end, we both flagged students' problem-solving and troubleshooting approaches, and tracked how much time they spent on solving the puzzles. This coding scheme was less reliant on the judgment of the researchers as the criteria was frequently discussed using screen recordings and researchers reached an agreement over the types of problem solving and troubleshooting approaches. Researchers coded the same video using the coding scheme to train on data analysis and to preserve inter-rater reliability.

Measures
We created three measures through our observation analysis: attempts, time, and strategy use. Attempts were recorded each time a student completed their solution and hit the play button to try their solution. Time was recorded by the timestamps attached to each attempt marker, which the researchers identified. As mention above, for each attempt we also noted down the type of problem-solving or troubleshooting strategy the students used.

RQ1: Problem-Solving Strategies used by Children
There was a total of 58 attempts between the 8 students, and in 55 of these attempts the students finished the puzzles. As can be seen in Table 1, there was a variation among the students in how many puzzles they were able to attempt and complete. While trying to solve the puzzles, our analyses showed that students utilized one of three approaches: varying-one-thing-at-a-time (VOTAT), building all at once or change all (CA), and a mixed method.
VOTAT is an exploration strategy in which students alter the potential cause of the problem in the game environment while holding the rest of the variables constant (Greiff, Molnar, Martin, & Zimmermann, 2018).
This approach was detected in the students' attempt to solve the puzzle by forming sequences one step at a time, holding constant the tiles they felt certain were unproblematic, while carefully altering the remaining tiles. Only 17 of the total attempts utilized the VOTAT approach, indicating a low frequency of use for this approach, despite the well-established effectiveness of this approach in problem-solving (e.g., Sonnleitner et al., 2017).
Students used the changing all (CA) variables strategy 33 times out of 58, during which students attempt to solve the puzzle by creating a sequence and altering all variables at once. Some students applied a mixed approach only 6 times, using a combination of both CA and VOTAT. The attempt was labeled as a mixed approach if a student switched from one method to another during the solving of a puzzle. Based on these findings, it can be argued that CA was the most frequent strategy (57%), while VOTAT (30%) and mixed

International Journal of Technology in Education and Science (IJTES)
251 approaches (13%) were used less frequently when students first attempted to solve a puzzle.  An important observation to note is that all students utilized CA as their default initial strategy to solving puzzles. In other words, during the first puzzle, all the students designed the algorithm for the robot at once, and not building it step by step and testing at each step. It could be argued that the simplicity of the initial puzzle was also a contributing factor to this choice. Following the first puzzle, however, strategies began to vary: displaying additional use of VOTAT or mixed approaches for some students. Students electing to use the mixed approach did so as the puzzles became more challenging. The attempts documented as mixed approaches always began with an initial use of CA followed by the use of VOTAT. Students employing the mixed approach made use of this strategy by completely switching over to a new method, as seen with Student 2 (S2).
Some students approached the GBA with a fixed method of only applying one strategy. S4, S5, and S8, for example, only used CA as their strategy of choice, notably, and were slower than the rest of the group to solve the puzzles (Table 2). While the table shows a majority of attempts favoring the use of CA, this method did not assist students in progressing through the levels. Students who applied VOTAT and mixed strategies advanced much further in the GBA. For example, students 1, 3, 6, and 7 used VOTAT or mixed strategies in addition to CA, and they were also the students who solved the puzzles the fastest. This difference in strategy used can be related to students' epistemic beliefs about how to solve problems (Jonassen, 2000). Notes: Data shows individual differences from the group mean. Negative scores (in green) mean faster problem solving compared to the group mean for that specific puzzle.

International Journal of Technology in Education and Science (IJTES)
253

RQ2: Troubleshooting: Start over vs. Problem Identification
Further analyses of students' problem-solving approaches indicated that students at times needed to troubleshoot their solutions and employed different troubleshooting strategies. We observed two different troubleshooting strategies. First, when faced with an issue, some students chose to start over by deleting the initial solution. This method of troubleshooting was more time consuming and took more effort from the students. For example, as in the case of students 2 and 4, the students' main troubleshooting strategy was to start over from scratch, meaning they had the longest time to complete the puzzles compared to their peers. The second approach to troubleshooting was to look at the solution and identify the tile that caused the problem by observing the actions of the robot and matching them with the tiles. Through this approach, students were quicker to troubleshoot and finish their puzzles.

Limitations and Future Study
The observer effect might have had an effect on the results of this study. The presence of an observer can have an effect on the participants being observed, and the viewpoints and goals of the observer can influence how the data is perceived (Fraenkel & Wallen, 2006). Unless an observer is hidden, participants might react to the presence of the observer, especially if the observer is foreign to them. This prevents the observation of participants in a more naturalistic state (Bernard & Bernard, 2013;Fraenkel & Wallen, 2006). Observer-bias might also have had an effect. If the participants were aware of what the observers were looking for (e.g., puzzle-solving in Lightbot), they might have changed the way they played the game a certain way in an attempt to satisfy this perceived desired outcome, possibly causing them to take less thoughtful approaches toward problem-solving. It should be noted that we employed measures to make both the data collection and analysis as objective as possible with this type of research design. Also, it is important to consider how information and communications technology (ICT), or one's ability to operate technology, may affect performance on a GBA (Greiff, Kretzschmar, Müller, Spinath, & Martin, 2014).

Comments on Lightbot as a GBA
The design of the interactive features of Lightbot were an observed obstacle for some of the participants.
Students were presented with brief tutorials addressing the components of the GBA with quick explanations and demonstration of the features. However, despite the tutorials, students encountered problems due to unintuitive design. GBAs require students to interact with the device in order to explore the game environment to build upon their rule knowledge and brainstorm possible problem-solving strategies (Fischer, Greiff, and Funke, 2012), and the design issues identified here might point to a potential confounding effect of the software on students' performance.
In Lightbot, the turning commands were illustrated as arrows in a turning motion rather than the more commonly known straight arrows pointing in one direction (See Figure 1). The forward command caused some additional trouble for participants, often confusing it for the jumping command. The puzzle was also presented Akcaoglu, Jensen,& Gonzalez 254 to participants from different viewpoints with the robot's starting point varying. The resulting confusion of the commands and varying start points presented a challenge for students who relied on embodying the robot to complete the puzzle. Successful completion of a puzzle becomes increasingly difficult if the student has trouble understanding the basic commands of the game. The complexity of these games has led researchers to postulate that ICT literacy may strongly influence performance on a GBA (Greiff et al., 2014).
Another limitation of the software, which can be closely linked to troubleshooting was related to puzzle solution and replay. To test their potential solutions to the puzzles in Lightbot, participants pressed the play button, which initiated a cycle through the sequence of moves they have selected, demonstrating in the end whether the solution was the correct one or not. As the solution played out, each move was highlighted in time with the Lightbot's movement on the playing field. The playback function always cycles through the entire sequence, and there was no way to slow down or pause this playback to see which part of the solution was highlighted.
The speed of this playback cycle was too fast for many participants, combined with the inability to pause or slow down the playback, made it difficult for participants to discern where they had made incorrect moves in the sequence.

Conclusions and Implications
Problem-solving is an important skill that a student develops and will benefit from greatly in their daily lives.
The main goal of this study was to observe the problem-solving strategies employed by students during their performance on the GBA: Lightbot. Observation of participants' engagement with Lightbot offered insight into the different problem-solving approaches employed by students and the effectiveness of each. Results demonstrated that while VOTAT was the more effective problem-solving method (Croker, 2011;Tschirgi, 1980), students made use of CA most often, with only a few students occasionally opting for a mixed approach as the puzzles became more complex. Regarding troubleshooting methods, students implemented the use of starting over or identifying the faulty tile. Although searching for the faulty tile would be more effective, starting over was the more commonly observed troubleshooting method while also being the more timeconsuming approach.
Prevalent use of the CA approach and starting over troubleshooting method was a finding consistent with that of Tschirgi (1980), in its support of the notion that younger children have a tendency to alter multiple variables at a time as a problem-solving approach. Student performance in the GBA varied. GBAs have great potential and allow for assessment and observation of student's abilities. It is particularly helpful with younger populations because the stimulating environment results in a more enjoyable experience for the student, increasing their metacognitive abilities. However, if the GBA appears to be too difficult for the student, it may have an adverse effect. This may result in a possible lack of motivation which may impact their performance negatively. In the case of Lightbot, students experienced some difficulty with the basic commands of the game which may have resulted in hasty problem-solving behavior in attempting to solve the puzzles.
Given the outcome of the students favoring of the CA approach and the apparent resulting fast-paced problem-solving behavior, it seems that the students may have not yet mastered effective problem-solving strategies. This could be the result of differences in cognitive abilities or lack of interest in computer programming. It may also be the case that because of their young age, they may require additional experience to recognize the necessity of the VOTAT problem-solving approach (Tschigi, 1980). However, given the students demonstrated the use of VOTAT in this observation, it evident that even young students can be taught problem-solving methods such as VOTAT, given their displayed ability to properly manipulate variables when needed (Tschirgi, 1980). Therefore, educators can design instruction to teach students these essential problem-solving strategies in different domains.