{'fraction_non_alphanumeric': 0.024972788057844815, 'fraction_numerical': 0.008459026589954906, 'mean_word_length': 5.310047095761382, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Analysis of unplugged tools used for the development of computational thinking in the teaching of algorithms: a review of the literature

. The development of computational thinking has been frequently pointed out as an efficient way to stimulate learning using the fundamentals of computing. Most of the publications about this approach report successful experiences using the most diverse tools, materials, and techniques. This work highlights the unplugged techniques, which are characterized by using alternative materials to computers. This article presents a systematic literature review that aims to analyze unplugged tools in the educational context, focusing on the development of competencies related to computational thinking. In the analysis, a diversity of tools, systems, and approaches of unplugged computational thinking used to conduct the activities was found, and, despite the heterogeneity of the studies, it was possible to perceive some patterns.


Introduction
The term "Computational Thinking" (CP) was first registered in 1980 by Seymour Papert in his work "Mindstorms: Children, computers, and powerful ideas" (Brackmann, 2017). Considered the father of Constructionism 1 , the author is also considered one of the pioneers in applying this form of reasoning in basic education without the use of computers (Ribeiro et al., 2017), recognizing that although technology plays a key role in the future of education, the main focus is not on the machine, but on the mind and, particularly, on how intellectual and cultural movements develop (Papert, 1985). This attitude seeks ways to produce maximum learning with minimum teaching, valuing the mental construction of the subject, supported by their own "constructions". In this way, the computational resource is often used as a playful tool to simulate situations in which students are confronted with situations that stimulate their autonomy in problemsolving. For Brackmann (2017), Computational Thinking can be defined as "a distinct creative, critical and strategic human capacity to know how to use the fundamentals of computing in the most diverse areas of knowledge, to identify and solve problems" (Brackmann, 2017, p. 29). It is clear that "fundamentals of computing" does not presuppose the need for the existence of a physical machine, but, to Furber (2012), recognize aspects of computing in a world that surrounds us, or even think about the problems so that a computer can solve them (Liukas, 2015). The same author further states that Computational Thinking is performed by people and not by computers. It includes logical thinking, the ability to recognize patterns, reason through algorithms, and decompose and abstract a problem. In the search for correlated works, 18 mappings and systematic reviews already published were identified, addressing an association between Computational Thinking and Unplugged Computing. Of this total, only two  and (Nunes et al. 2019) establish a direct relationship between these concepts in their objective. However, in both cases, the audience was restricted to basic education students and did not focus on detailed descriptions of de-plugged tools. While the former focused on evaluating the contribution of Unplugged Computing in the development of Computational Thinking using a manual search only in national databases, the latter performs a mapping establishing the state of the art with general information about the application of Unplugged Computational Thinking in basic education. The other papers, focused on only one of the areas, 3 predominantly on Unplugged Computing and 13 on Computational Thinking, with the evaluation of the development of Computational Thinking being the most covered subject, which shows the relevance of the subject nowadays and the great diversity of techniques and methodologies for application and evaluation, highlighting Unplugged Computing as an element of motivation and accessibility. In this context, this article aims to analyze scientific publications by conducting a Systematic Literature Review (SLR) in order to identify disconnected tools aimed at the introductory teaching of algorithms in the academic context, focusing on the development of skills related to Computational Thinking and thus answer the following research questions: 1) Which disconnected tools are used in the introductory teaching of algorithms? 2) What is the systematics used to conduct unplugged activities? 3) How have unplugged Computational Thinking approaches been evaluated? The answers to these questions will map the most popular physical resources used in the application of Unplugged Computing techniques and verify how the participants' productions are assessed, as well as what underpins the assessment of the development or not of Computational Thinking among the participants. The text is organized as follows: Section 2 describes the protocol used in the systematic review; Section 3 presents the extracted data and its synthesis in the formulation of answers to the research questions; Section 4 presents the final considerations and possibilities of future work based on the results found.

Protocol of the Systematic Review
V. 20 Nº 1, Agosto, 2022_RENOTE DOI: To perform the data survey necessary for the proposal of this work, a systematic literature review was conducted following the script proposed by Kitchenham (2004), which defines this type of study as a means to identify, evaluate and interpret all available research relevant to a specific research question, area topic or phenomenon of interest. To this end, the author establishes a script organized into three fundamental steps, which are planning, conducting and reporting. In the stage called "planning", the objective is to verify the real need to conduct the review and create the protocol that will be followed during the conduct process. In the "conducting" stage, the selection of primary studies in repositories is performed, following the inclusion and quality criteria also predefined, to perform the extraction of data from those selected. After synthesized, the data are analyzed qualitatively and quantitatively to answer the research questions and provide information considered relevant for future works aiming to compose the "report", the last stage of the review (KITCHENHAM, 2004). These processes are detailed below. The initial phase of the planning stage consists of verifying the relevance of developing the literature review, checking whether this research points to information that has not yet been mapped or systematized. Unlike similar works presented in the previous section, it was thought to identify specific elements related to the introductory teaching of algorithms, without considering a restricted audience. In this way, it would be possible to identify details in the unplugged artifacts and in the systems used to conduct the activities, as well as the predominant methodological characteristics in the evaluation of Computational Thinking more broadly. Thus, this SLR differs from the other studies found because it establishes a current panorama that includes articles from outside the national repositories, considering a diverse audience and listing detailed details related to the unplugged practice and the method of evaluation of the intervention, enabling the mapping of recurring characteristics while establishing relationships between them. Next, the protocol used to perform the systematic review is detailed. The definition of this script is necessary to reduce the possibility of manual interference of the researcher because it makes the process mechanical and impartial (KITCHENHAM 2004). Considering that a large number of keywords and the definition of very complex and rigid rules among these words excessively restrict the search returns, it was decided to use compound words and exclude those that could be searched in the conduction stage, applying the inclusion (IC) and exclusion (EC) criteria. It is worth mentioning that the string was adapted to the format required by each repository so that the returns would satisfy the initial idea of the search. For inclusion, the article or book chapter (CI 4) must have been published within the last five years (CI 1), be available in English or Portuguese (CI 2), full-text access availability (CI 3), and deal specifically with unplugged computing education (CI 5). Consequently, other types of publications (books, abstracts, technical reports, etc.) (EC 4) published more than five years ago (EC 1), in a language other than the ones specified (EC 2), with full text unavailable in the database (EC 3), repeated articles (EC 6) and that do not address the topic or part of it (EC 5), which do not specifically address the introductory teaching of algorithms (EC 5.1), do not address or detail the development of Computational Thinking (EC 5.2), do not detail the development of disconnected activities (EC 5.3) or do not have a practical approach (EC 5.4). The English language was used because it is the most commonly used language in articles that address technology and the Portuguese language aims to prioritize national publications.
V. 20 Nº 1, Agosto, 2022_RENOTE DOI: As a search source, the databases Scopus, Google Scholar IEEE Xplore and ACM Digital Library were used, which encompass the main scientific publications within the scope of the research. The conducting stage consists of applying the protocol in an unbiased manner in order to obtain objective and reliable feedback. Thus, it begins with the search in pre-defined sources. After applying the string to the automated search engines, a total of 365 texts were obtained. With a careful reading of the initial information (title, abstract, and keywords), and identification of inconsistencies with the research objective, 270 papers were eliminated, leaving 95 primary studies after this first filter. Then, with a, it was possible to identify another 36 inconsistencies, most of which were related to exclusion criterion number 5, since although in the abstract some studies address all the inclusion criteria, in the text they are addressed in a superficial way and without sufficient detail. The most recurrent exclusions, which demanded a greater criterion of analysis, were related to the deviation from the theme. Due to the importance of this criterion in the selection of works capable of answering the research questions, it was divided into specific sub-criteria, which enabled the clear identification of the justification for exclusion. There was once again a predominance of research related to the development of Computational Thinking, with few exclusions occurring due to subcriteria 5.1 and 5.2. However, the same cannot be said of the main reason for exclusion, the Unplugged Computing approach (subcriterion 5.3), which is usually cited as a possibility (which justifies its pre-selection in the automatic search) but is not used in many of the experiments, or is not sufficiently detailed to answer the specific research questions (QP2). With regard to the practical approach (subcriterion 5.4), the excluded studies presented preliminary results that pointed to a future practical intervention, which, although promising, still did not meet the expected criteria for the objective of this review. It is noticed that even after the first stage of selection, there were studies that did not meet the fifth criterion, with the exclusion of another 32 studies. This is due to the fact that only the full reading made it possible to confirm the level of detailing promised in the initial information (title, abstract and keywords). Also in the full reading, information related to the detailing of the evaluation methodologies (QP3) was sought, which was not included in the search string so as not to restrict the search results too much.
To answer each research question with the level of rigor and specificity established by them, specific data were established for standardized extraction, to systematize and classify the works to the review criteria. The extracted data were synthesized and served as a basis for analysis and discussion. The data selected to answer the first research question aimed to detail the artifact used in experimenting, extracting details such as the pedagogical objective, in order to identify what is taught, and the physical material used, with emphasis on the fact of having been acquired ready-made or made by researchers. In the case of handmade materials, the theoretical or physical model that inspired the making and its license for use is presented. In addition, map when there is an association between unplugged and plugged activities in the same experiment. To answer the second research question, the data collected refers to the detailing of the experiment. As already explained, there was no restriction of public, so, in this step, it is mapped about the age group. Regarding the specific characteristics of the study conducted, in addition to the type of study and its pedagogical rationale, the data collected show the number of participants, the type of activity used in the proof of concept, and the duration of the intervention. In the third research question, the focus is on the evaluation of the activity and the development of Computational Thinking, specifying if there is a predominance of the focus of the study in a specific area, how the evaluators corrected the practical activities and collected/analyzed the data that supported the results of the study. The main considerations about the collected data and the answers to the questions are described in the next section.

Summary and discussion of results
The first research question (QP1) aims to detail the tools used, from its intentionality to its execution, going through, mainly, the technical details of the product used in the experiment, which, being unplugged, basically consists of industrialized or even handmade objects. It was of interest for this research, to identify these objects in order to find similarities between the proposals and recurrence of techniques. The vast majority of artifacts used in conducting unplugged activities were made by the researchers themselves (51 -86.4%) from various models, the most recurrent being extracted from the Computer Science Unplugged book (17 -28.8%), some designed and built by the authors themselves (7 -11.9%). However, many other models were used and, although there was no concern in describing details regarding the license, it is known that repositories such as Computacional.com.br, csunplugged.org, Livro Computer Science Unplugged (BELL et al., 2007), Code 2 , etc., make models available to use and customize activities for educational purposes. The themes approached in a playful way always involved the teaching of computational concepts, which were categorized here as shown in Frame 1. Source: Authors themselves (2021) In addition to making objects using various materials and printing text and images, some works (13 -22%) also used body movement and sound as a way of reproducing the algorithm. These practices, usually simsimulatee robotic movements through voice commands. It is also worth mentioning that, in addition to the unplugged activity, some texts also used plugged activities (28 -47.5%) in the interventions. These activities were offered mainly to replicate the experiment developed or to increase the level of difficulty, sometimes comparing the development of Computational Thinking with and without the use of computers.
The study of Olmo-Muñoz et al. (2020) sought to demonstrate the possible benefit of a mixed approach that combines disconnected and connected activities. The work demonstrated the benefit of this methodology in the acquisition of skills and in the motivation of students, generating, to the authors, an "obvious double advantage". Interestingly, the experiment conducted by Martinelli et al. (2018), there is a predilection of teachers to develop activities of unplugged character. The researchers concluded that "the choice of this format occurred because they felt more confident about the organization and mediation", which may represent an entryway to the teaching of computing even by teachers still resistant to its inclusion in digital form for lack of technical domain. The significant use of plugged-in activities in association with the unplugged experiment infers an attempt to validate the latter, comparing their results, to do a way to prove the initial hypothesis. It is evident when comparing these approaches, the counterpoint between the levels of work involved in the preparation, conduct, and evaluation of the study is much higher in the unplugged practice of being developed completely manually. Nevertheless, the advantages inherent in the experiment without the use of computers have their main advantage in the definition itself, democratizing the experience so that a wide variety of materials can be used to simulate the behavior of the software. At this point, an important issue for analysis would be the possibility of creating an intermediate approach that, although unplugged, would use a restricted computer system to optimize one of the stages of the practice, attributing to it gains in time and reliability to make possible a more detailed analysis of the individual performance of the participants.

CINTED-UFRGS
For plugged practices, we highlight the use of software such as Bee-Bot, Scratch, Lightbot, Code.org, Pygame.org, LOGO, Visualg, Open Roberta, robots and applications developed by the researchers. The second research question (QP2) sought to understand how the activities were conducted, identifying the target audience and details regarding the experiment itself, such as its rationale and the way used to divide and evaluate the sample. With this information, it was possible to understand the main ways of working on Computational Thinking using unplugged tools, as well as the audience of interest in this technique, focused mainly on primary education. However, it is also possible to notice in the remaining studies (22 -37.3%) very significant results regarding the development of Computational Thinking, which points to the non-existence of age restrictions for application and it is assumed that the choice of students enrolled in formal education is due to the ease of forming groups. Regarding experimentation, most of them (33 -55.9%) were carried out in groups of up to 40 people in a quasi-experimental manner, (44 -74.6%). To Campbell et. al. (1963), these practices "constitute a class of studies of an empirical nature that lacks two of the usual characteristics of experimentation: complete control and randomness in the selection of groups". Because it is predominantly pre-selected groups and restricted to individuals with many characteristics in common, the results are also restricted and the conclusions are not generally applicable. The predominant proof of concept was in the form of a workshop, but several methodologies were used. A workshop is here understood as activities developed outside class hours, but with previously constituted classes, as is also the case of gymkhanas. Workshops and courses, on the other hand, count on groups assembled especially for the realization of the experiment.
A direct relationship is established with the target audience, which are predominantly students in basic education. Thus, it is natural that classroom interventions are more frequent when it is possible to associate the theme of the experiment with the content covered in the curriculum and the workshop format is the opposite way, when one does not want to contextualize directly the content with the experiment. It is also noticed that there is no direct relationship between the level of difficulty categorized in Table 1 with the audiences described in Graph 1, since each experiment, linked to its specific objective does not take into account the previous knowledge of the participant in computing concepts, being the level of difficulty increased gradually as the participant's response. As an example of this independence between computational concept addressed and target audience, Santos et al. Regarding the duration of the intervention, many did not detail the time (20 -33.9%), however, most of those who documented this information (39 -66.1%), lasted up to 2 hours (24 -61.53%). This restricted intervention time was sufficient for the presentation and application of the activity, because the assessment occurred later through the analysis and documentation of the observations, the results of standardized tests and open-ended questionnaires that were able to quantify the difference in learning and, also, relevant qualitative criteria such as interest, motivation and self-evaluation. It is clear that, in most cases, the time available to the researchers was restricted to a predefined number of meetings within the calendar of the grantor institution and this required precise planning in order to occupy this time efficiently. Considering that, as already mentioned, the unplugged activity develops and evaluates more slowly, this required a simplification of the experiments, often evaluated collectively, without considering the individual potential of students in solving the proposed problem, but only in standardized questionnaires at the end. This form of data treatment is justified by crossing the available time with the practice performed and points to a need to develop optimization techniques that enable more detailed data collection in the limited time of the experiments. The third and final research question (QP3) aimed to catalogue the ways in which the development of Computational Thinking could be assessed using disconnected techniques, detailing the instruments used in data collection and analysis. In addition, QP3 sought to understand how researchers correct algorithms in order to link the material with ways of possibly automating this process.
To answer this question, we first tried to identify the focus of the study, as although all the selected studies necessarily addressed unplugged techniques for the development of Computational Thinking, it is noticeable that each study is inclined towards one area or another. It was noticed, then, that the great majority of the selected studies (40 -67.8%) kept their focus on the development of Computational Thinking. In these cases, Unplugged Computing was used as a way to make the experiment accessible without the need to use computers or as a motivational factor. In the cases where the focus was on unplugged practice (11 -18.6%), the experiment aimed to evaluate the unplugged product in PC development directly or indirectly. In the other cases (8 -13.6%) there was a solid connection between the objective and the tool, showing intentionality in specifically evaluating how Computational Thinking develops with unplugged tools. Few of the analyzed works (6 -10.2%) present details about the form of correction, and in all cases, it was done manually. Although some of them mention forms of observation and collective oral correction, this happened as an alternative to find a possible solution, without allowing an individualized evaluation of the solutions proposed by the students. In all the works where the form of correction was not specified, it is assumed that it was manual, however the highlight serves to show how much this part of the experiment is underestimated by the researcher, since the results will be summarily based on the data extracted in the observations, tests and standardized questionnaires, sometimes only in the opinion of the participants. In this way, the individual production performed through unplugged resources taken as a form of stimulus, not being in the scope of evaluation most of the time. An important opportunity is lost here, available only in plugged-in tools, to allow for new instantaneous decision-making based on the hit/error response, which does not occur dynamically since the individualized correction, not being an automated process, is not compatible with the time available for the experiment, generally less than 2 hours. This weakness significantly reduces the effectiveness of the unplugged tool in relation to the plugged one, in the evaluation aspect, where it is possible to have a detailed assessment of the individual progress of the participants. In a simplified way, one can verify a balance between situations in which the process was evaluated in a qualitative way through records of the practice (28 -47.5%) or situations in which the results of tests performed during or at the end of the process were quantified (31 -52.5%). This becomes clear when we cross-reference this information with the analysis methodologies listed by Graph 3. Many studies (25 -42.4%) applied questionnaires, generally treated qualitatively, in order to identify the development of the pillars of Computational Thinking. Some of these even included the participant's self-assessment in the analysis. Others (16 -27.1%) chose to perform two stages of tests (pre and post) in order to measure the development after the unplugged experience compared to a value obtained previously. The remaining (18 -30.5%) base their conclusion only on the observation tools already mentioned.
To record the practice and as a way to collect a large amount of information in the restricted time of the experiment, the researchers used a number of resources, such as written reports of observations, video, photo, audio records, in short, once again it is noted the attempt to optimize the time focused on the application, leaving the evaluation to be performed later. It is noticed that there is a tendency to perform qualitative analysis due to factors such as sample, intervention time, and characteristics of the data collected. However, I once again it is noticed the loss of opportunity to collect more specific data of the individual performance possible through a dynamic methodology of correction of the students' productions.
For the analysis of the quantitative data collected, in most cases, it was performed arithmetic mean and simple comparison between the pre and posttests. However, some studies used specific techniques, such as Analysis of variance (ANOVA) (Miller, 1997) to find out if the results of the experiment are significant, its variant ANCOVA (Analysis of Covariance) (Tabachnick et al. 2013) and Pearson's Correlation Coefficient (Mukaka, 2012) using regression and independent variables and the T-test (Myers, 2009). The application of these statistical tools aimed to statistically validate the students' performance on the quantitative tests. To estimate the reliability of the applied questionnaire, one of the studies used Krombach's Alpha (Hora, 2010), which qualifies the result as it establishes an average correlation between the questions of the same. Some studies used a tool to measure the level of agreement of the student with a given statement, in addition to the response to the questionnaire. The Likert Scale (Likert, 1932) allows for the elaboration of a simplified questionnaire with predefined standardized answers, in which the participant can evaluate both his/her understanding (selfassessment) and the experiment itself with a predominance of a qualitative evaluation with little reference, often centered on the student's self-perception, without establishing a direct relationship between the experiment and the gain provided. Finally, the lack of automation tools incorporated into the unplugged activity, which could represent an optimization of time and, consequently, the possibility of collecting more data on the learning process, is highlighted.

Final Considerations
This review aimed to analyze scientific publications from the last five years in order to identify details related to disconnected tools aimed at the introductory teaching of algorithms, which aim to develop skills related to Computational Thinking. The review protocol used aimed to select primary studies and, through the collection of specific data, to answer three specific research questions, which detailed the tool used, the systematic application of the study and the evaluation. As a result, we identified a diversity of tools, systems and approaches of unplugged Computational Thinking used in the conduction of activities, and, despite the great heterogeneity of the studies, we noticed interesting patterns that have been recurrent in interventions of this type.
Regarding the specification of technical details of the artefacts used, it was clear that the unplugged approach is a low-cost alternative to perform exercises with the aim of developing Computational Thinking, and this feature is enhanced with the handmade production of physical learning objects. Such objects enable playful analogies with the functionality of real computational tools; simulating hardware and software so that the student has an experience similar to the one, they would have using computers. In addition to the tendency to reduce costs, another recurring fact is the use of free models available on digital platforms, which in addition to facilitating planning already bring with them a baggage of accreditation since they allow the comparison of results with other similar experiments.
With regard to the activities conducted, there was a predominance of interventions in regular basic education classes during class time or during after-school hours, which is justified by the ease of forming homogeneous groups of similar size. The intervention time was restricted within pre-established school breaks, which was determinant for the interventions to be carefully planned to optimize the use of time while collecting significant data during the process for later analysis. Quasi-experiments were recurrent with data collected mainly through questionnaires for quantitative analysis (performance in pre and posttests statistically treated and validated) and qualitative analysis (observation also recorded in text form, videos, photos, audio recordings, and researchers' report) where, in addition to performance evaluation, participants were given the opportunity to self-assess. The evaluation of individual productions usually occurred after the development of the practice, to prove or disprove hypotheses regarding the development of Computational Thinking. Because of that, some studies proposed to perform collective corrections with standardized final answers between the initial and final tests. It is noticed that this approach facilitates the process considering that in the restricted intervention time it is unfeasible to perform a simultaneous individualized follow-up, however, regarding the effective learning, it is understood that the participant in most studies did not have an individualized analysis of their production. Considering only the performance in standardized tests limits the interest of the student, since algorithms by definition do not have unique solutions and individual creativity is not usually measured. Although the studies represent important directions and contributions to research on the development of Computational Thinking through Unplugged Computing, there are some recurring weaknesses, such as the divergence between the heterogeneity of the studies and the similarity of the methods, highlighting the lack of criteria, in some studies, when planning the experiment aligning the activity, the audience and the pillars of Computational Thinking. Even so, the benefits demonstrated for learning are undeniable, even in cases with time constraints and refined data processing techniques. Research should be intensified into tools and methodologies that can optimize time and include practices of this kind in the curriculum so that Unplugged Computing and Computational Thinking can no longer be an occasional intervention, but become part of the educational trajectory of mainstream education.