{'fraction_non_alphanumeric': 0.025904160453761364, 'fraction_numerical': 0.007104639943864574, 'mean_word_length': 4.824931880108992, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Method to Evaluate Virtual Opponent Based on IA for an Accessible Educational Digital Game: a Case of Study with Dinobase

— The aim of this article is to present a method to evaluate the adequacy of a virtual opponent in a digital game. The motivation for this research arose from the lack of methodologies to evaluate a game in terms of its artificial intelligence (IA) applied to a virtual opponent when confronted with its players. The methodology for this research consisted of the case study applied with 17 children - 8 with intellectual disability and 2 with hearing impairment - in a basic school in play activities with Dinobase digital game. Dinobase is a digital game developed to help children to learn about mathematics, and it was created in a universal design approach, by considering children with disabilities. The virtual opponent must identify the level of ability from children and adapted its behavior. On previous research, the virtual opponent was evaluated with personas, on this research, we use data and analysis triangulation that involved log records, observations, interviews, and video. The results show that the methodology adopted allowed to identify the needs of improvement to the virtual opponent, that were not shown just using personas.


INTRODUCTION
Digital games can help students to best understand some phenomena and get them engaged in classes. According to [1], games improve cognitive development in students because they propose challenges, have rules, and demand higher order thoughts to achieve success. In Special Education, particularly, digital games can provide experiences and allow adaptations for students according to their needs [2]. Activities with digital games can be repeated and present different degrees of difficulty, which favors learning. Students learn in a playful way through digital games since they can play it.
The principle of Universal design, in this context, can be observed in order to provide a basis to the development of accessible games. According to Connell [3], universal design refers to "the design of products and environments to be usable by all people, to the greatest extent possible, without the need for adaptation or specialized design." Apply this principle to educational digital games require to plan the game to be accessible by children with different disabilities, like blindness, intellectual, deafness or mobility. This is not a trivial task.
Artificial intelligence is one of the strategies to development of accessible digital games by adapting the game to the student's cognitive abilities. This was the alternative adopted by Nagasava [4] in the development of a virtual opponent to the Dinobase digital game. This strategy is promising because the game adapts to the gamer, allow him to play in a similar level of capacity against the "computer", reduce frustration and enhance engagement. The virtual opponent must learn player behaviors.
positive results with the virtual opponent, but not uses real people. Strategies to evaluate games, like observations, interviews and logs files are proposed by [5] [6] [7], but they do not analyze artificial intelligence in games. Although some studies evaluate games, we found a lack of methods to conduct the analyses of the behavior of virtual opponents in games. This raised the need for research. This article has the purpose to present the method adopted to analyses the adequacy of a virtual opponent in a game. We adopt a case study of the evaluation of Dinobase game virtual opponent. We present in the next sessions the Dinobase game; the method to analyze its virtual opponent; the results of its analyses and the final considerations of this study.

II. DINOBASE GAME
Dinobase is a digital game based on Base 3 analog game. Base 3 proposes to develop mathematical concepts of exponentiation, specifically the base 3. It consists of a dice and colored cards representing values of this base: 3 0 , 3 1 , 3 2 e 3 3 , respectively represented by red circles, blue squares, green triangles, and white rectangle. On each turn the player throws the dice and acquire cards with values corresponding to the number obtained. It is winner of the match the player who first acquire the white rectangle (3 3 ), however, as the data only has values between 1 and 6, to win the card worth 27 it is necessary exchanges cards while the player progresses in the game. The child needs to understand, for example, that to get a green triangle (3 2 ), he/she needs three blue squares (3 1 ). Thus, it is necessary to understand the relationship among quantity and image, which are not clearly exposed, requiring complex process of abstraction.
In digital game Dinobase the cards were substituted by dinosaurs' graphics elements [8]. In Fig. 1 we see the main interface of Dinobase, which has the inventory of a player, presented in his/her turn in the game. As Base 3, in Dinobase there is no relationship between the figure and the value that it represents, neither relationship between the Base 3 cards with the imageries of the Dinobase. Thus, the required level of abstraction in both games is very high, making it even more complex in the case of digital game because the player does not have at hands tangible objects to manipulate. Adaptation between these games is not something direct because of the graphical difference among them, requiring an understanding of the game to realize that they are equal. Table 1 shows the relationship between values of the items and their representations in both games.

= 27
Source: [8] (adapted by author) Dinobase 2.0 was a new version of Dinobase that allow children play against a virtual opponent. The virtual opponent was developed using the agent intelligent approach by implementing a Finite-state machine. The agent observes the player's actions and tries to play like him/her, and this way children will have an opponent with him/her similar skills. The idea is that the game will not so difficult to cause frustration neither easy, to cause lack of interest [4]. This strategy aims to offer a game that intelligently adapts to the different cognitive abilities of children, thus leading to the concept of universal design. Dinobase 2.0 was available with personas. The technique of persona is a technique used to represent the profile of the group of users that will use the system, making possible to make implications to meet the needs of the target audience [9]. Nagasava [4] defined personas with three levels of abilities: lesser ability, intermediate ability and highly ability. In this way, the objective was to evaluate whether the virtual opponent would converge to a single skill level or would alternate between them, adapting to the skill level of the player. Tests with personas showed weaknesses with the virtual opponent, as in the case of children who played with highly ability and had difficulty to win, which can cause frustration. This motivated the research to evaluate the game with real players with different skill levels, including children with disabilities.

III. METHOD
The method to analyses the virtual opponent of the Dinobase digital game had a qualitative approach. The research was conducted through game play workshops with children, including those with disabilities. We chose to do the tests in a controlled environment, because this approach allows the evaluator to have control over the actions of the users and to verify the environmental and social influences that can affect the performance of the actions.
We adopt different ways of acquiring and analyze data with the goal to verify if the virtual opponent had its behavior adapted to the human gamer. According to Roger, Sharp & Preece [10], to collect data on user performance, a series of methods are used to collect data, as: users' video to capture facial expressions and body language; system logs for capturing data through the system, such as mouse movement and mouse clicks; microphones in the environment for voice recording; satisfaction questionnaires or questions that ask for information about system features; and structured interviews to collect information about what they liked about the product and how it was used. So, we observe children playing the game, collect game logs, made interviews with children, analyzed pre-existing studies with Dinobase (Fig. 2). The use of diverse data in research, which is known as methodological triangulation, allow to reduce biases or deficiencies caused when we use just one method [11]. In the sequence, we explain each approach of collect data.

Log Record
Data logs in software have the aim to register the user's relevant events while he/she operates the software. These data can be used to audit the software, analyze its use or diagnose problems. Kakeshita and Ohta [12], for instance, use data log to analyze students' performance in answering questions, their achievement level, and their learning process. According to the authors, the instructor uses the analysis to improve the educational contents. However, in our research, the purpose with data log register is to analyze if virtual opponent behavior is accordingly with student behavior.
In the case of Dinobase 2.0 there was no data collecting, so we implemented a log register with the aim to store all the user's and virtual opponent's actions. This new version of the game is called Dinobase 2.1. Each log register is composed by the following structure: player name (VIRTUAL if it is the virtual opponent), round, date, action, action value, number of elements eggs, amount of baby elements, amount of young dinosaur elements, amount of adult dinosaur elements, total amount of score. Each log record in the log file represents one action, that can be: 1. S -play the die; 2. C -buy items; 3. T -exchange items; 4. F -end of an innings; 5. V -victory; 6. W -attempt to finish an innings without buy all possible items; 7. X -attempt to acquire items without having points; 8. Y -attempt to make exchanges without selecting elements or not having necessary punctuation.
In our experiment with children, Dinobase 2.1 registered all actions performed by students and virtual opponent, which were analyzed and are presented at section Results.

Observation
Observation is a technic of data collection that allows researchers to see, to hear and to exam facts or phenomena that they want to study [13]. This technique requires direct contact with reality and allow researchers to identify and get proves about the person behaviors while in contact with the object of study. Observation as a scientific technique needs a plan, methodical registration, and verifications and controls for its validation. In our research, observation was conducted through a systematic approach, by observing students in workshops of game play.
Observation was conducted by two researchers, who take notes on a form previously prepared. This team observation, according to Marconi & Lakatos [13] is advisable since they can observe the phenomena through diverse angles. So, each question from the form was discussed by the researchers, that way there were a consensus about children behavior. The questions are: 1. Did the child present any difficulties in understanding the functioning of the game? 2. Is the child unsatisfied because the game is too difficult? 3. Is the child unsatisfied because the game is too easy? 4. Does the child show disinterest in playing the game? 5. Has the child lost focus during game play? 6. Has the child failed to purchase items drawn before the end of the innings? 7. Has the child failed to exchange available inventory items when possible?

Interview
The third approach to data collect was based on interviews with students after play Dinobase game. The interview is a meeting between two people to obtain information about some subject in an oral form [13]. On our research, interviews had the intention to get from the children their feelings, satisfaction, and fun playing Dinobase.
The interviews were standardized and structured [13], so the answers could be compared. For each question, we write down two (2) for the answer "always", one (1) for "sometimes", and zero (0) if the children answered "never". The questions were: 1. Did you like the game? 2. Would you like to play again in another opportunity? 3. Were you able to defeat your opponent? 4. Did you play against another child who was on another computer or played against the computer? This last question, in particular, aims to investigate how children perceive the virtual opponent. A virtual opponent must be as real as possible, that is, it is not enough to fulfill its basic function in the game, which is to interact with other players within the logic proposed for the game, it must also present natural behaviors of a human player, such as know the environment in which it is inserted, display emotions and personality as if it had a life of its own, with the goal of human players questioning themselves against who they are playing [14].

Video recording and analyses
Dinobase and Base 3 were used in former research, as tools to help children to understand digital games engines. On this research, children were filmed and analyzed thought narrative analyses [15]. We use those materials to analyze children behavior in playing the game, with the objective of validating the profiles classified by Nagasava [4] and to prepare to the workshops. The identification of a new profile would be of great impact, having an enormous gain for the improvement of the virtual opponent and to make it more complete.
The videos and narratives were analyzed to observe if during the rounds: (i) the player stopped making purchases even with available points and (ii) the player stopped making exchanges even with elements to do so. These behaviors show the student understanding about the game. The players behavior did not change during the match, due to the matches being short. No player presented significant evolution during one single match, that means, no one initiate a match presenting great difficulty and finished it with great dominion in understanding the thematic of the game. However, it was verified that the players acquire knowledge when playing several matches, that can influence the results of this research.
The analysis of videos and Alves' narrative [15] confirmed the profiles classified by Nagasava [4] that it, three levels of players' ability. This helped for the preparation for the workshops with children since we had data to assist in conducting the evaluations.

Research place and participants
The research was conducted in a public school in the city of Itajaí -SC, Brazil. The school selected was the Basic School José Fernandes Potter, located in the Espinheiros neighborhood. The choice of this educational establishment was due to its characteristics of specialized educational services, and for attending countless children with disabilities in the age group defined for the research, that means, 8 to 10 years old. the research was authorized by the Education Secretary of Itajaí, Santa Catarina, Brazil and all ethical aspects were respected.
The teacher of the SRM (Multifunctional Resource Room) -where the specialized education is conducted in opposite shift from regular classes -helped to define the group of children for the research. Her participation was important because Dinobase was created in the context of special education, notably for children with intellectual disabilities. To conduct the research, we needed children with diverse levels of intellectual ability, so we define to include children with and without disabilities. A total of 17 children were selected, among them 8 with intellectual disability and 2 with hearing impairment. All of them within the age range defined for the research. Parents were informed about the study and agreed to the participation of their children through consent form.
The workshops were held in a single day. Children divided into groups and invited to play Dinobase 2.1, each one on a computer, individually. The children received general instructions about the operation of the game before play. All actions that they performed within the game were saved in log files for further analysis, they were photographed and filmed, and we took notes on the observation form. The individual interview was made immediately after the child play the game. Fig. 3 shows some children playing Dinobase 2.1.

IV. RESULTS
With the objective of evaluate the virtual opponent of Dinobase 2.1, we performed the workshops with children, with and without disabilities. Too many data were collected on this activity, since we decided to construct a data triangulation so that we could conduct the results to have a realistic analysis about the artificial intelligence applied to the game. Thereby, the data analysis was divided into: (i) analysis of the data generated from the observation of children in game play activity and interviews with children after game play activity; (ii) analysis of the videos recording with the children in game play activity; and (iii) analysis of the log registered during children's activities.

Analysis of children in play activity: observations and interviews
Children behavior in game activity were analyzed through the notes made during the observation and the interviews with children after the game activity, with the intent to verify if there was any dissatisfaction generated by the virtual opponent that could make the child lose interest in the game and if they realized who was their opponent.
The analyze shows that five children (29.41%) at some point in the matches were dissatisfied because the virtual opponent made it easier to play; three children (17.65%) demonstrated at some point their dissatisfaction with the game because it was very difficult, among them two are children with more accentuated intellectual impairment; the other children (52,94%) showed satisfied with the game.
Looking to the difficulty to play Dinobase 2.1, five of the 17 children (29.42%) had difficulty in understanding the gameplay, eight (47.06%) presented some difficulty at some point; and four of them (23.52%), did not present any difficulties in any moment.
About the interest in the game, just four children presented at some point disinterest during the matches, and these were the same children who presented more difficulty in understanding the game. Other seven children lost focus on the match in some moments, but the videos of the workshops help to observe that those distraction due because some external factors draw attention, not due to the dissatisfaction for the game.
All children failed to make purchases and exchanges at some time, this shows that no child had full control over the operation of the game.
The interviews with children show that they liked the game and would play it again, this shows that the game was interesting and fun for them. All children answered that they managed to beat the virtual opponent, but observation shows that some children had not been able to beat it. When asked who the opponent was, all children responded that they were playing against the "computer" and not against another child. When asked why, some children responded that they did not know the answer, other, however, responded that the actions the computer took were very fast, so it did not appear to be actions done by a child. more difficult, also, children realized that they are playing against a software, and not another human being. Those results reveal that the virtual opponent behavior is not adequately balanced according to children's abilities. According to Monteiro & Santos [14], games become more interesting when the player believes he is interacting with another human, so his actions are not so predictable, which makes the game more challenging and fun. To Ravyse, Blignaut, Leendertz et al [16] some success factors for a serious game include realism, interaction, and adaptivity. If the game cause frustration, the player may decide not to play it anymore, and in the case of an educational game, the learning intention will not be fulfilled.

Video recording analysis
Children behavior and engagement while playing Dinobase were analyzed through the videos recorded during the workshops. We adapted the Involvement Scale proposed by Cathcart [17], so we analyzed children's level of concentration, energy, complexity, facial expression, persistence, time to play, and their comments. The Involvement scale allow understanding if the child is engaged in the activity, which is a necessary condition for her/his learning. For each child, the Likert scale values from 1 to 5 (where 1 is very low and 5 is very high) were defined for each item. The results are: • Concentration: defines the concentration level of children during the game play. From 17 children, 11 had very high concentration, 6 children had medium and high concentration. Those data demonstrate that children were engaged to play and trying to win the game; • Energy: defines child's willingness to participate in the workshops. Children who were quiet, shy and/or unwilling to attend workshops were considered with low energy. Many children started the workshop very shy because they were dealing with the researcher who is a stranger to them. Slowly, they were more relaxed and ask questions about the game, even so, children's energy level was assessed from low to medium. Of the total children, 4 received a low score, 7 received an average score and 6 children received a high score; • Complexity: child's ease in carrying out activities that require high mental effort. Low scores were given to children who demonstrated greater difficulty in performing tasks that required high attention. Of the total, 9 children received medium, low and very low scores, 3 of them have intellectual disabilities. Eight children received high and very high scores for this characteristic, 5 of them have intellectual disability and 2 of them have hearing impairment; • Facial expression and posture: define whether the child demonstrates satisfaction, doubt or astonishment in carrying out the tasks and whether the child maintains the posture during the difficulties presented in the game. Children received high and very high scores, with only 3 children receiving low scores for this characteristic, coincidentally they were children with intellectual disabilities who at some point did not accept that the virtual opponent was winning the match with some ease. The data show that the children maintained their posture even in times of difficulty and facial expression showed satisfaction in most of the time; • Persistence: defines whether the child maintains persistence in reaching the proposed objectives even in the most difficult moments. All children participating in the workshop received a high and very high score for this characteristic (14 children received a maximum score). This shows that children were very excited to participate in the workshop and overcome the difficulties that the game presented; • Accuracy: defines child's accuracy in the game activities, received a low score if he / she took the games in trial and error. Accuracy was the feature that received the lowest score among all the other features. Only 3 children received a high score, 8 children received a low and very low score; • Reaction time: defines the reaction time of children, if it takes a long time to perform a certain action. Even children with intellectual and hearing disabilities had good scores for this characteristic, in total, 8 children received high or very high scores (6 of them with intellectual or hearing disabilities). Only 4 children received a low or very low score; • Verbal comments: verification of the child's participation and interest in the workshop, whether with comments, interest in questioning colleagues, concern about making good starts. Only 5 children received a high or very high score for this characteristic. This defines that children were very quiet during the workshop, few of them asked questions about the actions or functioning of the game. Many children felt ashamed to ask or answer the questions the researcher asked.

Log analysis
In taxonomy -player modeling proposed by [18], Off-Line Review is the evaluation of a game log after its completion. The Off-Line Review proposes to analyze the different states of the player or agents applied to the game to reach a set of information for the purpose of create strategies or improve the characters involved.
The analysis of the logs allowed verify whether the virtual opponent was balanced in a way to behave similarly to the player, adapting to his skill level. The logs also sought to assess whether the virtual opponent facilitated or hindered the game for the player, an important factor for the child to have the opportunity to learn from the game.
Logs of 3,147 actions were collected over 31 matches. Some children played more than one match because they had more time available for the workshop. All the clicks the player made during the game were considered actions, that is, clicks on buttons, inventory items, purchasing actions, item exchanges and end of the round. Clicks on non-clickable areas of the game were not considered.
Virtual opponent won 9 matches out of the 31 disputed, that is, the percentage of victory was 29.03%. It was expected about 50% of victories if the virtual opponent had similar actions to the player, so this percentage was considered low. In a detailed analysis of the logs of these 9 victories, in 2 of them the player did not win because he/she did not make the correct move, it means, he/she did not realize he/she has the points to win. The other 7 matches the virtual opponent was very advantageous in the raffle of values and it would be very difficult for the player to be the winner. In other words: in 7 of the 31 matches (22.59%) the virtual opponent won easily due to having more luck in the raffle of values. On the other hand, from the 22 matches in which the player was the winner, 4 of them the virtual opponent already had the necessary score to win but it let the player win because he/she had great difficulty on playing. 8 of them the virtual opponent did not even did 21 raffle points, so the player had a lot of advantage in the raffle of values and the virtual opponent had no chance to a competitive game. These results shows that the luck factor of the game does not make it competitive. In addition, the match always starts with the player, causing a disadvantage to virtual opponent.
The exchange of items to progress in the game are fundamental actions in Dinobase because it is through them that the child develops their understanding of mathematical concepts involved and can achieve victory in the game. For the 31 matches, in 20 of them (64.52%) the percentage of rounds that the virtual opponent failed to make exchanges was higher than the percentage of rounds that the player failed to make exchanges; in 8 of them (25.80%) the percentage of rounds that the player failed to make exchanges was higher than the percentage of rounds that the virtual opponent stopped making exchanges and in 3 matches the player and the virtual opponent made the exchanges whenever possible, not accumulating elements in the inventory. The data show that the virtual opponent did not act in a similar way to the player's actions. Often, the virtual opponent "facilitated" the game by not making the possible exchanges, even though the player was playing optimally.
To verify the children's learning, the total of their actions were analyzed. The objective was to check if the players were understanding the game or just playing with trial-and-error actions. For this case, we considered the matches that presented more than 30% of the wrong actions as a learning margin. Only 6 of the 31 games were above the 30% learning margin. In other words, it can be considered that 19.35% of the matches were played through trial-and-error due to the children not understanding the real value of the elements to correctly carry out the purchase and exchange actions.
The analysis of the actions in play of some children, revealed relevant information for the assessment of the virtual opponent. One of them, a hearing-impaired boy, was very interested in the game and played 7 matches of Dinobase. He won 6 matches (85.72%), in which the virtual opponent did not reach the necessary score for the victory, that is, he won easily due to being luckier on raffle. After a few matches, the boy realized that the virtual opponent was performing purchase actions like his. So, he started to buy only elements of lesser value, with the objective of deceiving the virtual opponent, making him only buy the items of lesser value. Through the logs it was possible to notice that the virtual opponent considered the purchase of items more preponderantly, not reproducing the exchange actions in the same way as the player. The child's perception of this behavior allowed him to "cheat" the software.
Another player showed different behaviors. The boy with severe intellectual disability had difficulty in socializing, communicating, and understanding how the game works. From the logs of his matches, it is observed that the boy did not make the necessary exchanges to progress in the game and in most cases he's actions were by trial and error. Checking the videos of the workshop, the player frequently asked his colleagues or the teacher why he was unable to perform a certain action. The boy played 3 matches and won 1 of them. The luck factor was again observed in the raffle of points. The virtual opponent, in this case, behaved appropriately, making few item exchanges, just like the player.

Recommendations towards virtual opponent's Improvement
The triangulation of data made it possible to examine the game Dinobase and its virtual opponent through several dimensions. The analysis of children's behavior and their perceptions increased understanding of their relationship with the game, the fun promoted, and of the learning possibilities. The records of actions (logs) presented quantitative data that revealed weaknesses and potentialities of the virtual opponent and corroborated the actions of the children, observed empirically. This analysis made it possible to develop recommendations for improvements for future implementations of the game under study.
• Slower presentation of the virtual opponent's actions on the interface: all the children realized that they were playing a game against the "computer". According to [14], if the child does not realize that he/she is playing against a virtual opponent, he/she may take the game more seriously, having a greater rivalry for thinking that he/she is playing against another child, then it is recommended to leave the agent's actions slower. With this measure, the player will be able to better visualize opponent's actions, so he/she could reflect on the play and promote learning; • Visual move: the game visually presents just the result of the virtual opponent's move, not allowing the player to view the choices of items made for the exchange, that is, their movements in the inventory. It is suggested that this move is like that of the player, that is, that is, that visually the selected items are "clicked" by the virtual opponent. Learning also takes place with the observation of the opponent's play, so this aspect is fundamental; • Agent perception: Children found irregularities in the agent behavior and so they were able to define strategies to defeat him. The virtual opponent did not realize that it is playing with the same child and resumes his/she is learning with each match, while the child improves his/her skills. Thus, it is recommended that the agent keep the player's information and, when starting again, already have knowledge of his skill; • Start of the match: the first round in the game always starts with the player, giving him an advantage over the virtual opponent because he has more chances to add up the points needed to win. To minimize this problem the game must make a raffle to decide who starts the match; • Virtual opponent actions: data analysis showed that virtual opponent does not exhibit a behavior similar to the player, especially when exchanging items. It is recommended to revise the algorithm to improve the perception of the way the player performs his moves and adapt the virtual opponent's moves to approach the player's skill level.

V. CONCLUSION
The objective of this paper is to investigate methods to analysis of the virtual opponent's behavior in digital games, with the purpose of validating their suitability to the skill levels of players. For this purpose, the authors carried out a case study of the digital game Dinobase, a potentiation learning game proposes to develop mathematical concepts of exponentiation, specifically the base 3.
The analysis involved the triangulation of data that was collected in different forms: users' video in game activities; system logs; voice recording; and structured interviews. These data were analyzed in a qualitative and quantitative approach, and demonstrated several weaknesses of the virtual opponent, for which suggestions for improvements were presented.
The study revealed the need to evaluate the game under different aspects, emphasizing the participation of students with and without disabilities, and their different levels of skills and knowledge. The child's perception of the game is a major factor, however the analysis of the logs made it possible to observe in depth the actions, which in some cases contradicted the students' actions.
The recommendations presented in this article were implemented in a new version of the game Dinobase [19] and can be downloaded at https://univalildi.wixsite.com/univalildi/dinobase. It is suggested as a future work the revaluation of the game through the methodology proposed in this article, with the view to validate both the methodology and the effectiveness of the recommendations implemented in the new version of the game.