{'fraction_non_alphanumeric': 0.03535679690022603, 'fraction_numerical': 0.01554499746298261, 'mean_word_length': 4.6634012539184955, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 6, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
New Metrics for Learning Evaluation in Digital Education Platforms

Technology applied in education can provide great benefits and overcome challenges by facilitating access to learning objects anywhere and anytime. However, technology alone is not enough, since it requires suitable planning and learning methodologies. Using technology can be problematic, especially in determining whether learning has occurred or not. Futhermore, if learning has not occured, technology can make it difficult to determine how to mitigate this lack of learning. This paper presents a set of new metrics for measuring student's acquired understanding of a content in technology-based education platforms. Some metrics were taken from the literature"as is", some were modified slighty, while others were added. The hypothesis is that we should not only focus on traditional scoring, because it only counts the number of hits/errors and does not consider any other aspect of learning. We applied all metrics to an assessment conducted in a high school class in which we show specific cases, along with metrics, where very useful information can be obtained from by combining several metrics. We conclude that the proposed metrics are promising for measuring student's acquired understanding of a content, as well as for teachers to measure student's weaknesses.


Introduction
The school is one of the most important social and cultural institutions, as it provides knowledge integration and socialization. Therefore, schools need not only to insert technological resources into the teaching-learning processes, but also propose new methodologies and pedagogical practices that, integrated with new technologies, help students develop the skills and abilities needed nowadays. In this case, such challenge is much more complex than simply inserting new technologies. Using computational tools in the classroom motivates a real review of teaching methods. Such education redesign makes the learning process more attractive, especially by providing interaction with educational content through games, physical and/or virtual labs or challenges that stimulate problem solving.
Generally speaking, a virtual learning environment (VLE) is any online environment that aims at teaching and learning (Román, Torres, & Hernández, 2019). VLE is widely used in the context of distance education, because it allows the complete management of a classroom, as well as the use of various media, languages and resources that present information necessary for student learning. In addition, it has tools that allow for interactions between students and teachers, information exchange, submission of assignments and availability of study content. Through the VLE, it is possible to track the learning process of students and generate reports on their performance and progress.
The adoption of technology in education gave rise to the term technology-enhanced learning (TEL), which describes the application of technology to teaching and learning (Dunn & Kennedy, 2019). TEL can engage students with learning and improve knowledge acquisition due to the flexibility of learning it provides. In other words, TEL is any technology that enhances the learning experience.
Technologies make it possible to gather student interaction data throughout the learning process, thereby generating graphs and analysis that support the teacher in both assessment and decision making that involve, for example, the addition of pedagogical activities to reinforce the learning of a subject. Thus, data obtained from classes and assessments could better infer the learning profile and difficulties of a particular class or specific student. This way, the system can recommend several activities and strategies that are most appropriate and tailored to the real needs of the class or student.
The context of this work is virtual learning environments that use technology-enhanced learning. We propose that teachers adopt a learning environment to prepare their lessons using digital learning objects such as slides, videos, photos, texts, games, etc., and make assessments within this technology-based educational environment. Specifically regarding assessment, the platform should include a way to collect additional data at run time, such as the time each student spent on each question and how many times each student changed an answer to the same question.
To generate metrics, the educational platform should also allow the teacher to include other information for each question, such as the topic/subject, the expected time students need to answer the question, the level of difficulty, and the weight of each answer option. This paper only focuses on assessments based on multiple choice questionnaires using closed-ended questions. Students used devices such as mobile phones, tablets or laptops to access the classes and assessments contained in the virtual learning environment.
Assessing a student is not an easy task (James & Fleming, 2004). There are several factors that can influence a student's grade, which often go beyond the degree of knowledge about a topic. Sometimes other factors such as nervousness, somnolence, and even lack of basic knowledge about that topic/subject influence student performance. In addition, it is important to act in a timely manner in case of poor student performance.
Despite the several assessment types, we consider formative and summative assessment (Dixson & Worrell, 2016), where the main aim is to educate and improve student performance, not merely to audit it (Wiggins, 1998). Formative assessment is defined as an activity undertaken by teachers that provides information to be used as feedback to modify teaching and learning activities (Black & Wiliam, 2010). Summative assessment, on the other hand, is aimed at assessing the extent to which the most important outcomes have been reached at the end of assessment. Unlike formative assessments, which are generally used for providing feedback to students and teachers, summative assessments are generally high-stakes assessments and used to get a final assessment of how much learning has taken place, that is, how much a student knows (Dixson & Worrell, 2016). This kind of assessment is typically less frequent, and occurs at the end of instructional segments. Although teachers can adopt several formative and summative assessment methodologies, all of them are hard to implement, not automatic, and prone to errors. In this paper, we propose a novel method for teachers to measure students performance in such a way that, using the proposed metrics, teachers can adjust instruction to maximize student learning and develop interventions to improve student learning (Shepard, 2006). Therefore, the following problems were addressed by this paper: Is it possible to measure student's acquired understanding of a content satisfactorily and fairly? Using such metrics, is it possible to take action to mitigate student's poor understanding of a content?
Current assessments, which we call "traditional assessment," are binary in the sense that they only assess whether the question is right or wrong. In open-ended questions, besides right/wrong, "half-right" is also allowed. However, it seems that this is not the most effective and fair way to evaluate a student, simply because this type of evaluation generally neglects several factors, such as: if the student marked the wrong answer but is close to the correct one; the student's degree of doubt; the time taken to solve the question; the student's level of confidence in answering a question; and the estimated level of understanding of a topic.
In this case, when such items are neglected, a good opportunity is missed to alert (or recommend) students about several topics/subjects in which they could improve their performance. Some of these aspects are very easy to implement. For example, simply indicating which topics a student should focus could greatly help them. This paper presents the following contributions: • a review of the main learning metrics available in the literature; • new learning metrics and implementation of some literature metrics; • experiments with new metrics using a multiple choice assessment from a high school class; • two metrics that better analyze student behavior and identify students who need to receive more attention from the teacher; and • emphasize priority topics for students to study in order to improve the learning quality.

Review of metrics in the context of digital education
Traditional learning environments have been modified due to the increased technology applied in education. Thus, the addition of digital devices makes it possible to create new ways for educational assessment. Technology in education requires new ways to measure knowledge acquisition, individual behavior and the quality of collaboration during activities in virtual learning environments.
Although technology has been used in education, most works still adopt the traditional score as a parameter to measure student learning. However, some works use emoticons (Sales, Barroso, & Soares, 2007), or fuzzy logic (da Silva, Mendes, & da Mota Alves, 2013), but the assessment is still based on hits or mistakes.
The most common metric used to calculate student performance is Traditional Score (TS). Traditional score is typically an accuracy measure between the quantity of correct answers (c) and the total number of questions (n). From this metric, we can define the error rate as the complement of the traditional score by e = 1 − T S. The traditional score and error are normalized in the range [0, 10], according to the Equation 1.
However, we argue that traditional score is not enough to correctly assess student performance. Next, we present some metrics found in the literature to evaluate the degree of student knowledge and produce a score from an assessment.
A set of metrics that considers student activity history was described by Biswas and Ghosh (Biswas & Ghosh, 2007), which are: (a) Level of Understanding; (b) Student Learning Rate, described below; and (c) Difficulty level of a subject, topic or concept.

Level of Understanding -L u
This metric quantifies the relationship between difficulty indices, response time, and deviation. The difficulty indices (topic, concept and question) are described in Table 1. The response time is applied to catch the student's blind guesses and it has to be compared with the expected response time provided by the question author. There are two classes of Response Time, the Blind Guess with a value of 5 and the Normal Answer (or Educated Guess), with a value of 1. The deviation parameter is given according to the answer classification shown in Table 2, where 0 means "no idea" and 5 means "perfect match" or correct answer.
Biswas and Ghosh (Biswas & Ghosh, 2007) emphasize that the values of Table 1, Table 2 and Response Time were not derived mathematically, and are only applied to differentiate the   (Biswas & Ghosh, 2007) introduced the definition of Student Score, defined by SS(s, i, o), that presents the score of a student s on the i-th evaluation in relation to an element o, where these elements can be subject, topic or concept. Thus, student learning rate is the average increase in student score with respect to a set of evaluations. This permits the observation of continuous evolution and is expressed by the Equation 3.
where N is the number of questions about the ontology element o, and i is a variable expressing a specific assessment.

Proposed Metrics
As presented in Section 2, the existing metrics are only based on hits or proximity to the correct answer, such as the Traditional Score (TS) or Level of Understanding (L u ), which does not provide any information about student behavior in the assessment. Therefore, this paper proposes new metrics to overcome such limitation, which complement the evaluation of student learning in a digital education context.
The metrics can be divided in two parts: (i) five isolated metrics; and (ii) three metrics based on the combination of other metrics.

Isolated Metrics
The isolated metrics are: Weighted Score, Question Doubt, Assurance Degree, Student Response Time, and Level of Disorder.

Weighted Score
Weighted Score (WS) could be a solution to assess students, since we consider that traditional score is not enough to correctly assess student performance, as it only measures mistakes and hits and does not capture when a student almost gets an answer correct or makes a drastic mistake.
This metric is a learning score in the range of [0, 10], which is based on the weight (w i ) of each selected answer and the maximum weighted punctuation (mwp). Equation 5 shows the W S formula, where the weight value of the answers of each question (w i ) is an integer ranging from 0 to 4, where 0 is a completely wrong answer, 4 is a completely correct answer, and the others are intermediate answers (see Table 3). Obviously, if w i is equal to 1, 2, or 3, the answer is wrong. However, w i = 1 means almost wrong, and w i = 3 means almost correct. This metric is a fairer way of measuring learning performance, since it considers if a student is close to the correct answer. This metric can also be used to suggest what topics to focus on. The maximum weighted punctuation (mwp) is simply the total amount of questions in the questionnaire (|q|) multiplied by 4.
where w i is from Table 3; |q| is the total number of questions in the questionnaire. Table 4 is used as an example to explain some metrics proposed in this paper. It shows the grades of a student according to weights of the answer options selected in two subjects.
In case of Subject "A", the student missed questions Q2 and Q4. Thus, using Traditional Score, the student's grade is However, using the Weighted Score (WS), the student does not completely miss all of the questions since the weights (w i ) of each question are greater than 0. Therefore, applying Equation 5 results in: Thus, the Traditional Score (TS) is 6.67, while the Weighted Score is 8.75. Therefore, WS metric may be a better option for assessing student performance when compared to the Traditional Score (TS), since the answers to Q2 and Q4 are not completely wrong.
In case of Subject "B", the student missed questions Q1, Q2, Q3, Q4, and Q5. Thus, using Traditional Score, the student's grade is However, using Weighted Score, the student has: In this case, the discrepancy between WS and TS is more evident.

Question Doubt
Question Doubt (QD) refers to the number of times that a student returns to the same question and changes it. The Question Doubt for question i (QD i ) is calculated by Equation 6, where m j represents these changes on a given question j.
For instance, let's consider a Current file Overview multiple-choice question. If a student chooses an answer to question i and after a while decides to change it, then m i = 2 and QD i = 1. However, if the student chooses an answer to question i and does not make any changes, then Besides that, if no answer is selected, hence m i = 0 and QD i = −1. This allows us to separate the answered questions from the unanswered ones.
Tables 5 and 6 show a sample of "Markings" and "Question Doubts" for a six-question questionnaire, where the application of Equation 6 can be verified in the data from 4 students. In those tables, we can see that Student 1 did not mark questions Q1 and Q3; Student 2 only marked all questions once; and Student 4 marked questions Q1 and Q6 9 different times.
In addition, to facilitate data visualization for the teacher, the metric Question Doubt can be depicted as a bar graph ( Figure 1).

Assurance Degree
Assurance Degree (AD) measures the self-confidence of a student when answering a set of questions (from a questionnaire or parts of it) based on the relationship between the total number of correct answers (c -see Section 2) and T is the sum of all the answers of each question (m i ) of a questionnaire (or parts of it) containing n questions. Equation 8 depicts the AD formula.
For example, given a multiple-choice questionnaire, with three possible answers, if a student chooses the first option and changes it to the second option, and later changes it again to the third option before submitting the final decision, we can infer that this student was not sure about their final choice. This could be because the student is confused about the concepts or is able to self-correct. However, b combining the AD with the T S (Traditional Score -see Section 2) or QuCL (Questionnaire Comprehension Level -see Section 3.2.2), we can observe different student behaviors and, possibly detect which students have more difficulties.
Therefore, Table 7 shows some extreme cases that can be analyzed to understand the AD potential. In the first case, suppose that a student never changes their original answers (c = T ), therefore, their assurance degree is maximum (AD = 1 = 100%). The second case occurs when a student never changes their answers, but all of them are wrong, indicating that they interpreted the concepts incorrectly (c ≈ 0 or c ≪ T ), in which case AD is the minimum (AD ≈ 0 or AD ≈ 0%). Finally, the third case characterizes a less problematic behavior in which the student changes their answers many times, but the final score is the maximum (c ≈ |q|) indicating an ability to selfcorrect (0 < AD < 1). These cases illustrate the main differences between the simple analysis of the Traditional Score and the new Assurance Degree. Practical examples are shown in Section 5.

Student Response Time
The Student Response Time (SRT) measures the time spent to answer each question or a set of questions. The time spent to resolve specific questions could indicate that the learner did not understand the evaluated topic, the question has a very high difficulty level or it must be rebuilt.
Measuring the response time of a question can be crossed with the Assurance Degree (AD) (Section 3.1.3), which may help to visualize the need for further reinforcement of previously studied content or any recommendation to modify the question/questionnaire. For example, a low assurance degree for a given question (labeled as 'Easy' by the teacher) associated with a high response time to that question may indicate that the specification of the problem would need to be revised or students did not understand the topic well.

Level of Disorder
The Level of Disorder (D) is a metric that uses the concept of entropy from the information theory field, which helps quantify the degree of uncertainty of a random variable or the outcome of a random process (MacKay, 2003). Our main assumption here is that the teacher orders activities in an increasing degree of difficulty, but criterion is subjective and students may experience different degrees of difficulty compared to the teacher. In other words, the D metric represents the disorder level of the events recorded in a log file.
The entropy was originally defined by Shannon (Shannon, 1948) based on the probability mass function as: where p i is the probability of occurrence of the i-th symbol in a set N = {1, 2, . . ., n}. One of the major problems with this measurement is how to obtain p i from an unknown probability distribution function (PDF). We only consider two symbols (p 1 and p 2 ) since we only consider two possibilities, namely, in order or out of order. Thus, we can infer p 1 and p 2 from the sequence of events as a discrete temporal series, i.e., p 1 is the frequency of events that happened in order, while p 2 is the frequency of events that were not answered in the expected order.
The pseudo-code of the Algorithm 1. Consider the following sequence of answers S = {1a, 2a, 3c, 5b, 1b, 4c, 1a}, where the steps carried out to compute the Level of Disorder are: 1. to traverse S comparing the number of the correct answer (s i ) with the number of the following answer (s i+1 ). Every time that the condition s i ≤ s i+1 is satisfied, the counter p 1 must be incremented, otherwise the counter p 2 must be incremented; 2. to normalize p 1 and p 2 to obtain the probabilities; and 3. to apply the Equation 9.
Algorithm 1 Level of Disorder end if 8: end for 9: D = − p 1 p 1 +p 2 log p 1 p 1 +p 2 − p 2 p 1 +p 2 log p 2 p 1 +p 2 In our proposed method, the Level of Disorder (D) is useful for quantifying and understanding student behavior when answering questions during the assessment. When D = 0 , the student answered all questions strictly in a sequence. However, the higher the value of D, the greater the disorder, which provides clues about the learner's difficulties to answer a set of questions. Therefore, this metric must be associated with other metrics such as Student Response Time (SRT) or Grouping Deviation (GD) to confirm the hypothesis. Numerical examples and more details are shown in Section 5.

Composition of Other Metrics
The metrics based on composition of others metrics are: Question Comprehension Level, Questionnaire Comprehension Level, and Priority.

Question Comprehension Level
The Question Comprehension Level (QCL) measures the student's learning of the content considering the difficulty level of the question and the time it takes for the student to answer the question. Thus, QCL is based on Question Difficulty Index (QDI), Content Difficulty Index (CDI) from Table 8, the Weight of the student's answer (see Section 3.1.1 from Table 3) The values of QDI and CDI can be obtained from Table 8, where the value '1' represents the least difficult content or question, '3' is considered a normal difficulty and '5' corresponds to a more difficult concept to understand. Thereby, QCL considers the time the student took to answer the question, according to the  Table 9 demonstrates five case studies.
where t is the maximum expected time for the student to answer the question.
In Table 9, Q1 implies that the student hit the question, (w 1 = 4), but answered it very quickly (SRT ≤ t/4 ∴ 60 ≤ 62.5), which may indicate that he tried to guess (or cheat) the answer. In this case, by Equation 12, the QCL will only be 25%.
On the other hand, Q2 and Q3 were answered within the expected time (t/4 < SRT ≤ t). That is, (62.5 < 70 ≤ 250) and (45 < 60 ≤ 180), respectively. However, only Q2 is completely correct, obtaining a QCL equal to 100%. Q3 is almost correct, therefore, its QCL is 75%. In these specific cases, we can note that the QCL of a student only depends on the weight of his answer. In the third case, namely, when learners exceed the expected time (SRT > t ∴ 300 > 250), a small value is discounted to distinguish such student from the second case, independent of his answer. Therefore, if this student is more delayed than the prediction, it is likely that their comprehension level is a bit lower than the student that answered in the expected time. Therefore, even though the student answers question Q4 correctly, by Equation 12, their QCL will be 99.45%.
Finally, the fifth case shows that the learner also exceeded the expected time (SRT > t ∴ 650 > 300), but answered the question incorrectly. Thus, recieved a weight equal to 2. So, the learner score is calculated according to the third case of Equation 12 and is decreased proportionally to exceeded time. Thus, their QCL will be 49.42%.

Questionnaire Comprehension Level
Unlike the Question Comprehension Level (QCL) metric, the Questionnaire Comprehension Level (QuCL) measures the student's learning about the content related to a questionnaire (or a set of questions). This metric is based on QCL and the complement of the Assurance Degree, that is, 1 − AD for each question. Equation 13 details how to calculate the QuCL. In this case, for Subject 'A', while T S A = 6.67, and W S A = 8.75, QuCL A = 77.61%, and for Subject 'B', while T S B = 1.67 and W S B = 7.92, we have a QuCL B = 58.54%, where we not only consider the weights, hits and errors, but also the difficulty level of each question. We see that this metric uses elements of almost all the metrics defined so far and can more accuratley demonstrate student's understanding than Traditional Score (TS).

Priority (P) (see Equation 14
) indicates the relevance of a subject (or topic) to be studied by a student based on the scores obtained by the Traditional Score (TS) and Weighted Score (WS).
As previously shown (Subsection 3.1.1), for Subject "A" we have T S = 6.67 and W S = 8.75; and for Subject "B", T S = 1.67 and W S = 7.92. Therefore, we can use Equation 14 to calculate the priority for each subject resulting in: P A = (10 − 6.67) · 8.75 10 = 2.914 P B = (10 − 1.67) · 7.92 10 = 6.597 Where, the priority for Subject "A" is 2.914, and 6.597 for Subject "B". Observing Table 4, we can see that despite the better overall performance of Subject A, Subject B has more priority and, therefore, is recommended to study more. This happens because the student almost hit more questions in Subject B, which means that a low level of effort for learning and reaching a high performance is needed.

Experiments
To demonstrate our proposed metrics, we developed a simple educational platform that includes a multiple-choice web-based questionnaire containing five possible options for each question. Therefore, students can select or change their answers as many times as they want before giving their final answer. There are also navigation buttons that allow students to review their answers. As presented in Section 1, this platform collects some additional data at runtime, such as the time students spend on each question and how many times the student changed an answer to the same question. Details about this educational platform are beyond the scope of this paper.
The data used in this experiment is restricted to one 10th grade high school class with 33 students. We implemented a questionnaire with 40 questions including the following subjects: Portuguese, English and Spanish Languages, History, Chemistry, Physics, Sociology, Mathematics, Geography and Biology. Four hours was the maximum time given to answer all questions.
The interaction of the students with the questionnaire was recorded in a log file in CSV (Comma Separated Values) format, where the question ID and timestamp of each event were recorded. Thus, the set of events includes the selected questions and all chosen answers. The metrics were calculated in Python and stored in a NoSQL database using MongoDB.

Results and Discussions
This section presents some extracts from collected data, aiming to demonstrate the possibilities offered by the proposed metrics.

Assessing Students
In order to analyze the metrics, we used a frequency distribution method to classify the students into a set of groups (see Table 11), where the Floor value is a closed interval and the Ceiling value is an open interval. The total number of groups is given by the square root of total students, that is k = √ 33 ≈ 6. Considering that the groups should be in the interval [0, 1], the amplitude h for each group is given by h = (1/k) = (1/6) = 0.1667.   Figure 2 shows the total number of students in each group according to TS, WS, AD and QuCL metrics, which describe the students' performance on a set of questions. If we add up all the bars of all metrics, the value is always 33. When considering TS, AD and QuCL metrics, most students are in the groups 3-4, and consequently, their scores are between 0.3333 and 0.6667 (see Groups 3 and 4 in Table 11). On the other hand, when considering the WS metric, most students are in the groups 4-5, and consequently, their scores are between 0.5000 and 0.8333 (see Groups 4 and 5 in Table 11), indicating that this metric gave a higher score than the others. However, it is important to note the TS metric includes most students (19 people) in Group 4 (from 0.5000 to 0.6667); and QuCL and AD show a curve similar to a normal distribution. Figure 3 presents the quantity of students who recieved the minimum score for approval in Brazilian high school, i.e., a score equal to or higher than 0.5. Firstly, we confirm the tendency of WS to give a higher score than other metrics, since thirty students obtained a score higher than 0.5. This occurs because this metric is based on the weight of the answer, which gives some points when the answer is close to the correct one. Almost half of the students were not completely sure of their answers (AD metric). However, most of the students have high comprehension levels (QuCL metric).   Furthermore, after calculating and ordering the average of the priorities from all students in the class, Table 12 (normalized in the range [0, 1]) shows that three subjects with the highest priority are English Language (39.3%), Geography (37.3%) and Portuguese Language (32.2%), respectively. Thus, it can calculate the priority topics for each teacher to review in the classroom with all students. For instance, in relation to the subject Geography, Table 13 presents "Earth Dynamics" as the highest priority topic, followed by "Scale" and "Cartography", respectively. Figure 4 shows the average time spent on each subject, according to SRT Metric (Section 3.1.4). We can see that Math, Physics and Portuguese Language were the subjects where students spent the most time. All these subjects are above the general class average. In relation to Portuguese Language, Table 14 compares the expected answer time to the average effective answer time of all students, where only questions 4, 6 and 7 presented effective times that were lower than the expected time. This situation slightly decreases the QuCL metric since students fit the third case of Equation 12, i.e, SRT > t.  As presented in Section 3.1.5 (Level of Disorder), when disorder is equal to zero (D = 0) this means that the student answered in a rigid way, from the first to the last question. Table 15 presents the Level of Disorder (D) metric considering all subjects. The Level of Disorder is a number in the range [0, 1], but in that table it is shown as a percentage. The columns are: (i) the average disorder of each subject including all students; and (ii) the percentage of students with non-zero disorder in relation to class size.

Assessing Subjects
Thus, the column "Average (all)" shows that the highest disorder considering all students was the subject Mathematics, with an average of 32.52%. Besides that, in column "Percent.(D > 0)", Mathematics has the highest proportion of students with a level of disorder, i.e., 69.70%. This result could indicate that students may have more difficulties with Maths. This can be corroborated by Figure 4, which shows that students spend more time on Maths than on other subjects. However, even though almost 70% of the students answered the questions out of order in Maths, its level of disorder can be considered low since the average disorder level considering all students is 32.52%. This information should be considered in the analysis.  (18) In addition, we could compare the level of disorder with other metrics such as Assurance Degree (AD) and Student Response Time (SRT) to verify if the high disorder is due to student confusion (low assurance degree) or because they did not know the answers (high grouping deviation or time spent).
For example, regarding Mathematics, Figure 4 shows that the average time spent to answer a question is higher than all others subjects. We can also see in Tab. 16 that the average Assurance Degree (AD) of the class is only 36.29%. This implies that students had more difficulty in this subject than in others, as they spent more time and were less sure of their answers with high deviation in relation to correct answers.

Metrics Composition: Assurance Degree vs Questionnaire Comprehension Level
In order to determine the metrics Assurance Degree (AD) and Questionnaire Comprehension Level (QuCL), we clustered the results of both metrics into four groups. The chart in Figure 5 shows the composition. Therein, we plotted two perpendicular lines at the points 0.5 on each axis, as this is the grade established for student approval in most Brazilian high-schools.
Considering the concept of Cartesian Plane quadrants, the first quadrant contains students above or equal to 0.5 in both metrics, i.e., these students have good performance both in QuCL and in AD and, because of this, we do not need to worry when compared to students in the others quadrants. For example, the three students with QuCL greater than 0.8, have an AD greater than 0.75. This means that they have very high comprehension and confidence. On the other hand, in the third quadrant, we note students in an opposite situation, i.e. with QuCL and AD below 0.5. This situation is more worrying because students have both low comprehension and assurance levels. In Figure 5, all six students in this quadrant have an AD below 0.3, i.e. they have an Assurance Degree below 30%. A more critical case of QuCL is one student with 0.286. When analysing both metrics, the students with more difficulties are the three students with AD between 0.1 and 0.3 and QuCL between 0.2 and 0.4.
The other quadrants have students with different behaviors and needs. For example, in the second quadrant, there are students with high QuCL (above 50%) and low confidence (AD below 50%). This means that these students are getting the questions right, but still have a lot of doubts. However, even though these students have many doubts these results show that they are able to self-correct.
Finally, the fourth quadrant presents students with high degrees of confidence, but low levels of comprehension. This means that students could have answered the questions incorrectly or very quickly, possibly trying to guess. To better understand each behavior, it is necessary to verify the metrics of each specific question or set of questions.
Thus, in the next section, we present a detailed analysis of the most critical case.

Analyzing students from the Third Quadrant
As stated in Section 5.2, Figure 5 shows that students in the third quadrant have low comprehension and assurance levels and need more support. Therefore, we focus on some of them to demonstrate how the information from new metrics can be used to help overcome their difficulties.
In Table 17, we compared the metrics of the five students with the lowest QuCL of 3rd quadrant, i.e., values between 0.284 and 0.440. All data in Table 17 were normalized between 0 and 1 to make it easy to compare metrics. Furthermore, Figure 6 shows the time spent by students on each question of Portuguese Language, where the evaluated topics are available in Table 19.  Table 17 shows that for this group of students, TS is between 0.125 and 0.500, W S is between 0.344 and 0.688, AD is between 0.111 and 0.286 and just two students presented Disorder above 0%. It is worth noting that a low assurance degree may have occurred not because students frequently changed their answers, but because they missed many questions.
We can confirm this hypothesis by simply looking at the total number of students that got it right. In this case, the data from Table 18 shows that no one hit questions 5 and 7, only one student got the questions 1, 3, 6, and 8 correct, but four students got questions 2 and 4 correct. However, only using T S metric (common hits) we cannot confirm which topics students must prioritize. In view of this, we calculated the P metric for this questionnaire in Tables 20 and 21.    Table 19 shows the topic ID, its description, and which question evaluated this topic. In Table 20, the priority of the topic calculated for each student is presented in descending order. So, students "0322", "0290", and "0320" must prioritize studying "Text Interpretation"; student "0304" has to focus on "Linguistic Variation'; and student "0398" has to focus on "Verb". Afterwards, the second priority study area is suggested to students, and so on.
Aiming to best support the teachers in terms of reinforcement classes for all students, Table 21 shows the priority of each topic for the class (not for a specific student). In this case, the topic with highest priority is "Linguistic Variation", followed by "Verb", "Text Interpretation" and, finally "Orthography". Considering the time spent by these students and analyzing Figure 6, we can see that some students answered questions faster than the class average. In some cases, we might consider this a guess. See student 0322-questions 3 and 6; student 0290-question 2; student 0304-question 5; and student 0398-question 3.
By comparing the Table 14 and Figure 6, we can see that student "0322" exceeded the time for Question 2 by 03:22 min and the student "0398" by 03:33 min. Similarly, they exceeded the time for Question 8 by 01:58 and 02:50 min, respectively. From the data of these students for these specific questions, student "0322" hit Q8 but missed question 2, and the opposite happened with student "0398". Thus, the extra time spent on these questions was only helpful for one of them. Besides that, from Table 22, the Question Doubt (QD) for both questions and students was equal to zero, indicating that they thought about the answer for the whole time and only answered once. Despite the low value of QD for these questions, the Assurance Degree (measured for the whole questionnaire) is low for both students because, in general, they missed most of questions according to T S metric.

Conclusion
Assessment is the systematic process of using empirical data to measure knowledge, skills, attitudes, and beliefs. During assessment, teachers try to improve a student's path toward learning.
On the other hand, the evaluation focuses on grades and may reflect various situations within the classroom, as well as what was actually learned. Therefore, assessment is adopted to measure student learning, while evaluation is more often applied to measure performance. In this case, any metric can be adopted to measure either learning or performance.
In this paper, we propose several metrics for student assessment. The main purpose of these metrics is not only to assess student learning of the studied content, but also to provide opportunities for teachers to improve student learning. However, although we present some learning analytics based on the proposed metrics, the analysis, inferences, and recommendations are outside the scope of this paper.
Most educational institutions still adopt the traditional score, namely, based on hits or mistakes as a parameter to measure student learning. However, we argue that traditional score is not enough to correctly assess student's performance. Thus, this paper presents some new metrics to assess learning and give feedback to students. Despite the several kinds of assessment, we focus on formative and summative assessments, which primarily aim to educate and improve student learning.
The proposed metrics are divided in two parts: (i) isolated metrics (Weighted Score, Question Doubt, Assurance Degree, Student Response Time, and Level of Disorder); and (ii) metrics based on composition of other metrics (Question Comprehension Level, Questionnaire Comprehension Level, and Priority).
The Weighted Score (W S) is a learning metric that is different from Traditional Score (T S) as it considers when a student's answer is wrong but close to the correct answer. Thus, to determine scores based only on right/wrong answers is not enough to measure student's understanding of a subject nor to help improve the student's path toward learning. The Weighted Score could help dismiss such discrepancies. Question Doubt (QD) is a metric that calculates the number of times a student returns to the same question and changes it. This metric is important for determining the doubt level of a student in a question, which could show that a student is not confident in a topic or concept of a given question. Assurance Degree (AD) measures the self-confidence a student has when answering a set of questions. This metric relies on the total number of correct answers and the number of times the student answered the questions. Using this metric, we can estimate the level of assurance of a student, and if a student changes their answers often, their assurance level will be low. This metric is usually used in combination with other metrics. Student Response Time (SRT ) is a metric that measures the accumulated time spent to answer each question. This metric is very useful as it can provide information about student's difficulties to answer a question, when the student may not be confident with the topic/concept or if the question has a high level of difficulty. The Level of Disorder (D) is a metric that uses the disorder degree of the events. We assume that teachers present questions in order, from first to last. The Level of Disorder (D) can quantify and understand student behavior while answering questions during the assessment.
The Question Comprehension Level and Questionnaire Comprehension Level are composed of other metrics. The Question Comprehension Level (QCL) measures student's learning about a content considering the difficulty level of the question, and the Student Response Time (SRT ), which is the time it takes the student to answer that question. In this case, we consider three cases: (i) when a student answered faster; (ii) when a student answered in the time determined by the teacher; and (iii) when a student answered after the expected time. On the other hand, the Questionnaire Comprehension Level (QuCL) measures student learning of a content related to a questionnaire (or a set of questions). This metric is based on Question Comprehension Level and the Assurance Degree.
In order to give feedback to students, we propose the Priority (P) metric, which prioritizes a subject or topic to be studied by the students.
We present the results of an experiment including 33 10th grade high school students. Therefore, we describe all calculated metrics of the data collected from a 40-question questionnaire. When presenting the metrics, it was possible to see feedback possibilities for students and teachers.
Another outcome was a clustering-based method based on the composition of two specific metrics, namely, Assurance Degree (AD) and Questionnaire Comprehension Level (QuCL). In this case, we clustered both metrics into four groups considering the grade established for student approval( 50%) in most Brazilian high-schools. The results were plotted in four quadrants of a Cartesian Plane. The first quadrant contains students above or equal to 50% in both metrics, i.e., students with good performance both in QuCL and AD. On the other hand, in the third quadrant, we note students in an opposite situation, i.e. with QuCL and AD below 50%. This is a more worrisome situation because these students present both low comprehension and assurance levels. The specific students in the third quadrant were analysed to discover why they presented poor performance. We conclude that such data is useful for feedback and recommendations.
Regarding future research, we have a lot of data and there are many machine learning methods that could provide several learning analytics. Another future goal is to automatically provide these methods, based on the metrics, feedback and recommendations to both students and teachers.