{'fraction_non_alphanumeric': 0.03237362345760913, 'fraction_numerical': 0.009751890672681437, 'mean_word_length': 5.11810064935065, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Predicting Student Failure in an Introductory Programming Course with Multiple Back-Propagation

One of the most challenging tasks in computer science and similar courses consists of both teaching and learning computer programming. Usually this requires a great deal of work, dedication, and motivation from both teachers and students. Accordingly, ever since the first programming languages emerged, the problems inherent to programming teaching and learning have been studied and investigated. The theme is very serious, not only for the important concepts underlying computer science courses but also for reducing the lack of motivation, failure, and abandonment that result from students frustration. Therefore, early identification of potential problems and immediate response is a fundamental aspect to avoid student's failure and reduce dropout rates. In this paper, we propose a machine-learning (neural network) predictive model of student failure based on the student profile, which is built throughout programming classes by continuously monitoring and evaluating student activities. The resulting model allows teachers to early identify students that are more likely to fail, allowing them to devote more time to those students and try novel strategies to improve their programming skills.


Introduction
Nowadays there is an on-growing demand for Computer Science (CS) professionals. In Europe alone, according to European Commission, next year there will be a shortage of more than 800,000 CS professionals. To make matters worse, CS courses typically present high-dropout rates, aggravating the shortage of professionals [10,26]. Accordingly, in recent years, there has been a growing interest in teaching programming concepts to young people. Incorporating CS topics in high-school curriculum's, such as computer programming and robotics, has the benefit of developing students' skills, such as: problem solving, creativity and computational thinking [8,31]. Recognizing this, many entities and organizations promote countless initiatives to promote computer programming and computational thinking [2,4,6,7,11,21,22].
Programming is a process of transforming a mental plan of cur-rent terms into terms compatible with the computer [13]. When teaching computer programming, the main objective is to empower students with the skills needed to create computer programs that can solve real-world problems. In this context, programming requires quite particular characteristics and skills that students may struggle to obtain, often in a short period. Among these, Jenkins [17] identified the following: the abstract concepts inherent to programming; the competencies and mental abilities required to decompose and solve problems; the use of specific syntaxes that students are required to memorize; and the semantics and structure new non-natural languages. As such, among the topics CS students are required to learn, computer programming is particularly difficult. This is a well-know fact, among the Computer Science Education (CSE) community [3]. Not only teaching computer programming is a recent area (when compared to other knowledge areas) but it is also a fast-changing one. Knowledge and tools rapidly become obsolete, making it harder to teach and learn even the basics.
Although many studies seek to identify the different causes that lead to a lack of student motivation and high-dropout rates there is no panacea to solve this problem. Nevertheless, to counter students' frustration that may ultimately lead to dropouts, it is important to identify learning problems as soon as possible. Accordingly, in this paper, we propose a Neural Network (NN) predictive model of student failure based on students' profiles collected during the classes. The resulting model allows teachers to early identify students that are more to fail and take corrective actions.
The remainder of the paper is structured as follows. Section 2 describes related work. Section 3 details the algorithm used to create a predictive model for the student's performance. Section 4 explains the methodology used to create the aforementioned model. Section 5 presents and discusses the results obtained and finally, in Section 5 conclusions and future work are addressed.

Related work
The first programming courses play an important role in the student success [26], because they can stimulate all passion, beauty, joy, and awe for programming [9]. Unfortunately, the lack of success on the introductory programming courses, can also be a demotivating factor that may ultimately lead to abandonment. Therefore, several works focus on predicting the success of students. Most of these are based on analysis of student interaction with the programming language compiler, such as: investigating novice programming mistakes [1]; exploring compilation behavior [16]; analyzing response to compiler error messages [23]. Other studies focus on comparing and testing traditional predictors of performance and new datadriven predictors [27]. Several works have found that the mathematical ability and exposure to mathematics courses are important predictors of performance on introductory programming courses [25,28,30]. Moreover, there are also studies, that link prior programming experience and nonprogramming computer experience to the student's programming performance [5,12,14,29]. Additionally, the role of cognitive factors, such as problem-solving, abstract reasoning, logical ability, and cognitive style for predicting the programming performance has also been studied [15,24].
Recently, Machine Learning (ML) methods have also been used for predicting student's performance on introductory programming courses. In [18], support vector machines are used to predict the students' final exam scores based on data automatically collected for instructors. In [26] a prediction model, named PreSS (Predict Student Success), based on ML algorithms is proposed, in order to predict the students' success on introductory programming courses. Moreover, the study, analyses and compares the performance of several ML methods.

Multiple Back-Propagation with a Neural Selective Input Model
Multiple Back-Propagation (MBP) is an algorithm, proposed in [20] for training Multiple Feed-Forward (MFF) networks -a special type of NN that combines a main network and space network as depicted in Figure 1. The main network contains selective activation neurons, whose contribution to the NN output depends on the stimulus presented to the network. Each selective activation neuron, k, possesses an importance factor m k p , that defines its relevance according to the pattern (sample), p, presented to the network. Accordingly, the output of these neurons, y k p , is given by (1): where ℱ is the neuron activation function, its activation, the bias and the weight of the connection between neuron j and neuron k. The farther from zero is the more important the neuron contribution becomes. On the other hand, a value of zero means the neuron is completely irrelevant for the network output and one can interpret such a value as if the neuron is not present in the network. bias [20].
Both the main network and the space network, responsible for determining the importance factors of the selective neurons, are trained together as a whole. To that end, MBP uses the same rule for updating the weights as the Back-Propagation (BP) algorithm, i.e. the main network weights are adjusted by (2): where γ is the learning rate, the local gradient of neuron k, Δ the weight change for the last pattern l, α the momentum term and the local gradient for the output, o, and hidden, h, neurons, are given respectively by (3) and (4): To deal with missing data a Neural Selective Input Model (NSIM) was proposed in [19], such that the act of obtaining the value of , represented by a random variable, ∼ Be(qj) (with Bernoulli distribution), is taken into consideration. To that end, the values are transformed by (5): using a neuron, k, with selective activation (named selective input), containing a single input, , and an importance factor set to , as depicted in Figure 2.

Methodology work
In order to build a predictive model for determining which students are more likely to fail, a profile for each student was built as described in Section 4.1. The collected data was then divided into training and test datasets. This process is described in Section 4.2. The training dataset was then used to build the model while the test dataset was used to validate it. Section 4.3 presents the metrics used to assert the quality of the models.

Building the student profiles
The idea of building a profile with the student's programming competencies is based on the same concept of current video games such as FIFA 19 or Assassin's Creed. The characters are invited to build and improve their characteristics and skills in specific areas to complete their tasks or change their level. For example, a FIFA player may train a penalty, dribble, free kicks and corner kicks practice and other actions to improve his/her abilities during the game. Likewise, we want each student to be able to improve and deepen their programming skills by performing a set of appropriate and worked exercises for each student and situation. The profiles, build throughout the programming classes, by continuously monitoring and evaluating student activities will then be used to build a predictive model able to determine which students require more help in order to overcome their difficulties and achieve the necessary set of programming competences (see Figure 3). Table 1 describes the 17 variables (attributes) collected in order to build each student profile.

Data analysis and pre-processing
This study involved a group of 85 students of an introductory programming course (Introduction to Programming), lectured to the first year, first semester, students of the CS course at the Polytechnic of Guarda (IPG), Portugal -an institution of higher education located in the interior of the country. In this course the C language is used to teach the basic programming concepts. In our opinion, our study group has very special characteristics that might affect the learning process: • Usually, the IPG Computer Science course, is not the first choice of students. Naturally, this affects negatively student's motivation and engagement. • In recent years, the score needed to enroll the IPG Computer Science course is typically low. Thus, most students entering this course are below average. • Most students reveal general difficulties in the area of CS. • Typically, students were never exposed to programming concepts nor had the opportunity to practice computational thinking activities.
Since students don't always attend classes, the compiled dataset contains Missing Values (MV). On average 13.01 ± 2.70 attributes were collected for each student and we were able to collect all the attributes only for 7 students. The resulting dataset, containing 23.46% of MV was randomly divided into a training set containing 40% of the samples (students) and a test set, containing 60 of the samples (observe Figure 3). Accordingly, the training dataset encompasses 34 samples and the test dataset encompasses the remaining 51 samples. Moreover, the training dataset contains 24.91% of MV while the test dataset contains 22.49% of MV. On average, each student on the training dataset has 12.76 ± 3.03 attributes and each student on the test dataset 13.18 ± 2.47 attributes. Table 2 presents the percentage of MV per attribute for each dataset. Moreover, Table 3 presents the number of students that approved/failed in each dataset.

Training and evaluating the models
In order to build a NN predictive model for student failure, Multiple Back-Propagation software, available at http://mbp.sourceforge.net/ was used. Several networks were trained using the train dataset while varying the number of neurons in the hidden layer.
For evaluating the performance of each network (model) and asserting its quality, we use several metrics, based on the confusion matrix, which contains the number of correctly and incorrectly classified examples for each class (Failed/Approved), in the form of true positives (TP), true negatives (TN), false positives (FP) and false negatives (FN). Figure 4 presents the confusion matrix for our problem. The accuracy given by (6): The accuracy represents the proportion of students that are correctly classified. Although this metric gives an overall estimate of our model performance, in our case, it can be misleading, since there is a big discrepancy between the number of samples of each class (see Table 3). Therefore, we also use other metrics, such as the precision and recall (sensitivity), respectively given by (7) and (8): A model presenting a high precision rate is rarely wrong when it predicts that a student will fail the course. On the other hand, a classifier exhibiting a high recall rate rarely mis-classifies a student that will fail. Usually there is a trade-off between the precision and the recall, and generally it is important to balance and maximize both. This can be accomplished by using the Fscore (9): (9) Figure 4: Confusion matrix of the model to predict which students will fail the Introduction to Programming course.

Results and Discussion
In order to build a NN predictive model for student failure, Multiple Back-Propagation software, available at http://mbp.sourceforge.net/ was used. Several networks were trained, varying the number of hidden neurons in the main network. The best network had a single neuron with selective activation. Figure 5 presents the confusion matrix of the resulting NN model in the test dataset. Moreover, Table 4 presents the performance of the model in the test data.  The resulting NN model presents high-accuracy. Only three students are misclassified. However, as depicted in Figure 5, from these, only one is incorrectly classified as approved when it should be classified as failed (FN). Ideally this value would be zero, since these students will not be given special attention, because the model predicts they will succeed and unfortunately, they will fail. Nevertheless, the proposed model allows the teacher to accurately determine which students require additional attention and intervene early on in order to reduce attrition rates.

Conclusion
In this paper we presented a Neural Network predictive model for student failure based on the student profile, which is built throughout computer programming classes by continuously monitoring and evaluating student activities. The resulting model allows teachers to early identify students that are more likely to fail, allowing them to devote more time to those students and try novel strategies to improve their programming skills. Future work will focus on more gathering additional information to explore the possibility of creating models for determining the students' performance on specific contents (e.g. defining variables, using iterative or conditional structures).