{'fraction_non_alphanumeric': 0.03155708641475028, 'fraction_numerical': 0.019856144260966466, 'mean_word_length': 5.011875761266748, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Driver Drowsiness Detection Using Ensemble Convolutional Neural Networks on YawDD

Driver drowsiness detection using videos/images is one of the most essential areas in today's time for driver safety. The development of deep learning techniques, notably Convolutional Neural Networks (CNN), applied in computer vision applications such as drowsiness detection, has shown promising results due to the tremendous increase in technology in the recent few decades. Eyes that are closed or blinking excessively, yawning, nodding, and occlusion are all key aspects of drowsiness. In this work, we have applied four different Convolutional Neural Network (CNN) techniques on the YawDD dataset to detect and examine the extent of drowsiness depending on the yawning frequency with specific pose and occlusion variation. Preliminary computational results show that our proposed Ensemble Convolutional Neural Network (ECNN) outperformed the traditional CNN-based approach by achieving an F1 score of 0.935, whereas the other three CNN, such as CNN1, CNN2, and CNN3 approaches gained 0.92, 0.90, and 0.912 F1 scores, respectively.


Introduction
In the last few decades, computer vision applications such as drowsiness detection for the safety and well-being of employers have been an important and concerning factor among researchers and is one of the hot topics in the field of Artificial Intelligence. With the advent of revolutionary, sophisticated, and human interactive machines and technologies, the research has been capable of developing real-time applications for human safety [1]. Detecting drowsiness by detecting behavioral features such as eyes, mouth, facial features [2], etc., is one of the very important aspects considered by the researchers [1]. However, different approaches can be regarded as for driver drowsiness arXiv:2112.10298v1 [cs.CV] 20 Dec 2021 detection, such as vehicular-based approach (such as steering movement detection, force on acceleration pedal detection, etc.) and physiological-based approach (EEG, ECG, etc.) [3]. A lot of work has been continuously carried out to improve drowsiness detection by enhancing the accuracy and precision of the model [4,5,6]. The target or aim to driver drowsiness detection can be achieved effectively by using some of the techniques mentioned above such as neural network,CNN, and deep CNN which has shown remarkable results. Video-based driver drowsiness detection with deep learning techniques includes different parameters such as proper feature extractions, adequate computing power, and synchronized human-computer interactions. Additionally, behavioral feature extraction plays a significant role in drowsiness detection. To develop a suitable video-based driver drowsiness detection model, proper facial frames must be extracted within the working domain [7,3]. Using deep learning [8,9,10,11] techniques instead of conventional machine learning approaches provides better results in terms of accuracy, precision, and prediction [12]. Convolutional Neural Networks (CNN) is one of the good image classifier approaches because of its capability to detect spatial and temporal features in the image with the application of appropriate filters or kernels [13]. Figure 1 displays referenced literature in drowsiness detection using deep learning from 2012 to 2020. Different training architectures have been implemented by many researchers to improve the performance of drowsiness detection. The behavioral approach such as eye-tracking, yawning and face detection plays an important role in drowsiness detection [3]. In the eye-tracking-based approach, Viola-Jones algorithm demonstrates faster performance compared to deep neural networks. However, when the object is moving, the performance of viola-jones algorithms in eye or face localizing significantly degraded [14]. A recent study conducted by Revelo el al. (2019) demonstrated that DNN and LSTM based approach performs well for yawning detection, where Spatio-temporal feature analysis was considered as one of the main factor [6]. Xie et al. (2019) also used DNN approaches to detect the human drowsiness in real time. Their model showed robustness even in different situations, such as in the case of lack of illumination, positions, and view angles [15]. Some of the recent CNN based approaches such as AlexNet [16], VGG-FaceNet [17], Flow ImageNet [18], ResNet [19] also shows promising result in detecting drowsiness. However, the performance of those CNN artchitecture varies depending on the designated computer vision problems. For instance, AlexNet performs well compared to other models in different contexts and environmental shifts, such as indoor and outdoor, day and night; VGG-FaceNet is performed well for the extraction of facial features such as racial groups. On the other hand, for behavioral characteristics and head movements, Flow ImageNet, and for hand gestures, ResNet demonstarted better performance [20]. Ensemble-based techniques, on the other hand, have recently gained much popularity due to their versatility in application. Typically, Ensemble approaches are used to train a "large" number of CNN models. Voting or averaging the total performance during the prediction is used to evaluate the performance. To begin, multiple neural networks are trained, with each network returning the likelihood of each class mark. Before the training phase, the models chosen to be Ensemble, which is loaded with their weights and architecture [21], those probabilities are weighted together, and the final classification is obtained [22]. There has been substantial progress in behavioral feature recognition for driver drowsiness detection, just as there has been lot of progress in signal processing for physiological feature-based driver drowsiness detection. A lot of multimodel methods have been considered to solve these issues [14,23]. In the last decades, there has been a lot of deep learning based research has been conducted in the field of computer vision on driver drowsiness detection to ensure the safety of the driver [24]. This paper aims to develop a video-based driver drowsiness detection model using the deep learning technique on the YawDD dataset. However, video-based driver drowsiness detection is not a simple task, as demonstrated in Figure 2, and requires multiple pre and post-processing approaches.

Methodology
The driver's behavior for drowsiness detection is usually categorized as alert or drowsy or attentive or non-attentive. So, depending on these classes' images/frames are extracted from input video samples [25]. The number of frames to be extracted varies based on the designated problem statement, complexity of the work, computing ability and processing time. The frames extracted from the video database are then used for Face detection, which is the next step after pre-processing. It is one of the very common approaches that can detect drowsiness by facial features like yawning, eye blinking, etc. Hence, it is very important to detect faces to obtain facial features to move further for processing and recognition. Many face detection algorithms are being used by researchers, such as Viola-Jones, Local Binary Pattern Histograms (LBPH), Eigenfaces, Fisher face, OpenCV, and, DLIB [26,27]. Conventional algorithms acknowledge the face by identifying and extracting facial highlights from the face image. For example, many researchers have been using facial landmarks (like 68 landmarks) [28] to extract the facial key features, which calculates and examines the state of the eyes, mouth, etc. The extracted features would then be used to look for similar matching features of other images. In recent years the industries have moved and focused on deep learning techniques for different applications. CNN is a very strong performer, has been used for the calculation of acknowledging face detection. These calculations, which are achieved in the form of pixels from images, describe the arrangement of the picture information. These assimilated features include the PERCLOS (percentage of Eye Closure) [29,30], the ratio of mouth opening [31], movement of the head or pose variation [32]. CNN separates or distinguishes a huge number of these highlighted features from the image very well.
The cropping procedure comprises removing the portion of unwanted regions from the picture to remove incidental unnecessary features from the image during the face detection to improve its surroundings or highlight the regions (face) from its background. Once the cropping is done, the image resizes, and the size of the image varies and depends on the requirement. During this experiment we have resized all the image into 90x90 pixels to reduce the processing time while preserving the features or attributes of the image with this specific dimensions [13]. Note that, the classification accuracy depends on how well the features are extracted during image pre-processing. Therefore, classification accuracy will automatically improve by optimizing the selected features [33].

CNN structure
Convolutional neural network (CNN) is one of the most well-known deep learning (DL) algorithms on which, when an image is passed; it applies learnable weights and bias to a different point in the image; this way, it is able to separate features from one another. CNN requires less pre-processing compared to other DL algorithms. The design of CNN is inspired by the visual cortex of the human brain. Convolutional Net is an arrangement of different types of layers, and these layers of CNN perform different operations when input passes through these layers. The general architecture of CNN is described as follows: • CNN layers-Convolutional layers are the layers under which a deep CNN filter is added to the original image or other feature maps. A filter in a CNN is like a weight matrix which used to produce a convoluted output; we multiply a part of the input image. For example, let's assume we have got a size of 28*28 images. To form what is known as a convoluted output, we randomly allocate a filter of size 3*3, which is then multiplied with various 3*3 parts of the image.
• ReLu-ReLu is an activation function also known as rectified linear activation unit. The main advantage of using ReLu is that it has a constant derivative value for all inputs greater than 0. The constant derivative value helps to train the network faster. Instead of sigmoids, the recent networks prefer using ReLu activation functions for the hidden layers. The function is defined as: The output of the function is 1 when X>0 and 0 for X<=0. Figure 3 demonstrates the graphical representation of ReLu. • Pooling layer -The pooling layer is used to reduce the dimensionality of the network. The pooling layers are usually two types max-pooling (takes the maximum value in a particular filter region) and average pooling (takes the average value in a filter region). For example, the most common type of pooling is a pooling layer of filter size (2,2) using the MAX operation. What it would do is, it would take the maximum of each 4*4 matrix of the original image. Figure 4 displays the max pooling output with 2x2 filters when applied on 4*4 matrix.
• FC-FC also known as Fully connected layers, that are placed before the classification output of a CNN and are used to flatten the results before classification. Here in Figure 5 illustrates the flow diagram of CNN based approached applied on image classification.

Convolutional Neural Net 1
Three convolutional layers, one pooling layer, two fully connected layers, and a softmax unit with polling applied to each layer to comprise the Convolutional Neural Net 1 (CNN1) architecture. The first, second and third layer contains 3x16, 3x32, filter, 3x64 filters respectively. Each layer is followed by rectified linear units (ReLU) and a 3x3 stride and with 2 max-pooling layer. The batch normalization layer is followed by two FC layers and the output layer, respectively. Optimization was performed using the Stochastic Gradient Descent (SGD), with a learning value of 0.001 and a mini-batch size of 32. The maximum number of iterations was set to 1200. To avoid overfitting, the first two FC layers are followed by dropout layers with a 50 percent dropout ratio.

Convolutional Neural Net 2
To develop the Convolutional Neural Net 2 (CNN2) four convolutional layers along with polling for each layer is applied. Each layer is followed by rectified linear units (ReLU) and a max-pooling layer of size 3x3 with stride 2. The first, second, third and fourth layer consists 3x3, 4x16, 3x32, 3x64 filters, respectively. Again each layer is followed by ReLU and a max-pooling layer of size 3x3 with stride 2. Layer 3 is followed by three fully connected layers with 348, 192, and 96 neurons, respectively, which followed by batch normalization layer. For optimization, adaptive moment estimation (ADAM) was utilized, using a learning parameter of 0.0001 and a mini-batch size of 64. The maximum number of iterations was set to 1250.

Convolutional Neural Net 3
To develop Convolutional Neural Net 3 (CNN3),five convolutional layers each with a filter size of 3x8, 3x16, 6x16, 3x32, and 12x32 are implemented. Each layer is subjected to polling. The convolutional layer is composed of a number of components, including the filter size, padding, and stride. Each layer is followed by ReLU activation functions and a max-pooling layer with a size of 3x8 and a stride of 2. The batch normalization layer is then followed by four FC layers each having 796,348, 192, and 96 neurons respectively. To prevent overfitting, the first two FC layers are followed by 60 percent dropout layers. Figure 6 shows the Ensemble CNN structure applied in this study. Ensemble-based CNN is used to improve the performance and accuracy of the model. When the individual CNN nets are combined to form the Ensemble net, the model's performance is increased in the form of an F1 score, which can be seen in the Results section. To create a single output, the separate outputs of all models are combined using the simple averaging procedure. Between the non-drowsy and drowsy states, a threshold value is specified. If the result exceeds the threshold value, the driver is considered drowsy; otherwise, non-drowsy. Selecting databases is one of the essential steps to build a model. In terms of suitability, usability, and detailing, the data involved should be adequate. Many datasets are available that can be used to frame the models of video-based drowsiness detection. In this study we have used YawDD dataset [4], which gathered face videos of a vast number of participants (107 participants). This dataset has been made publicly available to promote further research on drowsiness detection by yawning detection. YawDD consists of 57 male and 50 female participants who were asked to record the video of drivers sitting in the driving seat and pretending to be driving in different light conditions and different facial characteristics like participants wearing glasses and sunglasses, having beards, mustaches, etc. Each participant video lasts for about 15-40 seconds. However, we only evaluated two states/expressions in this experiment: Alert and Drowsy, with labels of 1 and 2 respectively. Some of the most widely used datasets are mentioned in Table 1 along with YawDD dataset in details. At first, we divided the data into YawDD [4] 107 normal driving (no talking), 342 videos talking or singing, and yawning while driving three categories: 70% for training, 15% for validation, and 15% for testing. On the YawDD dataset, four CNN-based approaches were utilized to assess overall performance. Table 2 shows the overall data assignment for this study.    To better visualize the overall performance of prediction, confusion matrices were used. Figure 8 demonstrates that the proposed CNN1 performs nearly identically on training, testing, and validation datasets, effectively reducing the risk of models overfitting.

Implementation and Results
The validation accuracy of the CNN2 is shown in Figure 9. Adam, an adaptive learning rate optimization technique with an initial learning rate of 0.001, was used to train the network, with a Minibatch Size of 32,4 epochs and 2416 iterations. CNN2 has a 98.85 percent accuracy rate and took 61 minutes and 45 seconds to complete. Additionally, Figure 10 depicts the CNN2 model's confusion matrix for testing, training, and validation data. The model's performance in training and testing is nearly identical, while the CNN2 performance on the validation dataset is slightly worse.
Here, the validation accuracy of the CNN3 is shown in Figure 11. With an initial learning rate of 0.001, a Minibatch Size of 64,4 epochs, and 2816 iterations, the network was trained using stochastic gradient descent with momentum (SGD). CNN3 has a 99.42 percent accuracy rate and took 67 minutes and 27 seconds to complete. The confusion matrix for CNN3 models in the training, testing, and validation dataset is shown in Figure 12. The best results are seen on the training and validation datasets, whereas the worst results are seen on the testing dataset. Table 3 displays the Recall, Precision, and F1 Score of CNN1, CNN2, CNN3, and Ensemble Net for the YawDD test dataset, demonstrating that the Ensemble net outperformed individual nets on average in terms of precision, recall and F1 score.

Discussion
Four CNN-based models for detecting driver drowsiness and alertness were proposed and assessed in this work. The results revealed the superiority of Ensemble-based techniques over traditional CNN models employed alone for this purpose. The Ensemble technique outperformed the other CNN models, achieving recall scores of.903 and.995 in the drowsy and alert stages, respectively, compared to the other CNN models. During the alert state, the CNN3 outperformed all other models, achieving a precision score of.977, which was superior to all others. When it came to precision in the drowsy state, Ensemble outperformed all other models with a score of.97, which was higher than    There are some limits to our study, which we believe present immediate potential for future research. These are as follows: • The experiment was conducted using only one dataset instead of multiple datasets.
• Instead of comparing the performance of the proposed model to that of the referred literature, only four different CNN-based techniques were proposed.
• Finally, we did not investigate the compatibility of our proposed models with real-world events in our research. Transnational research should investigate the possibility of bridging that gap as soon as possible in the future, with a higher priority.

Conclusion and Future Works
In this work, we present four distinct Deep Learning Convolutional Neural Network (CNN)-based models for detecting the driver's drowsiness and alert status using the YawDD dataset. CNN1, CNN2, and CNN3 have an F1-score of 0.972, 0.970, and 0.965, respectively, for alert classification. With an F1 score of 0.978, the proposed Ensemble approaches resulted in improved performance. On the other hand, the F1-scores for CNN1, CNN2, and CNN3 are 0.92, 0.90, and 0.912, respectively, whereas an F1 value of 0.935 indicates improved performance when ensemble methods are used. Thus, based on the findings of our investigation, we can conclude that, of the four offered models, the Ensemble-based CNN technique is the best. Some of our work's shortcomings can be addressed by conducting experiments on significantly imbalanced big data, evaluating our approaches' performance across different datasets, and constructing models with explainable artificial intelligence on a mixed dataset.