{'fraction_non_alphanumeric': 0.026829207981998466, 'fraction_numerical': 0.010583318909027967, 'mean_word_length': 4.928173556141894, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Driver Activity Recognition for Intelligent Vehicles: A Deep Learning Approach

Driver decisions and behaviors are essential factors that can affect the driving safety. To understand the driver behaviors, a driver activities recognition system is designed based on the deep convolutional neural networks (CNN) in this paper. Specifically, seven common driving activities are identified, which are the normal driving, right mirror checking, rear mirror checking, left mirror checking, using in-vehicle radio device, texting, and answering the mobile phone, respectively. Among these activities, the first four are regarded as normal driving tasks, while the rest three are classified into the distraction group. The experimental images are collected using a low-cost camera, and ten drivers are involved in the naturalistic data collection. The raw images are segmented using the Gaussian mixture model to extract the driver body from the background before training the behavior recognition CNN model. To reduce the training cost, transfer learning method is applied to fine tune the pre-trained CNN models. Three different pre-trained CNN models, namely, AlexNet, GoogLeNet, and ResNet50 are adopted and evaluated. The detection results for the seven tasks achieved an average of 81.6% accuracy using the AlexNet, 78.6% and 74.9% accuracy using the GoogLeNet and ResNet50, respectively. Then, the CNN models are trained for the binary classification task and identify whether the driver is being distracted or not. The binary detection rate achieved 91.4% accuracy, which shows the advantages of using the proposed deep learning approach. Finally, the real-world application are analyzed and discussed.

than 90% light vehicle accidents are caused by human driver misbehavior in the United States, and the accident rate can be reduced by 10% to 20% with a precise driver behavior monitoring system [1]- [5]. Therefore, the recognition of driver behaviors is becoming one of the most important tasks for intelligent vehicles. For the conventional advanced driver assistance systems (ADAS), the driver is in the center of the RVD loop. The understanding of driver behavior enables the ADAS to generate the optimal vehicle control strategies which is suitable for the current driver states [6]- [9]. Regarding the intelligent and highly automated vehicles, such as the Level-3 automated vehicles (according to the definition in Society of Automotive Engineers standard J3016), the driver is responsible for taking over the vehicle control under emergencies. At this moment, the real-time driver behavior and activity monitoring system has to decide whether the driver can take over or not.
Therefore, in this study, a deep learning-based driver activities recognition system is proposed to monitor and understand the driver behaviors continuously. The recognition models are trained to identify seven common driving-related tasks and also to determine whether the driver is being distracted or not. With this end-to-end approach, intelligent vehicles can better interact with human drivers and properly making decisions and generating human-like driving strategies.

B. Related Works
Driver behaviors have been widely studied over the past two decades. Previous studies mainly focus on the driver attention [42] and distraction (either physical distraction or cognitive distraction) [37], driver intention [8] [10], driver styles [43], driver drowsiness and fatigue detection [11]- [13], etc. The National Highway Traffic Safety Administration (NHTSA) defined driver distraction as a process that the driver shifts their attention away from the driving tasks. Four types of distraction are clarified by the NHTSA, which are the visual distraction, auditory distraction, biomechanical distraction, and cognitive distraction [38]. To understand the driver behaviors, most of the studies require capturing the driver status information, such as the head pose [14], eye gaze [9], hand motion [15], foot dynamics [16], and even the physiological signals [17] [18].
Specifically, in [19], the video information for the driver head movement along with the audio signals was collected to identify the secondary driving tasks. In [20], the drivers head pose, eye gaze direction, and hand movement were combined to identify driver activities. In [21], the driver's head poses estimation was proposed and applied to the rearend crash avoidance system. Despite the vision-based feature extraction methods, the physiological signals, such as the electroencephalogram (EEG) and electrooculography (EOG) are also widely used for real-time driver status monitoring. In [22], EEG signals were collected to predict the driver braking intention. In [23], the EEG and EOG signals were used to estimate the driver drowsiness and fatigue status. The EEG signals are proved to be closely related to the driver behaviors and can illustrate an earlier response to the human mental states compared with the outer physical behaviors.
However, as aforementioned, most of the existing driver behavior studies require extracting specific features in advance, such as the head pose angle, gaze direction, EEG, and the position of hand and body joints [24]. These features are not always easy to be obtained, and some even require specific hardware devices, which will increase either the temporal or the financial cost. Therefore, in this work, an end-toend driver activity recognition system is proposed based on the deep CNN models, which is accurate and easy to be implemented. To study the driver distraction behaviors, visual distraction, auditory distraction, and biomechanical distraction are involved. While the cognitive distraction is not considered since it has been well studied in [39] [40], which can be effectively detected with a non-vision-based approach.
Regarding the current development of deep learning techniques, significant progress has been made in the computer vision area dues to the development of deeper CNN models, parallel computing hardware, and the large-scale annotated dataset. Deep CNN models have achieved the state-of-art results in object detection, classification, generation, and segmentation tasks. Meanwhile, it has been successfully applied to some driver monitoring tasks [25] [26]. In this work, three different CNN models will be evaluated for driver activities recognition and distraction detection tasks. The only sensor required in this study is a low-cost RGB camera. Based on the report in [27], seven most common in-vehicle activities for both manual driving and automated driving vehicles are selected, which contains normal driving activities as well as secondary tasks. The CNN models take the processed images directly without any manual feature extraction procedure. By applying the transfer learning scheme, the pre-trained CNN models can be efficiently fine-tuned to satisfy the behaviors detection task.

C. Contribution
The contribution of this study can be summarized as follows. First, a novel deep learning-based approach is applied to identify driver behaviors. Unlike existing studies that require complex algorithms to estimate the driver status information, the proposed algorithm takes merely the color images as the input and directly outputs the driver behavior information. With the deep CNN models, the manually feature extraction process can be replaced by an automatic feature learning process.
Second, transfer learning is applied to fine-tune the pretrained deep CNN models. The models are trained to deal with both the multiple classification tasks and the binary classification task. The algorithm is proved as a practical solution for non-intrusive driver behavior detection. Besides, this study also shows that transfer learning can successfully transfer the domain knowledge that learned from the largescale dataset to the small-scale driver behavior recognition task.
Finally, an unsupervised GMM-based segmentation method is applied to process the raw images and extract the driver body region from the background. It is found that by applying a segmentation model prior to the behavior detection network, the detection accuracy on the driving activities recognition can increase significantly.

D. Paper Organization
The remainder of this paper is organized as follows. Section II introduces the experiment setup and data collection. Section III proposes the deep convolutional neural network models and transfer learning schemes for driving tasks recognition. Then, the tasks recognition results and model evaluation are performed in Section IV. Section V presents the discussions and future works. Finally, this paper is concluded in Section VI.

II. EXPERIMENT AND DATA COLLECTION
This section describes the experimental design and data processing for driver behavior recognition. Fig. 1. illustrates the general system architecture. First, raw RGB images are collected using the Kinect camera. Then, the cropped images are segmented using the GMM algorithm. Finally, the CNN model is adopted for the activities recognition task. Specifically, driver behavior images are collected with a Kinect camera. The Kinect enables the collection of multi-modal signals, such as the color image, depth image, and audio signals. It was initially designed for indoor human-computer-interaction and has been successfully used for driver monitoring systems [28] [29]. As mentioned in [24], the drivers' head poses, and upper body joints also can be detected using the Kinect. While in this study, only the RGB images are used.
According to the Kinect application requirements [?], it was mounted in the middle of the front window, facing the upper body of the driver so that not to interfere the drivers field of view while driving. The device setup is shown in Fig.  2. The sampling rate for the image collection is 25 frames per seconds. According to the study in [19], short-term driver behaviors like mirror checking can last from 0.5 to 1 second. Therefore, the sampling rate is fast enough to capture these behaviors. The data are recorded with an Intel Core i7 2.5GHz CPU, and the codes are written in C++ based on the Windows Kinect SDK and OpenCV. To store the images, the raw images are compressed to 640ˆ360ˆ3 format to increase the computation efficiency.
Ten drivers are involved in the experiment. They were asked to perform seven activities, which consist of four normal driving tasks (normal driving, left mirror checking, right mirror checking, and rear mirror checking) and three secondary tasks (using in-vehicle radio/video device, answering mobile phone,   and texting). It took about 20 to 30 minutes for each driver to finish all these tasks, and about 34 thousands images were captured in total. In this experiment, five drivers were asked to perform these tasks during driving in a testing field, while the rest five drivers were asked to mimic the driving tasks and not drive the vehicle. This is because, during normal driving, it is dangerous to perform the secondary tasks so that secondary data are limited. However, the steady scenario can be used to collect enough secondary behavior data safely. The number of images for each task and the quantitative comparison between normal driving and secondary tasks is shown in Fig.  3. Unlike some human activity recognition studies that require temporal information, in this study, no temporal information is considered, and each image is processed individually. The reason is that during driving the driver outer behaviors are always explicit, such as mirrors checking and performing secondary tasks. Therefore, no temporal information is required for inferring driver behaviors since most of the images carry enough information for activity recognition.

III. METHODOLOGIES
This section describes the algorithm framework that is used in this study. Specifically, Section III.A introduces the image pre-processing and segmentation based on the GMM algorithm. Section III.B describes the three deep CNN frameworks as well as the transfer learning scheme.

A. Image Pre-processing and Segmentation
The original images are stored in the format of 640ˆ360ˆ3. The raw images are cropped to speed up the CNN training process and increase the classification accuracy. An interest of region (ROI) which mainly contains the driver body region is selected. The left part of Fig. 1. indicates the raw image and the selected ROI. After the raw images are cropped, these images are transformed into the size of 227ˆ227ˆ3 to satisfy the input requirement of the AlexNet and 224ˆ224ˆ3 for the GoogLeNet and ResNet, respectively.
Then, the GMM algorithm is applied to segment the images and extract the driver body region from the background. GMM is an unsupervised machine learning method, which can be used for data clustering and data mining. It is a probability density function that is represented by a weighted sum of sub-Gaussian components [30]. One of the advantages of using GMM to unsupervised segment the images is it requires no manual labeling and can be flexible to modify the model by adjusting the cluster centers [32]. To train a GMM-based segmentation model, each image is represented by a feature vector according to the pixel intensity. The feature vector for the GMM is a three-dimensional vector that contains the RGB intensity of each pixel. Fig. 4. illustrates the segmented images of the ten drivers for model training and testing. Driver head and body region can be identified with the GMM segmentation method. Since the camera is fixed inside the vehicle cabin, the drivers seat position and the corresponding head position will be fixed within a certain area. The driver body region can be determined based on a set of pre-defined points which are located around the drivers head position. The points around the head position and the corresponding label will be used to indicate the driver regions. In the future, the manual selection method can be replaced by using an automatic detection method. For example, a precise driver head position can be first detected using the head detection algorithms and then the driver body regions can be determined directly or using a simple semantic segmentation network. As shown in the next section, the segmentation-based method can dramatically increase the model recognition accuracy.

B. Model Preparation and Transfer Learning
1) AlexNet Model: Currently, deep convolutional neural networks have gained a tremendous improvement in the domain of computer vision. One of the key reasons is the distribution of ImageNet dataset [33]. ImageNet is a large-scale dataset, which contains more than 15 million high-resolution annotated natural images of over 22,000 categories. A large number of annotated images benefit the training of deeper and more accurate CNN models. In this work, three deep convolutional neural network models, namely, AlexNet, GoogLeNet, and ResNet50 are chosen as the basic model structures for the recognition of driver behavior. The AlexNet was first proposed by Alex Krizhevsky in 2012 [34], who won the ImageNet Large Scale Visual Recognition Challenge (ILSVRC12). The model was trained for the classification of 1000 categories in the ImageNet dataset. There are five convolutional layers and three fully connected neural network layers with non-linearity and pooling layers between the convolutional layers. In total, AlexNet contains 60 million parameters and 650,000 neurons. An simplified model structure for AlexNet is shown in Fig. 1.
2) GoogLeNet Model: GoogLeNet is another deep CNN model, which won the ILSVRC14 [41]. GoogleNet is significantly deeper than the AlexNet, and it achieved more accurate classification results on the ImageNet dataset. Despite the model depth, the main contribution of GoogLeNet is the utilization of Inception architecture. As shown in [41], the most common ways of increasing CNN model performance are to improve the network size (either the depth or the width of the model). However, it gives rise to the requirement for larger scale dataset and more computational burden. Based on this, the Inception layers was introduced into the CNN model to increase the sparsity among the layers, and reduce the number of parameters. Each Inception layer consists of six basic convolution filters and one max pooling filter. With different scales, the parallel-arranged convolutional filters will have more accurate detailing and a broader representation for the information from previous layers. A typical dimension reduction Inception layer is shown in Fig. 5. In total, there are two traditional convolutional layers at the lower level of the GoogLeNet and nine Inception layers are concatenated at higher levels. With the application of Inception layers, the general quantity of the parameters in GoogLeNet is 12 times less than that in the AlexNet.
3) ResNet Model: Recent evidence has shown that the network depth is of importance to the feature representation and generalization [45]. It is common to see that simply stack the convolutional layers to increase the depth of the model cannot give a better training and generalization performance [46]. Accordingly, in [44], Kaming, et al. introduced a novel deep CNN model, namely, Residual Networks (ResNet) to enable the construction of deeper convolutional neural networks. By introducing the residual learning scheme, the ResNet achieved the first place on the ILSVRC 2015 classification competition and won the ImageNet detection, ImageNet localization, CO-CO detection 2015, and COCO segmentation.
As shown in the left part of Fig. 6 function for the basic residual block can be assumed as H(x). The x represents the inputs to the first layer. The residual network supposes an explicit residual mapping function F(x) exists such that F(x)=H(x)-x, and the original mapping can be represented as F(x)+x. The core idea behind the residual network block is that although both H(x) and the F(x)+x mapping is able to approximate the desired functions asymptotically, it is much easier to learn the mapping of F(x)+x. The added layers through the shortcut connection are the identity mapping. The right graph in Fig. 6 represents the full structure of a deep residual network, where residual learning is performed for every few stacked layers. By introducing the identity mapping and copying the other layers from the shallower model, the deep residual network can efficiently solve the model degradation problem when the models getting deeper [44].

4) Transfer Learning:
A large-scale annotated dataset like ImageNet is needed to train the deep convolutional neural networks like AlexNet, GoogLeNet, and ResNet from scratch. However, in general, large-scale annotated datasets are not always available for specific tasks. Therefore, the common ways to use the pre-trained deep CNN model are either treating the model as a fixed feature extractor without tuning the model parameters or fine-tune the pre-trained model parameters with a small-scale dataset. In this study, the CNN models will be used in the second manner, which is to fine tune the last few layers of the models with the driver behavior dataset. Since the original models are trained to classify the 1000 categories, the last few layers have to be modified so that the models can satisfy the seven objects or the binary classification task. Specifically, the original last fully connected layer and the output layer, which generate the probabilities for the 1000 categories, are replaced by a new fully-connected layer and softmax layer that output the probabilities for the seven categories.
The basic structure and properties of the convolutional layers is remained so that these layers can keep their advantages in the feature extraction and representation. Meanwhile, the knowledge that learned from the large-scale ImageNet dataset can be transferred to the driver behavior domain. A small initial learning rate is selected to slow down the updating rate of the convolutional layers. On the contrary, a much larger learning rate for the last fully connected (FC) layer is chosen to speed up the learning rate in the final layers. In this study, the convolutional layers are not frozen as we found that the performance will decrease when totally freezing the convolutional layers. Therefore, a small updating rate was chosen so that the convolutional layer will try to adapt to the new classification task. With this kind of combination, the new models can be trained to solve the new classification tasks.

IV. EXPERIMENT RESULTS AND ANALYSIS
In this section, the analysis for the driving activities classification are proposed. The system performances are evaluated from four major aspects: the impact image segmentation on multi-behaviors recognition, deep CNN model visualisation, the binary classification results on the distracted behavior detection, and the performance comparison with other methods.

A. Evaluation of CNN models on the Multiple Driving Behaviors Recognition
In this section, the activities recognition results for the ten participants are evaluated. The seven driving-related tasks are ordered as normal driving, right mirror checking, rear mirror checking, left mirror checking, using radio/video device, texting, and answering mobile phone. Table1, Table 2, and  Table 3 illustrate the classification results of the seven tasks based on AlexNet, GoogLeNet, and ResNet, respectively. T1 to T7 represents the seven tasks and D1 to D10 indicates the ten different drivers. The models are trained with MATLAB Deep Learning toolbox and evaluated using the leave-one-out (LOO) cross-validation method. To get the activity identification results for each driver, the images from one driver are used as testing images, whereas the rest images of the nine drivers are used for training. Therefore, for each driver, the data are completely new to the CNN models and the identification performances equal the model generalization on this new dataset.
As shown in Table 1, the general identification accuracy for the segmentation-based AlexNet achieved an average of 81.4% accuracy. The raw-image based AlexNet was also tested, which achieved only 69.2% recognition accuracy. In Table 1, the average performance in the rightmost column is defined as the average detection results for each driver, while  T1  T2  T3  T4  T5  T6  T7  Ave   D1 T1  T2  T3  T4  T5  T6 T7 Ave the mean accuracy in the bottom row represents the average detection rate for each task. Regarding the detection accuracy for each task, the answering mobile phone activity gets the most accurate detection results among the ten drivers for all three models. The worst result happens in the rear mirror checking (T3) case for the three models. One explanation is that the rear mirror checking behavior require few body and head movement, which can be easily misclassified into the normal driving task. Another evidence that can be drawn from Table 1, Table 2, and Table 3 are the CNN model achieved better detection results on the secondary tasks in general. This is mainly because when performing the secondary tasks, the driver has to move his/her body and hands instead of only rotating her/his head, which is more distinct and easier to be detected. Table 2 indicates the activity classification results given by the GoogLeNet. The general detection results is similar to the results in Table 1 except that the overall detection accuracy for the ten drivers are slightly lower. The GoogLeNet does not achieve better classification results than the AlexNet as it does on the ImageNet dataset. However, the classification results for the GoogLeNet trained with raw images are better than that in the AlexNet case. The general classification results for the GoogLeNet with the raw image is 74.7% accuracy, which is 5% higher than that for the AlexNet. Table 3 illustrates the activity classification results given by the ResNet. Same to the GoogLeNet, the ResNet does not show its advantage on the activity classification task. Instead, the precision is the lowest among these three models. The general classification accuracy is 74.9% for the GMM-ResNet and 61.4% for the Raw imagebased ResNet. Discussions on the results will be proposed in the next section. Fig. 7. illustrates the confusion matrix for the ten drivers using the AlexNet model with GMM segmentation. The green diagonal shows the correct detection cases for the class. The bottom row shows the classification accuracy with respect to the target class, while the rightmost column shows the classification accuracy with respect to the predicted labels. As shown in Fig. 7, all of the driving tasks except the third task (rear mirror checking) achieved reasonable detection rates. There are 353 cases of rear-mirror checking are misclassified into normal driving and 747 cases are misclassified into the left mirror checking.

B. Visualisation of Deep CNN models
To further understanding of how the model responses to the segmented images, the activation map and feature visualization for the CNN models are analyzed. Fig. 8. shows the raw image and the segmented image of in-vehicle texting behavior for one participants. The image pair will be used to generate the model activation map. Fig. 10. shows the activation maps of AlexNet and GoogLeNet with respect to the images shown in Fig. 8. As  Fig. 10, the Relu5 layer for the G-AlexNet model remains much more features than that in the R-AlexNet. The segmentation-based method can extract the driver from the background more precisely so that the CNN model can maintain more relevant features of the driver. The strongest activation of the G-AlexNet model keeps the driver head rotation and other channels can store the arm position information as well.
Since the GoogLeNet is much deeper than the AlexNet, only the activation map of the second convolutional layer is analyzed. The right part of Fig. 10. indicates the activation of the conv2-ReLu layer in the GoogLeNet. As shown in the figure, the G-GoogLeNet trained with segmented images carries more driver related features instead of background features. As the driver-related features are not well maintained in the beginning layers, the deeper Inception layers of the R-GoogeLeNet also cannot learn very representative features for the driver. Based on this, the activation maps explain why the segmentation-based CNN models lead to a better classification result than the raw image-based models.
Similar results can be found in the ResNet case which is shown in Fig. 10. The upper images show the activation maps for the G-ResNet and R-ResNet, respectively. The lower part indicates the corresponding strongest activation map. Specifically, the green box in the top row images is the strongest activation channel of the final ReLu layer of the G-ResNet, and its corresponding channel in the R-ResNet map. The red boxes in the top images are the strongest activation channel of the R-ResNet, and its corresponding channel in the G-ResNet. As can be seen in the lower part of Fig. 10, the strongest activation channel of the G-ResNet is able to capture the head rotation and position features, while the R-ResNet fails to learn a precise representation for this behavior.  Finally, the time cost for model training and testing are compared in Table 4. As shown in Table 4, the general training for the GoogLeNet is two times longer than the AlexNet, and it takes about one hour to train the ResNet on the local computational device. It takes about 12ms to process one image for the AlexNet while the testing time for the GoogLeNet and ResNet are 45ms and 140ms, respectively. The model training and testing are implemented on an Intel Core i7 2.5GHz CPU and NVIDIA MX150 2GB GPU.

C. Driver Distraction Detection using Binary Classifier
In this section, the three CNN models are modified and trained to detect whether the driver is distracted or not. In this case, the first four tasks are grouped together, while the last three tasks constitute another group. The CNN models are fine-tuned to solve the binary classification problem. The distraction detection results for the AlexNet, GoogLeNet, and ResNet are shown in Table 5. As shown in Table 5, the segmentation image based AlexNet leads to the most accurate results. The general classification accuracy for the G-AlexNet based model is 91.4%. The general classification accuracy for the GoogLeNet and ResNet methods are 87.5% and 83.0%, which are slightly lower than the results given by the AlexNet. It should be noticed that there are no smoothing algorithms applied to the distraction warning module. In realworld situations, the driver assistance system will only warn the driver if the distraction happens continuously in a short period. Therefore, if applying a short period smoothing or voting techniques, the distraction detection system can be more suitable for the real-world application.

D. Comparison with Other Methods
To further evaluate our method, additional experiments are made to compare the proposed method with conventional hand-craft feature extraction and shallow CNN methods. Specifically, the approaches used for comparison include: FC7+ANN: The method proposed in [35], which extracts the posture features with a pre-trained AlexNet CNN model. In this part, the activation of 'fc7' layer of AlexNet is extracted, and an FFNN ANN model with 300 neurons is constructed based on the feature set. The dimension for each of the 'fc7' feature vector is 4096.
PHOG+SVM: The pyramid histogram of oriented gradients (PHOG) followed by support vector machine (SVM) method. A pyramid HOG feature extractor, which concatenates two different scale HOG extractors is used. The block size for the HOG feature extractors is 22, and the cell sizes are 88 and 1616, respectively. The dimension for each of the PHOG feature vector is 32328.
OP+ANN: The method proposed in [52] [53], which recognize the motion with optical flow. Specifically, the optical flow of the video sequence is extracted with the Lucas-Kanade method, and a 51529-dimensional feature vector is  concatenated for each image. Then, another FFNN model with the same structure of the one that proposed in [24] is used.
OPsCNN: Based on the magnitude of the optical flow, a shallow multi-class CNN is proposed. Three convolution layers are used following with three fully connected layers. The input images are rescaled into the size of 120ˆ120. The filter size is selected as 5ˆ5 for the first two convolutional layers, and 3ˆ3 for the third convolutional layer. Batch normalization, non-overlap pooling, and ReLu non-linearity layers are applied between the convolutional layers. The number of neurons for the three FC layers are 512, 128, and 7, which is similar to the architecture used in [51].
GMMsCNN: The shallow CNN with the GMM segmented color images are also tested. Finally, as the dimensions of the feature vectors given by different algorithms are too high, a principal component analysis (PCA) algorithm is used to reduce the feature dimension and reduce the training cost for the first three models. The dimension for each feature vector is reduced to 500. The model comparisons are illustrated in Table  6. As shown in Table 6, the PHOV and optical flow features are unable to accurately represent the driving tasks and far less precise than the transfer learning method. The recognition results of the optical flow-based and shallow CNN-based methods are slightly better than the feature extraction methods. The high-level features from the FC7 layer of AlexNet with FFNN gives better results than the rest four methods. However, the average results for the ten drivers are still significantly lower compared with the proposed method.
In Table 7, the proposed method is also compared with relevant studies in the literature. It should be noticed that difficulties exist in making a precise cross-platform comparison between the existing studies since different algorithms, platforms, and experimental methods were used. Based on Table 7, some researchers have tried to analysis the driver distracted behaviors with either real vehicle and simulated data. For example, in [19], Li, et al. proposed a machine learning framework for the detection of driver mirror checking behaviors and secondary tasks. The general framework follows a standard machine learning application procedure, which consists of feature extraction, model training, and testing. The detection rates for secondary tasks like radio operating and phone-talking are around 75% to 80%. We believe this should attribute to the absent of driver body features. In [24], the driver's behaviors are detected with a Kinect device, which enables the analysis of both driver's head and upper body features. However, that work also heavily rely on the complex feature extraction and analysis, which is time-consuming and requires extra hardware for calibration. Similar work can be found in [51], where the authors evaluated the performance of different types of CNN models on ten different driving activities. Although high detection accuracies are achieved in the study, the data are collected on the driving simulator and did not stand for the real-world in-vehicle performance.
Another reason that can significantly influence the model accuracy is the evaluation method. In Table 7, the Loo method is more strict than the cross validation method as it indicates the model generalization capability on the unseen dataset. If we use the cross-validation method and simply separate the data into training and testing group, the GMM-AlexNet in this study can achieve 98.9% accuracy for multi-tasks detection.
However, we still suggest using the Loo method as it can reflect the performance variance on different subjects. Based on the comparison with existing studies, the proposed method in this study show three advantages. First, a naturalistic in-vehicle dataset is collected for the fine-tuning and validation of the deep CNN models. The fine-tune method is very efficient in real-world application as it is hard to collect large scale annotated driver distraction data. Although some studies use side view images as the images show clear driver body features [48]- [51], the side view method is less efficient and robust compared with the front view method in the real vehicle as the side view can be occluded by the passengers. Second, the leave-one-out model evaluation is used so that the results illustrate an independent performance on the different drivers. Third, the segmentation-based CNN models achieved stateof-art detection accuracy on distracted behaviors such as the phone answering (93.2%) and texting (94.5%) with naturalistic data.

A. Transfer Learning Performance
With the analysis of different deep CNN models, it can be found that deeper CNN model like ResNet50 does not contribute to a higher detection accuracy as it did in the ILSVRC competition. The reasons can be multifold, and we try to explain this phenomenon merely based on the evidence in this study. First, as the GoogLeNet and ResNet are deeper than the AlexNet, the model may need more data to be optimized. Second, the transfer learning approach is different from training from scratch such as using the ImageNet dataset. The fine-tune transfer learning method mainly focus on the tuning of a few layers while keeping the main characters of the convolutional layers. However, as the model getting deeper, the domain knowledge learned from the much larger dataset may not very suitable for the smaller dataset. This conclusion is only made according of the results in this study and further evaluation is expected. The primary object in this study is not to evaluate the classification performance of different CNN models. However, this study aims to provide an efficient endto-end approach to understand driver behaviors. Therefore, more experiment and analysis are expected in the future to obtain a more precise explanation.
Although it is essential to understand which features are more critical to the driver behavior recognition, the traditional machine learning framework is less efficient and usually has a specific requirement on the system hardware such as head pose measurement and skeleton tracking. Therefore, in this work, an end-to-end deep learning approach is designed to solve the driver behavior recognition task. The only hardware required for the proposed system is an RGB camera. Meanwhile, as shown in this study, the CNN models can automatically capture the head and body features. The deep learning method achieved competitive detection results compared with the methods that rely on the head and body detection [24] [47]. However, the end-to-end process shows its advantages in realworld detection since no complex head pose and body joint estimation algorithms are needed. Besides, this study also evaluated the binary classification results and found that the deep learning approach can provide an accurate estimation of the distraction status. This approach can be easily integrated into most of the current ADAS products dues to its efficient and low-cost properties.

B. Real-time Application
In this experiment, the system is implemented to a Windows operating system using MATLAB platform and a single lowcost GPU device. The testing cost of the AlexNet for each image is about 13ms, also, it cost 50ms for the GMM to segment each image. The total computational cost for each image is around 60-70ms, and the general processing ability of the system is about 14fps. Therefore, the proposed system can satisfy the real-world computational requirement. Meanwhile, regarding the in-vehicle embedded systems in the real-world, the Linux platform along with C++/Python programming usually can be more efficient than the MATLAB environment. Also, since more powerful embedded GPU devices has been published, the in-vehicle graphics processor can provide more powerful parallel computation than the current platform. Hence, the algorithm has no significant limitation in the realworld application.
Next, a sliding-window is applied to the detection to smooth the result. The current driver state is selected according to the majority state within the sliding window. The smoothing results of the secondary tasks detection and distraction detection are shown in Fig.12 and Fig.13. The upper images of Fig.12   and Fig.13 indicate the comparison between the ground truth label and the predicted values concerning the seven driving activities detection and distraction detection, while the bottom images represent the comparison between the ground truth label and the smoothed version of the predicted values. The sliding window can be used to smooth the result, and eliminate some false detection cases. However, it should be noticed that as the sliding-window uses the voting scheme, detection delay can happen for the secondary tasks and the horizon of the sliding-window will control how much delay the detection system has. A larger horizon of the sliding-window will lead to a smoother result; however, it will also cause a larger detection delay. In Fig.12 and Fig.13, the window is selected as seven samples, which can cause a 500ms delay. Considering each task, especially the secondary tasks can last several seconds, this 0.5s late detection is normally acceptable.

VI. CONCLUSION
In this work, a driving-related activity recognition system based on the deep CNN model and transfer learning method is proposed. To increase the identification accuracy, the raw RGB images are first processed with a GMM-based segmentation algorithm, which can efficiently remove the irrelevant objects and identify the driver position from the background context. The classification results indicate that the segmentation contributes to a much more precise detection result than the model trained with the raw images. Another comparison is made between the transfer learning and other feature extraction methods. Finally, if using the CNN models as a binary classifier, the driver distraction detection rate can achieve 91% accuracy. In the future, the data will be further analyzed, and the model will be updated to increase the system robustness and detection accuracy. Meanwhile, the system will be tested and used for driver/passenger behavior analysis on the partially automated vehicles in the real world.