{'fraction_non_alphanumeric': 0.03173040100661962, 'fraction_numerical': 0.011926254171453581, 'mean_word_length': 5.0111805327195, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Multimodal Physiological Signal Emotion Recognition Based on Convolutional Recurrent Neural Network

In order to solve the problem that the emotion recognition rate of single-mode physiological signals is not high in the physiological signals based emotion recognition, in this paper, we propose a convolutional recurrent neural network based method for multi-modal physiological signal emotion recognition task. The method used convolutional neural network to learn the spatial representations of multi-channel EEG signals and the Long Short-term Memory network to learn the temporal representations of peripheral physiological signals (EOG, EMG, GSR, RSP, BVP, and TMP). The two representations are combined for emotion recognition and classification. In the two emotion dimensions of Arousal and Valence, our experiments conducted on the open source dataset DEAP shows that, this method achieve 89.68% and 89.19% average accuracy in the EEG emotion classification, 63.06% and 62.41% average accuracy in the peripheral physiological signal emotion classification, 93.06% and 91.95% average accuracy in the combined feature emotion classification. The experimental results show that the convolutional recurrent neural network based method that we proposed efficiently extract multi-modal physiological signal feature to improve the emotion recognition performance.


Introduction
Emotion is a comprehensive state generated by whether objective things meet people's own needs. Different emotional states affect people's cognition, thinking, and decision-making. The recognition of different emotional states has a wide range of applications in distance education, medical treatment, intelligent systems, and human-computer interaction (HCI). In recent years, there has been an increasing interest in emotion recognition [1].
At present, the following approaches are mainly used to achieve emotion recognition: speech, facial expressions, comment, limb movements and physiological signals. However, speech, facial expressions, text, and body movements are non-physiological signals that are external manifestations of human emotions, except physiological signals. So it can be easily concealed by humans. Physiological signals, including electrocardiogram (ECG), my electricity (EMG), skin electricity (GSR), respiration (RSP), and electroencephalogram (EEG) produced through changes in the nervous system and endocrine system are not easily affected by human subjective consciousness control, so it can achieve higher emotion recognition objectivity and accuracy [2]. Previous research mostly used single-modal data for emotion recognition, such as, [3] [4] recognize emotions through facial expressions, and [5] [6] extract features from EEG for emotion recognition. [7] Has shown that certain emotional states because the same changes in individual physiological signals. For example, both "anger" and "surprise" cause heart rate to increase. The difference is that "anger" reduces heart rate variability and increases skin conductance, while "surprise" does not have significant changes. Therefore, it is possible to obtain more objective and accurate results by fusing the characteristics of multi-modal physiological signals. [8] used Hidden Markov Model (HMM) to extract physiological signal characteristics, and fused the three physiological signals of EEG, GSR, and heart rate (HR), proving that multiple physiological signals can form complementary advantages and improve emotion recognition effect. A lot of previous related researches have proved the effectiveness of traditional machine learning based emotion classification. However, the effect of machine learning methods on emotion classification is very dependent on the extraction of emotion-related features. Among them, the most commonly used features are Fourier transform (FT), power spectral density (PSD), and wavelet transform (WT). In recent years, the use of deep learning to extract abstract features has become a new trend. Deep learning not only reduce the workload of feature engineering but also improve recognition accuracy. With the rise of deep learning, convolutional neural networks have achieved good results in video image processing, but there are still rare application in feature extraction and classification of physiological signals. A number of studies have attempted to use Convolutional Neural Networks (CNN) to extract EEG emotional features directly, but did not achieve the desired classification effect. [9] Directly adopted CNN to recognize the electroencephalogram signals of motor imagination, and obtained only 45% classification accuracy. In [10], each electrode of the EEG and its nearest four electrode channels are combined into a new matrix. The accuracy of this method by directly using a convolutional neural network for feature extraction and classification has reached 70.11%. However, most CNN-based methods now rely heavily on complex preprocessing and feature engineering, or ignore the subtle spatial information contained in EEG signals. For example, in [11], the original EEG signal was converted into an image, and the ability of CNN to automatically learn features and share representations was not fully utilized.
This paper proposes a convolutional recurrent neural network based method for multi-modal physiological signal emotion recognition. Aiming at the problem that EEG emotion recognition relies on complex preprocessing and feature engineering, CNN is used to extract the subtle spatial features of EEG, and Long Short-term Memory (LSTM) is used to extract peripheral physiological signals (eye electricity, my electricity, skin electricity, breathing Rhythm, blood volume pulse, and body temperature) temporal representations, ready for emotional recognition. In order to solve the problem of emotional recognition limitation of single-mode physiological signals, the spatial characteristics of EEG and the temporal characteristics of peripheral physiological signals (PERI) are combined to form multi-modal physiological signal feature for emotional recognition, thereby achieving the purpose of improving emotion classification accuracy.

Convert One-dimensional EEG Signal to Two-dimensional EEG Matrix
The EEG-based BCI system uses an international 10-20 system to obtain EEG signals. Depending on recording channels number, different BCI system was used for obtaining EEG data. As shown in Fig.1, the circles in red indicate the test points used in the DEAP data set. In general, the original data collected by the EEG signal acquisition system at time t is a one-dimensional data vector , while is the data recorded by the nth electrode channel at time t. The EEG acquisition system has a total of n channels for reading EEG data, and n equals 32 in this paper. From the EEG electrode map, it is observed that each electrode is physically neighboring multiple electrodes which records the EEG signals in a certain area of brain, while the one-dimensional EEG vector sequence are restricted to two neighbors. Furthermore, different brain regions correspond to different EEG signals in different emotional states. In this paper, based on the spatial information of the electrode distribution of the acquisition system, one-dimensional EEG data sequences are converted into a two-dimensional EEG data matrix, as shown in Fig.1.

Figure 1. Conversion of one-dimensional EEG sequences into two-dimensional EEG matrices.
In this paper, we generalized the International 10-20 System with test electrodes used in the DEAP dataset to form a matrix (h × w), where h is the maximum point number of the vertical test points and w is the maximum point number of the horizontal test points. With the DEAP dataset, h equals w equals 9. The one-dimensional EEG data sequence at time t corresponds to the transformation function of the two-dimensional EEG data matrix , as shown in following eq.1: In , "0" indicates an unused electrode channel in the DEAP data set, which has no effect on the neural network. Through the transformation of the transformation function , the one-dimensional EEG data vector sequence is converted into a two-dimensional EEG data matrix sequence . During the duration , the dimension of the two-dimensional EEG data matrix is L+1. After performing the transformation function , the Z-score normalization is performed on non-zero elements for each EEG data matrix, as shown in eq.2:   Among them, x is non-zero elements in the EEG data matrix, μ denotes the mean of all non-zero elements, and σ denotes the standard deviation of all non-zero elements.
Finally, to divide the two-dimensional matrix sequence into multiple sets of two-dimensional matrix sequences a 1s sliding window is used. As shown in Fig.2, each matrix sequence has a fixed dimension and there is no overlap between two consecutive matrix sequences. The matrix sequence segment is expressed as shown in eq.3: Among eq.3, S is the matrix sequence dimension, and the subscript j is the matrix sequence segment number.

Using 2D-CNN to Extract the Spatial Features of the EEG Matrix
In order to extract the spatial features of each two-dimensional EEG matrix, this paper applies three consecutive 2D convolutional layers (2D-CNN), as shown in Fig.3. The j th input matrix sequence segment is defined as eq.4: There are S data matrices of size h×w. Each data matrix are input to 2D-CNN and parsed into a spatial feature vector , as shown in eq.5: Each layer of 2D-CNN has the same kernel size of 3×3 for the correlation between EEG channels extraction. The kernel of this size is widely applied in the field of computer vision. To prevent missing the information at the edge of the input data mesh we use zero-padding techniques in each convolutional operation. This method uses 64 feature maps for the first layer of convolution, and the second and third layers of convolution have 128 and 256 feature maps, respectively. In the classical CNN architecture of computer vision, the convolutional layer is often followed by a pooling layer to reduce the data dimension. But the size of the data matrix in this paper is much smaller than that in computer vision, in order to reduce the loss of data information, no pooling operation is performed in the CNN model of this paper. In order to speed up model convergence and prevent overfitting, a batch normalization layer (BN) is introduced after each layer of convolution operation. After performing three layers of 2D-CNN on each data matrix , in order to obtain the EEG spatial characteristics during the duration [t, t+S-1], each convolutional feature matrix are fused in depth. After that, the fused 9×9× (256×S) matrix sequence is convolved with 15 convolution kernels with size of 1×1 to obtain the matrix sequence size of 15×9×9, and it is expanded into a spatial feature vector to prepare for feature fusion.

Using LSTM to Extract the Temporal Features of Peripheral Physiological Signals
In the DEAP data set, the signals collected on the 33-40 channels are 2 EOG, 2 EMG, 1 GSR, 1 RSP, 1 BVP, and 1 TMP, then a one-dimensional vector composed of peripheral physiological signals at a time t can be expressed as eq.6: After segmenting the data by using a 1s sliding window, we can obtain j th inputted data segment of RNN, as shown in R j : Where S represents the sliding window size. As shown in Fig.4, the peripheral physiological signal data vector R j is input to a Recurrent Neural Network (RNN) to learn the temporal representations. In this paper, LSTM units are used to construct two stacked the RNN layer. Each layer of RNN has S LSTM units, and the input to the second RNN layer is the output of the previous RNN layer. The hidden state of the LSTM unit of the first RNN layer at current time step t is denoted as h t , and the h t-1 is the hidden state of the previous time step t-1. The hidden state from the previous time t-1 as the input of the current time t affects the output of the LSTM unit. The input sequence of the second layer LSTM is the hidden state sequence of the first layer LSTM. The output at the last time step t+S-1 is the temporal representations of the peripheral physiological signal, as shown in eq.8 and eq.9: Where v is the hidden state size of the LSTM unit. The fully connected (FC) layer before LSTM is used to enhance the representation of temporal information.

Feature Fusion of Multi-modal Physiological Signals
After the operations of feature extraction, the spatial representations of EEG and the temporal representations of PERI can be obtained. As shown in Fig.5, the extracted EEG spatial features are fused with the temporal representations of the PERI. Finally, the fused features are fed into the softmax layer to predict the emotional state, as shown in eq.10: Among them, denotes EEG spatial feature vector, is PERI temporal feature vector, and n represents emotion category. In this paper, n equals 2.  Valence (1-9); Arousal (1-9); Dominance (1-9); Liking (1-9)

DEAP Dataset
Signal 32-channels EEG; 8-channels PERI (2EOG; 2EMG;1GSR; 1RSP; 1BVP; 1TMP) DEAP is a multi-modal physiological signal dataset for analyzing human emotions [12]. DEAP stimulated the emotions of the experimental participants through a one-minute music video, which recorded the 32-channels EEG and 8-channels PERI corresponding to emotions in 40 different experiments of 32 experimental participants. The data were cut into segments of 63s. The first 3s are the pre-trial baseline. The original physiological signal in the data set was collected at a sampling rate of 512 Hz, and the final signal was down-sampled to 128 Hz. The EEG signal is band-pass filtered at 4.0-45.0 Hz, and the EOG (eye movement) artifacts are removed. The specific description is shown in Table  1.

Emotion Model
Emotions should be defined and represented quantitatively, for emotion recognition. Relevant scholars first proposed basic definitions of emotions decades ago, but precise definitions have not been widely accepted by psychologists. Scholars are more inclined to simulate emotions in the following two ways. One is to divide emotions into different basic categories. Ekman and Friesen consider emotions is discrete and propose six basic emotions: happiness, sadness, anger, fear, surprise, and disgust [13]. The other is to use multiple dimensions to express emotions, such as the emotion wheel of Plutchik [14] and the titer-wake scale of Russell [15]. The emotion model in this article is Russell's titer-wake scale, as shown in Fig.6.

Data Pre-processing
In order to improve the accuracy of emotion recognition, 60s experimental data minus pre-trial baseline data for the first 3s to remove the baseline interference. The 3s pre-trial baseline data is divided into 3 segments with a sliding window of 1s size, and then the 3s pre-trial baseline data is averaged. Finally, the 60s experimental data divided into 60 segments are subtracted from the pre-trial baseline data average, as shown in eq.11: (11) Among them, is the j th pre-processed data obtained by subtracting the pre-trial baseline from the raw EEG signals, is the j th 60s experimental raw EEG data, and is the i th data of the pre-trial baseline data.

Results and Analysis
This section mainly applies the method of this paper to the DEAP dataset for emotion recognition of multi-modal physiological signals. In this paper, we propose a convolutional recurrent neural network based method to extract the spatial features of EEG signals and the temporal representations of peripheral physiological signals, and then, both of the two features are integrated to recognize emotional states. Finally, the accuracy of emotion recognition is used to show the difference between multi-modal physiological signal emotion recognition and single-modal physiological signal recognition. Fig.7 describe the accuracy of emotion recognition that use multi-modal physiological signals, EEG, and peripheral physiological signals from 32 participants in the two emotion dimensions of Valence and Arousal. Among them, we recorded the accuracy of emotion recognition of multi-modal physiological signals, EEG signals and PERI in square-point curve, circular-point curve and triangular-point curve. In As shown in Fig.8, this paper compares related studies, which are based on DEAP datasets. Garcí a. Uses a Gaussian process hidden variable model (GP-LVM) to learn hidden space representation [16]. This model maps high-dimensional data in low-dimensional hidden space, and finally uses support vector machine (SVM) to identify emotions; Chen. proposed a novel multimodal feature fusion method, which uses Hidden Markov Model (HMM) and multimodal feature set to identify emotions [17]; Tang. developed the Bimodal-LSTM model to perform emotion recognition tasks using EEG signals and surrounding physiological signals, and obtained the latest results with an average accuracy rate of 83.53% [18]; Liu. Applied a Bimodal Deep Auto Encoder (BDAE) to classify emotions on EEG and eye movement signals, with an average accuracy rate of 82.85% [19].

Conclusion
In this paper, we propose a convolutional recurrent neural network based method for multi-modal physiological signal emotion recognition task. The method used CNN to learn the subtle spatial representations of multi-channel EEG signals and the LSTM to learn the temporal representations of PERI. The two features are combined for emotion recognition and classification. The method was tested on DEAP dataset. In the Arousal emotion dimension, the average emotion recognition accuracy is 93.06%; In the Valence emotion dimension, the average emotion recognition accuracy is 91.95%. The experimental results show that compared with single-modal EEG signal emotion recognition, the accuracy rate of multimodal physiological signal emotion recognition of the model is improved, which proves the reliability of the model.
In the process of multimodal physiological signal emotion recognition, the signals of all 32 channels EEG and 8 channels PERI were all selected. The experimental results show that the emotional recognition accuracy of 8 channels PERI is low. Choosing the appropriate EEG channels and peripheral physiological signals for emotion recognition requires further research.