{'fraction_non_alphanumeric': 0.026824666902487915, 'fraction_numerical': 0.008607475533545572, 'mean_word_length': 5.243283032756717, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
EMOTION RECOGNITION USING AUTOENCODERS AND CONVOLUTIONAL NEURAL NETWORKS

Emotions demonstrate people's reactions to certain stimuli. Facial expression analysis is often used to identify the emotion expressed. Machine learning algorithms combined with artificial intelligence techniques have been developed in order to detect expressions found in multimedia elements, including videos and pictures. Advanced methods to achieve this include the usage of Deep Learning algorithms. The aim of this paper is to analyze the performance of a Convolutional Neural Network which uses AutoEncoder Units for emotion-recognition in human faces. The combination of two Deep Learning techniques boosts the performance of the classification system. 8000 facial expressions from the Radboud Faces Database were used during this research for both training and testing. The outcome showed that five of the eight analyzed emotions presented higher accuracy rates, higher than 90%.


Introduction
Facial emotion detection is the process of identifying the feeling that a person is expressing at a particular moment. Potential applications of emotion recognition include the improvement of student engagement [1], the design of smart health environments [2], the analysis of customers' feedback [3], and the evaluation of quality in children's games [4], among others. Face recognition within multimedia elements, including images and videos, has been identified as one of the current challenges pursued by artificial intelligence. Several powerful techniques from the Deep Learning field have been recently implemented seeking an improvement in emotion detection, such as Convolutional Neural Networks (CNN) [5], Deep Belief Networks (DBN) [6], and Auto Encoders [7]. Deep Learning is a novel research area in the machine learning territory which focuses on the learning of high-level data representations and abstractions, namely images, sounds, and text by using hierarchical architectures, being neural networks, convolution networks, belief networks, and recurrent neural networks the most well-known in several artificial intelligence areas, for instance image classification [8], speech recognition [9], handwriting recognition [10], computer vision [11], and natural language processing [12].
Identifying the sentiment expressed by a person is one of the side objectives achieved by face detection. Recent research [13] has proven that emotion recognition can be accomplished by implementing machine learning and artificial intelligence algorithms. To assist in this task, several open-source libraries and packages, being OpenCV [14], TensorFlow [15], Theano [16], Caffe [17] and the Microsoft Cognitive Toolkit (CNTK) [18] the most notorious examples, cut down the process of building deep-learning-based algorithms and applications. Emotions including anger, disgust, happiness, surprise, and neutrality can be distinguished.
The aim of this paper is to analyze the performance of a Convolutional Neural Network which uses AutoEncoder Units for emotion-recognition in human faces. The combination of two Deep Learning techniques boosts the performance of the recognition system despite of the complexity introduced by both algorithms. 8000 facial expressions from the Radboud Faces Database were examined in different phases of the experiments for training and evaluation purposes.
This paper is organized as follows. Background information introducing Emotion Recognition, Convolutional Neural Networks, AutoEncoders, and the Radboud Faces Database is presented as part of the theoretical background section. Afterwards, the problem solution is described by explaining the methods and methodology that were used for this comparison. Evaluation results are shown subsequently. Finally, conclusions are presented in the final section.

Emotion Recognition
Emotions are a strong representation of feelings about people's situations and relationships with other people. The most basic process a human has to reflect how they feel is by using facial expressions. Speech, gestures, and behavior are also used to describe a person's current state. Emotion recognition can be defined as the process of detecting the feeling expressed by a human being. Basic emotions include anger, happiness, sadness, fear, surprise, and disgust. It has been demonstrated that humans are able to identify facial emotions even since early ages [19]. Following machine learning's objective of imitating human thinking, algorithms have been developed for this purpose. Emotions play a key role in decision-making and human-behavior, as many actions are determined by how a person feels at some point.
Typically, these algorithms consider either a picture or a video (which can be considered as a set of images) as input, then they proceed to detect and focus their attention on a face and finally, specific points and regions of the face are analyzed in order to detect the affective state. Machine Learning algorithms, methods and techniques can be applied to detect emotions from a picture or video. For instance, a deep learning neural network can perform effective human activity recognition with the aid of smartphone sensors [20].

Convolutional Neural Networks
A convolutional neural network (CNN) is a class of deep, feed-forward artificial neural networks that has been applied in visual images analysis. CNNs were inspired by biological processes in which the connectivity pattern between neurons is influenced by the organization of the animal visual cortex. Individual neurons respond to stimuli only in a restricted region of the receptive field. Receptive fields of different neurons partially overlap to top the entire visual field. CNNs use relatively little pre-processing compared to other image classification algorithms, meaning that the network learns the filters which in traditional algorithms were hand-engineered. This independence from prior knowledge and human effort in feature design is a major advantage. They have been applied for image and video recognition, recommender systems [21] and natural language processing [22].
A CNN contains three main layers: convolutional layers, pooling layers and fully-connected layers, each one with their own purpose. In the convolutional layers, several kernels convolve, i.e. entwine, an image and intermediate feature maps, creating new feature maps, as shown in Fig. 1. Benefits of convolution include a minimal number of parameters, correlation learning between neighboring pixels due to local connectivity, an unchanged object location, and a faster learning process than for fully-connected layers. Secondly, a pooling layer usually follows a convolutional layer in order to reduce feature maps dimensions. This layer is translation invariant as well because the computation considers neighboring pixels by using strategies such as average pooling or max pooling. As an example, an 8x8 feature map is reduced to a 4x4-dimensional output with a max pooling operator of 2x2 size and 2 for the stride value. Finally, a fully connected layer is similar to a traditional neural network and uses about 90% of the parameters in a CNN. It makes possible to feed forward the neural network into a vector with predefined length and can be used for image classification or follow-up processing.

AutoEncoders
AutoEncoders are feedforward neural networks where both the input and output are the same; they can be used to reconstruct their own input in a lower dimensional space. By assuming that one image is sent to this type of network, random representations in its center-most hidden layer are generated at first. By feeding the network with more and more similar images, the network will be benefited, developing a unique construct which contains the elements of the subject's face encoded in it. Leveraging that intuition, the concept is that an autoencoder network is capable of learning a specific emotion shape for different classes in the training set. In other words, by feeding the autoencoder network with different images of people smiling, the network would be able to learn that the feature to encode is the emotional distinctiveness of happiness.

Radboud Faces Database
The Radboud Faces Database (RaFD) is a high-quality set of pictures of 67 models (between male, female, Caucasian, Moroccan, children and adults) displaying 8 emotional expressions (anger, disgust, fear, happiness, sadness, surprise, contempt, and neutrality), as displayed in Fig. 2. The database contains 28,709 faces. This initiative by the Behavioural Science Institute of the Radboud University Nijmegen is available for research purposes upon request. Each emotion was shown looking to three directions (left, front, and right), with five camera angles. However, frontal images were selected only for this research.

Methods And Methodology
This research focuses on the development of a combination of two deep learning techniques, an AutoEncoder used as an input for the Convolutional Neural Network for emotion detection in human faces.
Convolutional autoencoders are a special variant of CNNs which encode their input using convolution into a compact internal representation which is then decoded using de-convolution into the original space, trying to minimize the reconstruction error. Such encoder/decoder pairs can be optionally stacked, as described in [24], to increase the complexity of the internal representations. Past halfway into the first decade of the 21st century [25], research demonstrated that autoencoders could be used in neural networks pre-training and overcome the deep neural networks training obstacles, including: the differences between magnitudes of gradients in the lower and higher layers., the landscape of the objective function is difficult for stochastic gradient descent to find a good local optimum, and the fact that deep networks have various parameters, which can easily recall training data, thus complicating the generalization process.
Pre-training process of a deep network is divided into two steps: Pretraining and Fine-tuning. Pretraining consists of training a sequence of autoencoders, greedily one layer at a time, using unsupervised data. For Fine-tuning, first the last layer is trained by using supervised data. After that, backpropagation is used to fine-tune the entire network using supervised data.
Researchers have demonstrated that this pretraining idea improves deep neural networks as pretraining is done one layer at a time, meaning it does not suffer from the difficulty of full supervised learning. This approach was widely accepted as it resembles the brain's likeliness for unsupervised learning, which is also more appealing since it does not make use of expensive unlabeled data.
Most of the research in emotion detection focuses on the usage of a deep learning technique alone [26,27,28,29]. The experiment consists of two parts. First, an autoencoder network with one hidden layer containing 300 and 500 nodes was developed, with Rectified Linear Unit (ReLU) designed as the activation function, which accelerates the convergence of gradient descents faster than a sigmoid function. To overcome the ReLU problem, referred as "dead neurons", i.e., neurons which are never activated across the dataset, a -0.01 threshold slope was considered. The AutoEncoder network was trained on the RaFD database, as previously mentioned, with 8000 images, all of which were normalized to 48x48 dimensions and grayscale colors, while categorizing each of them per emotion depicted on. The implementation of the AutoEncoder was done using the Tensorflow deep learning framework with Python code.
Afterwards, the generated images were used as input for an 8-layer designed CNN which included three convolutional layers, three pooling layers, and two fully connected layers. For each of the convolutional layers, a 5x5 filter size was used, except for the third convolutional network, which considered a 3x3 filter. The stride value was set to 1, while the filter count was stablished at 512. Between the first two convolutional layers, an average pooling was considered in order to reduce complexity and maximize the low-level features extraction, such as edges. Between the last two convolutional layers, a max pooling over a 3x3 window was considered in order to extract specific features inside the face, such as the mouth, the eyes, and the sentiment in general. Accordingly, the Tensorflow framework was used to implement the CNN in Python.
The Fold cross-validation technique was used during the experiment to get a training set and 10 sets for testing. For each emotion, 10 folds were generated, each one consisting of 240 images: 120 elements included the evaluated sentiment while the rest incorporated the other seven emotions (20 images per sentiment were included in order to balance the distribution between all classes).
For each emotion, we generated a confusion matrix for each fold from the analysis. Examples of results sets are shown in Tables 1 and 2 for Neutral and Contempt emotions, respectively. Furthermore, Tables 3-10 present the statistical measures obtained from the performance of our classification model when analyzing each emotion. The following metrics are exhibited in these tables: Accuracy (Acc), Sensitivity (Sens), Specificity (Spec), Precision (Prec), Negative Predictive Value (NPV), and F1 Score. We considered these calculations as the most significative ones during our analysis.

Discussion
In order to get a detailed overview of each emotion, a confusion matrix was generated for each sentiment. Furthermore, Accuracy, Sensitivity, Specificity, Precision, the Negative Predictive Value rate, and the F1 score were the statistical measures of the performance of our model selected for this analysis and included in the tables. As each test is binary, i.e. the predicted sentiment belongs or not to a specific class, a more specific analysis based on the false positive, false negatives true positive, and true negative rates was allowed rather than a simple accuracy report. We detected during he analysis that five of the eight emotions were greatly benefited due to the usage of the proposed combination of deep learning techniques. The surprise, anger, neutral, happiness, and sadness sentiments present sensitivity and specificity values greater than 90%, meaning that the model is able to identify these feelings with a considerable level of trust and also report those emotions who do not belong to these classes. Since accuracy involves both specificity and sensitivity in its calculations, the accuracy for these five emotions are close to 90% as well. In a previous research [30], our findings were no higher than 80% for all emotions accuracy. Another improvement was detected for the disgust feeling, which presents an approximate 85% value in both accuracy and precision. However, contempt and fear feelings got 35% and 30% error rate values -the fraction of misclassified cases-respectively, which means the model is not perfect at all for these sentiments. As explained in [31], action units are taken into consideration in order to detect an emotion. An action unit is a specific characteristic of the face, for instance, the chin raise, the lip press, and the brow lower, among others. Contempt is an emotion with only one action unit: the dimple. However, it can be so small that even a person would be confused whether the expressed sentiment is neutrality or contempt. Likewise, fear and sadness share two action units: the brow lower and inner brow raise, making the distinction between any of the feelings not an easy task. An example of this complication is illustrated by Fig. 3. The following future work will be aimed to solve this. Finally, the F1-score was included as well in the results table as it is considered as the harmonic mean of both precision and accuracy and also is in agreement with our findings.

Conclusions
This paper deals with the results of a research about a proposed combination of two deep learning techniques for emotion detection in human faces. While work has been conducted by using a deep learning algorithm for both face detection and emotion recognition, few analyses implementing more than one technique have been performed. AutoEncoders and Convolutional Neural Networks were the selected models for this research. The achievements include an improvement from our previous work, for instance, 90% and higher accuracy rates for five sentiments were obtained. Several other statistical measures including sensitivity, specificity, and F1-score were also included in the analysis, with values in agreement to the accuracy obtained. Future research will include emotion detection in videos by using the proposed combined deep learning techniques, as well as a broad exploration of the contempt and fear emotions, both of which obtained a low accuracy rate in the current investigation.