{'fraction_non_alphanumeric': 0.030462542281288155, 'fraction_numerical': 0.013109701178872366, 'mean_word_length': 5.0262935713424195, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 1, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Deep Learning Neural Networks for Emotion Classification from Text: Enhanced Leaky Rectified Linear Unit Activation and Weighted Loss

Accurate emotion classification for online reviews is vital for business organizations to gain deeper insights into markets. Although deep learning has been successfully implemented in this area, accuracy and processing time are still major problems preventing it from reaching its full potential. This paper proposes an Enhanced Leaky Rectified Linear Unit activation and Weighted Loss (ELReLUWL) algorithm for enhanced text emotion classification and faster parameter convergence speed. This algorithm includes the definition of the inflection point and the slope for inputs on the left side of the inflection point to avoid gradient saturation. It also considers the weight of samples belonging to each class to compensate for the influence of data imbalance. Convolutional Neural Network (CNN) combined with the proposed algorithm to increase the classification accuracy and decrease the processing time by eliminating the gradient saturation problem and minimizing the negative effect of data imbalance, demonstrated on a binary sentiment problem. The results show that the proposed solution achieves better classification performance in different data scenarios and different review types. The proposed model takes less convergence time to achieve model optimization with seven epochs against the current convergence time of 11.5 epochs on average. The proposed solution improves accuracy and reduces the processing time of text emotion classification. The solution provides an average class accuracy of 96.63% against a current average accuracy of 91.56%. It also provides a processing time of 23.3 milliseconds compared to the current average processing time of 33.2 milliseconds. Finally, this study solves the issues of gradient saturation and data imbalance. It enhances overall average class accuracy and decreases processing time.


Introduction
With the rapid development of e-commerce and social networks, the Internet contains more and more usergenerated reviews [1][2][3] consisting of user opinions and emotions, which have significant business implications [4,5]. Because the emotions reveal user attitudes towards a product or service, they can be used for predicting sales, stock market trends, and more [6]. Therefore, research on emotion classification techniques of texts has significant commercial value for companies [7,8]. A short example from a movie review illustrates some of the problems: This paper proposes an Enhanced Leaky Rectified Linear Unit activation and Weighted Loss (ELReLUWL) algorithm for enhanced text emotion classification and faster parameter convergence speed. It defines the inflection point and the slope for input values as smaller than the inflection point for the activation, which stops the gradient from saturating and is less computationally complex. It also defines each class's weight when backpropagating the loss, which compensates for the negative effect of unbalanced classes in the dataset. The algorithm significantly impacted the average class accuracy for emotion classification in user generated online reviews and reduced the processing time by combining a modified leaky rectified linear unit function with a modified loss function. This study, which addresses a binary sentiment problem, aims to reduce the risk of gradient saturation and reduce the negative effect of the imbalance in the training data, thus improving the classification ability of the system and accelerating the parameter convergence. Training data imbalance is common in any form of sentiment analysis and, more so, for emotions, where a wide range of different classes of emotion may lead to "skewed training data" [23].
The structure of the remainder of this paper is as follows. Section 2 is a literature review that outlines current methods in text emotion classification using deep learning. Section 3 presents our proposed solution while the experiment setup and the results are illustrated in Section 4. The discussion is presented in Section 5. Finally, we conclude this paper in Section 6, which summarizes contributions and limitations and suggests directions for future work.

Literature Review
Deep learning systems, which do not rely on extensive hand-crafted feature engineering [14], have attracted researchers' attention in NLP for some time as they can automatically extract features. A range of these techniques used for text-based emotion classification is here reviewed. These models are mainly based on three types of neural networks: Convolutional Neural Networks (CNN), Long Short-Term Memory (LSTM), and a Hybrid of CNN and LSTM. The baseline SOTA is presented at the end of this section.

Emotion classification based on CNN
CNN is frequently selected for emotion classification and have achieved excellent results in a range of studies. Stojanovski et al. [14] proposed a model to improve sentiment analysis emotion classification. They proposed a model that combined CNN with GloVe word embedding to obtain enhanced accuracy. They included a sigmoid activated layer and a tangent activated layer in the CNN to improve performance further. They also suggested that adding more layers to the network would improve performance. This research achieved considerable improvements with an accuracy of 58.84 in emotion identification and an F1 score of 64.88 in sentiment analysis. However, the training data imbalance problem still needs to be solved [24]. Zhang et al. [9] improved sentiment classification in a text by proposing a model using three different word embeddings to integrate the sentiment meaning into semantic embeddings. The coordination of the attention vector with CNN enables the model to extract both local and global features.
Additionally, cross-modality consistent regression (CCR) further enhances the model's performance. It achieved remarkable high performance with 0.6% improvement in the F1-score and an accuracy of 88.83. However, the overfitting issue that commonly occurs in the deep neural network was ignored, which decreases overall average class accuracy [25]. Moreover, multiple layers increase computational cost and processing time.
Liu [21] solved the overfitting problem and provided improved emotion classification from texts. He adopted a dropout strategy in the softmax classifier which significantly reduced the overfitting problem commonly occurring in traditional CCN methods. He further utilized a Continuous Bag of Words (CBOW) language model for the CCN to improve emotional information capture. The solution provides robust and precise text emotion classification in both English and Chinese, with an accuracy of 87.2% and 90.5%, respectively. However, the sigmoid function used in the convolutional layer is prone to gradient saturation which, as argued by Zhao et al. [26] in their evaluation of the sigmoid function, decreases the overall Cite as: Hui Yang, Abeer Alsadoon, P. W. C. Prasad, Thair Al-Dala'in, Tarik A. Rashid, Angelika Maag & Omar Hisham Alsadoon (2022). Deep learning neural networks for emotion classification from text: enhanced leaky rectified linear unit activation and weighted loss. Multimed Tools Appl . https://doi.org/10.1007/s11042-022-12629-1 4 | P a g e performance. In addition, there is no strategy for dealing with the imbalanced data, which is commonly encountered in this domain [24]. Phillips et al. [20] solved the gradient saturation problem and increased the average class accuracy with an 8layer CNN model with optimized structure. After experimenting with different functions, they selected a leaky rectified linear unit as the activation function and 'max-pooling' as the pooling function. This work achieved outstanding performance with an accuracy of 97.65%, sensitivity of 97.96%, and specificity of 97.35%, respectively. However, the multiple layer structure increases the computational complexity [27] and is not efficient in handling small-size datasets. Thus, the hyperparameters need to be optimized through random search methods rather than only by experience [20].
Gu et al. [28] improved the performance of aspect-based sentiment classification with a cascaded model with 2-level CNN. The proposed cascaded CNN contains aspect mappers to map the input sentence into the corresponding aspect accurately. When the input belongs to predefined aspects, the sentiment classier will determine the sentiment polarity. The pre-trained word2vec embeddings significantly improve the performance of aspect mapping and emotion classification. This solution outperforms traditional machine learning approaches with F1 score up to 83.74 in aspect mapping, and an accuracy of 84.87% in sentiment classification. However, the system contains only a single emotion classifier, which is not suitable in complex textual environments [29]. When encountering more than one emotion type in a sentence, the classification will not be accurate. Zhang et al. [2] proposed a model to enhance text emotion classification. They proposed a 3-way-CNN with a confidence divider to separate the test data into three classes (i.e. negative, positive and boundary). A combination of Support Vector Machine and Naive Bayes (NB-SVM) is used as the enhanced model to reclassify data in the boundary region. Although the solution provides improved accuracy at 90.3% in text emotion classification, its performance is restricted by CNN and NB-SVM. If the test data is difficult to classify for both CNN and the enhanced model, this model will not be able to provide an accurate prediction [2]. Additionally, although the confidence function is essential for this model to effectively classify the boundary data, it still needs optimization to achieve further improvements.

Emotion classification based on Long Short-Term Memory (LSTM)
Unlike CNN, LSTM, a variation of Recurrent Neural Networks (RNN) [30], can identify long-term dependencies in the input [31,32]. Different models using LSTM have been proposed to improve the capability of emotion classification. Mahmoudi et al. [33] compared a range of approaches from traditional machine learning methods to deep learning approaches for improved emotion prediction in the stock market. Then they introduced a solution using LSTM to seize long-term dependencies to emphasize the linguistic objects. They adopted domain-specific word embedding GloVeST with non-static capabilities that allow manipulation of word positions to reflect the word's positive or negative character more accurately. The solution achieved enhanced investor emotion classification performance, as confirmed by the Wilcoxon Sum-Rank Test (WSuRT) with p≤0.0283. However, the researchers identified the limits of their work as originating from the small size of the data set as they had to rely on 'StockTwits' alone. Furthermore, this research classification is limited to only two types (i.e. bullish and bearish) which becomes a problem in a bullish market where the training data class imbalance is of such magnitude that it requires extensive modifications before it can be applied to other domains [6].
Chatterjee et al. [34] improved deep learning baseline approaches to predict emotions from user utterances. They proposed a solution with two LSTM layers with semantic and sentiment features for more precise emotion classifications. This significantly improved emotion prediction over other solutions with p<0.005 in the McNemar's test. However, the conversation context is not taken into consideration, which may lead to classification errors. Furthermore, accuracy can be further improved by extending the research to more emotion classes such as fear, surprise, and so on [6]. Rao et al. [35] improved the accuracy of documentlevel emotion classification. They proposed a model which contains two LSTMs to extract enough successive document representations from a document. The first LSTM can capture the semantic representation of sentences; while with the second LSTM layer, the model can extract the associations of sentences in a document. It achieves average class accuracy of 0.639 and MSE of 0.46, respectively. However, setting a maximum number of sentences in a document limits performance [36]. Also, this work models documents sequentially, which is not suitable for domains where documents are not composed in a sequential way. Kratzwald et al. [24] introduced several remedies to improve the performance of inferring the emotional state within narrative contents. The proposed customized LSTM model integrates dropout regularization, bidirectional processing, and weighted loss functions to deal with the data imbalance problem. The function loss can significantly reduce using regularization [3,37]. It outperforms the other options by 23.2% for the F1-score and 11.6% for MSE, respectively. Also, sent2affect implementation of transfer learning further enhances improvements by up to 6.6%. However, this area's datasets are still exceedingly small compared to other deep learning applications [6]. Thus, larger datasets need to be created to take full advantage of deep learning.

Emotion classification based on a Hybrid of CNN and LSTM
Many approaches combine CNN and LSTM models to capture both short-range and long-range dependencies [29], and in many cases, these combinations have provided SOTA systems. Chen et al. [36] improved accuracy in sentence-level sentiment classification by proposing a model containing a bidirectional LSTM and conditional random fields (BiLSTM-CRF) to classify the sentence type. The model also includes a 1-dimensional CNN to classify the emotion, tailored to specific sentence types by identifying the number of emotions expressed in each of the sentences. By dividing these sentences into groups, it reduces the complexity for each category which means, the learning algorithm has less complexity to deal with in the training data. This pipeline strategy obtains improved accuracy with the binary labeled version of Stanford sentiment treebank (SST) (88.3%) compared to 85.4% for Customer Reviews (CR). However, since in the SST neutral sentences have been removed, a real comparison is limited. This work is also limited in terms of sentence-level analysis [35] as the maximum sentence length is set and sentences exceeding this length are shortened while sentences that do meet the minimum requirements are lengthened by 0 vectors. The CNN will not be able to generate enough relationship information between sentences in a long text [17]. Also, the fixed maximum length of a sentence needs optimization to avoid losing useful information. Lin et al. [38] achieved enhanced emotion prediction in multi-media. They introduced an Explicit Emotion Signal based emotion extraction method to catch the middle-level features from images and texts. This solution uses a network to train each modality (LSTM for text and CNN for images) and then combines the features. It achieves an acceptable performance (accuracy of 56.4% in text sentiment learning) while only requiring 3% training samples for the Visual Geometry Group (VGG) model in images and 43% training samples for the LSTM model in texts, to achieve the same performance. However, a multi-label learning problem exists, which slows down processing [38].
Sun and Zhang [30] improved the accuracy of emotion classification from social media. They introduced a hybrid method combining CNN and LSTM. It also involves a multivariable Gaussian method for multidimensional emotion processing and enables better anomaly detection. It obtains an average class accuracy of 94.19%, abnormal emotion detection accuracies of 83.49% (in terms of the individual user) and 87.84% (in terms of the different month). It considers the local semantic information (which is extracted by CNN) and takes global contextual information (which is done by the LSTM channel) into account. However, selecting thresholds for abnormal emotion is time-consuming, which slows down the system [34]. Therefore, a solution needs to be found to shorten the computation time for selecting the optimal threshold for a more efficient model. Colnerič and Demšar [6] enhanced emotion prediction with three emotion classification systems (i.e. POMS's, Plutchik's and Ekman's classifications of emotions) at one time. They proposed a unison model which works on a large amount of data while using a weighted sampling strategy for more precise results. This unison model can predict three classifications at once and achieve comparable performance in emotion prediction over single models, although the class imbalance problem remains unsolved [24]. Therefore, relations between emotion classifications need to be explored to expand the prediction to other classifications and develop a universal emotion detection algorithm that is not constrained to one domain.
Sun [39] advanced the performance of short text emotion classification. They proposed a hybrid method to Cite as: Hui Yang, Abeer Alsadoon, P. W. C. Prasad, Thair Al-Dala'in, Tarik A. Rashid, Angelika Maag & Omar Hisham Alsadoon (2022). Deep learning neural networks for emotion classification from text: enhanced leaky rectified linear unit activation and weighted loss. Multimed Tools Appl . https://doi.org/10.1007/s11042-022-12629-1 6 | P a g e extract local features and global context information to improve emotion classification accuracy. They also used multi-granularity data augmentation [40] to pre-train the hybrid model to create sufficient data representation for enhanced classification ability. This solution shows great improvements in short text emotion classification, with F-score reaching 0.9054. The combination provides a model with more in-depth insight into data, despite data imbalance problems [24], and unsupervised pre-training [27].
Clearly, there are numerous options for selecting a deep learning algorithm, depending on the intended application. However, there remain challenges in the form of overfitting, gradient saturation, data imbalance, small data sets and dropout regularization. From among the above research, we considered Liu [21], Phillips et al. [20], and Vinay Kumar et al. [4] for our study. Liu [21] was considered since they solved the problem of overfitting through the addition of a dropout strategy in the softmax classifier. Phillips et al. [20] was considered because they used a weighted loss function to deal with data imbalance, and Vinay Kumar et al. [4] was selected because they addressed dropout regularization. From among these, we chose Liu [21] as our base paper because it is built around CNN, which allows larger data sizes. Vinay Kumar et al. [4] also used CNN, but the multiple layer structure they built increased computational complexity, and their search method was based on experience only to which randomization needs to be added to optimize results. Whilst we will draw on elements of Vinay Kumar et al. [4] for our work, it is not suitable as a base paper because it uses LSTM which cannot process large data sets. The following section explore Liu [21] work as it's our baseline SOTA.

Baseline SOTA
This section describes our base model's features and techniques for text emotion classification, proposed by Liu [21]. As shown in Fig.1, useful features are highlighted inside the broken blue line, and the limitations are highlighted inside the broken red line. Liu proposed Dropout regularized CNN combined with Continuous Bag of Words (D-CNN+CBOW) to enhance text emotion average class accuracy. This solution applies a dropout strategy to the softmax classifier (highlighted in blue in Fig.1) to effectively prevent overfitting and enhance generalization ability [21]. Therefore, the overall average class accuracy has improved. CNN can automatically extract semantic features from the text, without a large amount of manually labelled data. The CBOW model continuously constructs distributed word representations for CNN to capture emotional information more effectively. Moreover, the combination of CBOW and D-CNN increases convergence speed. Compared with traditional methods, this solution provides more robust and more precise text emotion classification in both English and Chinese, with an accuracy of 87.2% and 90.5%, respectively. This model has two main stages (see Fig.1): 1) CBOW language model for constructing word vector representation; 2) Dropout regularized CNN for feature extraction and classification.
CBOW model constructs word representations: This includes input, projection, and output layers. The input layer first reads the corpus words sequentially and maps them in the projection layer. The index of each word is obtained using the hash table. The projection layer calculates the cumulative sum of surrounding context words for the target word. After that, the output layer performs regression analysis operation on the cumulative sum of context to generate the target word's vector value. Unlike traditional word bag models, this system generates continuous distributed representations of words, which forms more suitable word embeddings for DNN to capture text features more effectively [41].
Dropout regularized CNN for feature extraction and classification: This consists of the convolutional layer, pooling layer, and a fully connected layer. The CNN takes the word vectors obtained from the CBOW model as input, then transforms them into a matrix, where the number of rows is the sentence length, and the number of columns is the word vector dimension [42]. The filters (sizes of 2, 3, and 4, each size has 2 filters) in the convolutional layer convolve upon the matrix to create the feature maps. Then the feature maps are condensed by the pooling layer to decrease the number of parameters. The maximum value of each map is extracted, forming a single feature vector as the next layer's input, which captures the most important information. After that, the dropout algorithm is applied in the softmax layer by randomly disabling parameters according to a predefined ratio [43]. This dropout can effectively minimize the possibility of overfitting and increase CNN's generalization capability [44]. Simultaneously, the classification result is produced by using the softmax function with the final parameters of the network. In addition, this training process also adopts a backpropagation strategy, which distributes the overall error backwards and updates the weights and biases through a random gradient. The advantage of backpropagation is that it optimizes the network settings and minimizes errors [21,24].
The experiment results illustrate that this solution has achieved remarkable text emotion average class accuracy, with 90.5% in Chinese and 87.2% in English, respectively. However, the sigmoid function is used as the activation function in a convolutional layer, which is likely to be affected by a gradient saturation problem in gradient-based learning models [26]. Another limitation is that there is no strategy for handling the imbalanced data, which is commonly involved in natural language processing. The gradient of the sigmoid function tends to vanish when the function value approaches 0 or 1. It causes the neurons to backpropagate the gradient approaching 0, which means the neural network fails to update the parameters to optimize the training [45]. Therefore, the use of the sigmoid function in convolutional layers impacts the model's recognition ability and reduces overall class accuracy. It also slows down convergence, thus making the model less efficient. Furthermore, without handling imbalanced training data, the model is prone to bias towards frequent training data classes [14,24]. This bias causes inaccurate results and leads to a decrease in classification performance. This solution provides a precise text emotion classification in both English and Chinese, with an accuracy of 87.2% and 90.5%, respectively. The whole training algorithm is presented in Table 1, while the flowchart is presented in Fig. 2. In the convolutional layer, the output of each neuron is calculated by the activation function. It uses the Cite as: Hui Yang, Abeer Alsadoon, P. W. C. Prasad, Thair Al-Dala'in, Tarik A. Rashid, Angelika Maag & Omar Hisham Alsadoon (2022). Deep learning neural networks for emotion classification from text: enhanced leaky rectified linear unit activation and weighted loss. Multimed Tools Appl . https://doi.org/10.1007/s11042-022-12629-1 8 | P a g e weight and bias and the preceding layer's input to collect the features from the text. In this solution, the sigmoid function is used to compute neurons' output, which is expressed in equation (1) [21]. And the input of the sigmoid function is generated with the input of the previous layer, weight, and bias, which is expressed in equation (2) [21]. However, the sigmoid function is prone to suffer from gradient saturation, which affects the average class accuracy and reduces efficiency. This problem can be solved using the leaky rectified linear unit (LReLU) function [20].
Where, j: the j th feature map e: the natural logarithm : the activation function : the input, which can be expressed in equation (2): Where, i: the i th neuron n: the total number of input neurons l: the l th layer j: the j th feature map −1 : the input neuron from the l-1 th layer for the j th feature map : the weight of the ith neuron for the j th feature map : the bias at the lth layer for the j th feature map The goal is to minimize the cross-entropy loss [46], which can be defined in equation (3). However, the imbalanced data problem has not been considered, which can impact the overall average class accuracy. This problem can be solved by calculating the weighted loss, which multiplies the loss of each input with a corresponding weight [24].
Where, k: the numbers of the classes n: the numbers of samples L: cross-entropy loss : the desired output of class k : the predicted probability of class k

Proposed System
After reviewing various feature extraction and classification methods using deep learning for text emotion classification, each method's advantages and drawbacks are analyzed. It was found that the activation function, loss function, and weight are the paramount factors that affect CNN's performance.
From the list of reviewed classification methods, the solution proposed by Liu [21] is selected as the baseline SOTA work and used as the foundation for the proposed solution. The main reason for selecting this solution is to propose a Dropout regularized Convolutional Neural Network (D-CNN) to prevent overfitting. Because of overfitting, it is difficult for the neural network to generalize the unknown data, which results in poor performance [35]. Dropout randomly selects parameters participating in the training. Therefore, the neural network can be turned into multiple combinations to enhance generalization ability effectively and avoid overfitting, thus improving average class accuracy [21]. In addition, the D-CNN is combined with a Continuous Bag of Words (CBOW) model, which further enhances overall average class accuracy. The CBOW model constructs continuous distributed word representations which enable CNN to capture emotional information more effectively.
However, there still are some limitations to this work. The activation function is the sigmoid function, which is likely to cause gradient saturation problems [20]. The gradient saturation impacts the overall average class accuracy and slows down the convergence speed [45]. Another limitation is that there is no strategy for handling the imbalanced data, resulting in a bias towards more frequent classes in the training data [24]. While Liu [21] used the same data sets for testing and training, there were some inconsistencies in terms of corpus size and data split. The IMDB data set is perfectly balanced, although ten times larger than the COAE2014 for which data are skewed towards the positive by 42%.
This bias leads to inaccurate results and to decrease the training data classification performance. To overcome the gradient saturation problem, the proposed solution introduces a modified rectified linear unit (MLReLU) function inspired by Phillips et al. [20]. They draw their insights about RELU from their own 2015 work in which they relied on Hara et al. [47] to justify the use of this tool. They have argued that RELU is a generic non-linear function that merely facilitates passing data from one deep learning layer to the next and that currently, RELU is the choice for many applications. They also point to its versatility by referring to the work of Microsoft, IBM, Yahoo!, Twitter, and Adobe who work with RELU and RELU versions for text extraction.
Another modified feature of this solution is calculating the weighted loss adapted from the work by Kratzwald et al. [24], to deal with the imbalanced training data. These two enhanced features solve some of the current solution's problems and improve average class accuracy and convergence speed. The proposed system consists of two main stages (Fig.3): (1) word representation and (2) feature extraction and classification. Word representation: follows the continuous bag-of-words architecture by Liu [21] which consists of the input, projection, and output layer. The input layer first reads the corpus words sequentially and maps them in the projection layer. Then the index of each word is obtained using the hash table as shown in Fig.3. The projection layer calculates the cumulative sum of context words for the target word. After that, the output layer performs a regression analysis operation on the cumulative context sum to generate a vector value for the target word.
Feature extraction and classification using CNN: The CNN model consists of a convolutional layer, a pooling layer, and a fully connected layer. The CNN takes training samples as input and maps them into word vectors according to the CBOW word representation collection. The training sample is then converted into a matrix where the number of rows corresponding to the sentence length and the columns are the word vector dimensions. The filters in the convolutional layer convolve on the matrix to create a variable feature map. The activation function calculates each unit in the convolutional layer. Here the modified LReLU (MLReLU) function is used. Compared to the sigmoid function, MLReLU is simpler and solves the gradient saturation problem. It allows the neurons to update effectively and accelerates the convergence of the CNN model [20]. After that, the feature maps are refined by the pooling layer to extract each map's maximum value. Then all the maximum values form a single feature vector, which contains the most relevant information. And then the model randomly drops out parameters at a predefined rate. After dropout, the classification result is produced by using the 'softmax' function with the network's final parameters. Then the cross-entropy loss is calculated using a modified cross-entropy loss function. The modified loss function takes each class's weight into account and reduces the bias towards the more common classes [24]. After that, the calculated loss is backpropagated through the neural network to update the weights and bias for The complete training strategy Enhanced Leaky Rectified Linear Unit activation and Weighted Loss (ELReLUWL) algorithm is presented in Table 2, and the flowchart of this algorithm is presented in Fig. 4. Convert all the words in s into word vectors Step 2: Compute the convolutional outputs to extract the feature values using MLReLU activation Step 3: Combine all the feature values into feature vectors Step 4: Perform the max (feature vectors) to extract the most important features Step 5: Perform dropout on the feature vectors Step 6: Calculate the probabilities for each class using softmax function Step 7: Compute the error using the modified loss function Step 8: Backpropagate the error through the model to update the parameters of hidden layers END

Proposed Equation
The proposed solution replaces the sigmoid function with a modified activation function to reduce gradient saturation, which would otherwise impact the classification performance. It uses the leaky rectified linear unit (LReLU) function as the basis, as the gradient of LReLU will never saturate. The LReLU function is expressed in equation (4) adapted from Phillips et al. [20]: where, ℎ( ): the activation function x: the input 0.01: the constant slope for the negative inputs However, the slope and the inflection point in equation (4) are constant, which is not the best parameter for the function [48]. To solve this, we also adopt the idea from the displaced rectified linear unit (DReLU) function [22], which defines a better inflection point rather than the origin. The DReLU function is expressed in equation (5): Where, ( ): the activation function x: the input a: defines the point where the inflection happens But the gradient of equation (5) for the inputs that are on the left side of the inflection point is 0, which is undesirable. The slope definition in equation (4) can solve this problem. Therefore, equation (1) will be modified by combining the advantages of the equation (3) and (4)  The proposed solution further enhances the accuracy with a modified loss function. It combines the crossentropy loss function with each data sample's weight to reduce the bias towards the frequent classes in the training data. More specifically, it multiplies each sample's error with a corresponding weight. The modified loss function is expressed in equation (7).
The weight for each class is calculated with the number of samples and the number of classes, as well as the number of samples belonging to that class that depends on its ground truth label [24]. The calculation of the weight for each class is expressed in equation (8).
Where, m: the m th training sample n: the number of samples k: the total number of classes j: the j th training sample 1{ = m } is the indicator that the j th sample shares the same ground truth label with the m th sample Applying equation (8) to equation (7), the modified loss function can be expressed in equation (9): Where, ML: the modified loss k: the number of classes n: the number of samples j: the j th training sample m: the m th training sample 1{ = m }: the indicator that the j th sample shares the same ground truth label with the m th sample : the desired output of class k : the predicted probability of class k Thus, the enhanced overall classification performance can be expressed in equation (10):

Area of Improvement
Two equations were proposed to improve the performance of the neural network. First, the modified activation function to calculate the output of neurons, as shown in equation (6). It uses the modified LReLU (MLReLU) function as the activation function instead of the sigmoid function. The purpose is to prevent the gradient saturation problem, which would impact the model's recognition ability. With the help of MLReLU function, the model is simpler and more powerful in terms of recognizing the unseen data. It, thus, has higher average class accuracy and faster convergence speed. In addition, the MLReLU function enhances performance by defining the most appropriate point where the inflection should happen. Second, the crossentropy loss function is combined with the weight of each data point. It multiplies the error with each data point's weight to counterbalance the negative effect of uneven emotion class distribution in the training data. With the help of the proposed equation (9), it reduces the model's risk inclining to more common classes and provides more accurate classification. Hyperparameter validation experiments with deferent input values were conducted to define the best parameter value of MLReLU. The five experiments' mean value was calculated, and 0.03 was selected as the best parameter value.

Why Enhanced Leaky Rectified Linear Unit activation and Weighted Loss (ELReLUWL) algorithm?
Using a modified LReLU function as the activation function in CNN effectively avoids the gradient saturation problem. It uses the identity for input values larger than the inflection point, and non-zero slope for the remaining inputs. The proposed system can effectively solve the gradient saturation problem as the MLReLU function's gradient is always greater than 0. With the definition of an appropriate inflection point, it further enhances performance. In addition, the MLReLU function is simple and fast to compute, resulting in a more efficient training process. Moreover, the proposed modified loss function takes the imbalanced class distribution in the training data into account. It multiplies the error with weight to alleviate the negative effect of imbalance. Each data's weight is determined by its ground truth label and is inversely proportional to the number of samples sharing the same class. Preventing the gradient from saturating in the neural network can improve the model's average class accuracy and training speed. Due to gradient saturation, the neurons backpropagate the gradient close to zero. When the gradient becomes too small, the neural network's early layers take a long time to train. It results in the latter layers losing important information from early layers when learning, which reduces the overall classification performance. It is also difficult for the neurons to update the parameters to tune the model as it learns. This phenomenon causes poor recognition performance and slow parameter convergence. Therefore, the proposed solution that solves the gradient saturation problem enhances the average class accuracy and accelerates the parameter convergence. In addition, highly imbalanced data is commonly involved in emotion analysis. Without a suitable strategy to deal with the imbalanced data, the model will generate results like a majority vote. The introduction of the weight in the loss function effectively prevents this classification bias towards more frequent classes. The activation function used in the baseline SOTA system is the sigmoid function which is prone to suffer gradient saturation problems. The proposed system solves this problem with the modified LReLU function, which its gradient does not saturate. In other words, the proposed system has a higher classification ability and takes less time to train. Furthermore, the proposed system deals with the imbalanced training data by multiplying errors with corresponding weights. The modified loss function can effectively reduce bias and enhance average class accuracy. Python 3.6.4, along with libraries such as Keras, Tensorflow, NLTK, and matplotlib [6], were used to implement this research. Two freely available datasets of online reviews and can download from [50], the Polarity dataset [51] and the IMDB movie review dataset [35] were used. For experimental purposes, we divided the IMDB dataset into two different datasets, i.e., IMDB dataset 1 and IMDB dataset 2 to allow testing of results from datasets with and without imbalance. Therefore, three datasets in total were used, and they vary in the number of samples and data balance degree, properties of which are listed in Tables 3 and 4. Five-fold cross-validation was used, where the data were randomly divided into five equal-sized sets, of which one was used for testing and the remainder for training and validation [34]. We opted for a high number of training data due to the hyperparameters' complexity in the algorithm [52] and because many researchers used an 80/20 split when k=5.
The model was trained using stochastic gradient descent with a dropout rate of 0.4, a learning rate of 0.2, a filter size of 3, 4, 5, respectively. Batch size of 100 and a hidden unit of 200 were used. For the experiment, a system configured with Intel® Core™ i5-3337U CPU@ 1.80 GHz and 4.00GB RAM was used. The experiment is divided into two parts. The first part covers three scenarios, i.e., a small dataset where the number of samples is relatively small (Polarity dataset), a large dataset where the number of samples is relatively large (IMDB dataset 1), and an unbalanced dataset where the number of negative review samples is twice that of positive review samples (IMDB dataset 2). From each dataset, 20% of the data was used for validation, for which results are shown in Table 5. The second part was based on two classes, i.e., negative and positive reviews. A comprehensive experiment has been performed on the negative and positive review samples with results shown in Tables 6, 7 Samples of text processed within the IMDB datasets highlighting the complexity of such reviews: Positive: "Almost missed it. While visiting friends in Philadelphia sometime in the early 1980s, I was channel surfing after everyone else went to bed. It wasn't just Bogart I was obsessed with; but rather the entire era of those old flicks those of my age know so well. Add to that a plot like The Maltese Falconwhere so many different characters were interacting with Sacchi -and you have a piece of art as far as I'm concerned. About ten years later it appeared on TV, and I taped it." This message is largely neutral interspersed with positive markers.
Negative: "Seagal needs to get back to basics breaking bones and kicking butt. No more of this slow-motion crap like a foreigner and in the shadows fighting like half past dead. Exit wounds showed more of his fighting skills with some wires which were ok, but then he went back to be a movie director".
This message is more complex as it contains an element that commends Segal on his fighting skills. However, the overall tone is one of dissatisfaction and disappointment. For such types of comments, finegrained analysis achieves more accurate results.  In the feature extraction and classification stage, the CNN model was created to extract features from the labelled training data automatically. The features form feature maps which are fed into the max-pooling layer. The fully connected layer then uses the pooling layer's output to generate the emotion classification (i.e. positive or negative) with the help of the 'softmax' function. After being trained, the model is evaluated using the validation data. Fig.5 presents accuracy performance on the IMDB dataset 1 at the training stage for both the baseline SOTA solution [21] and our proposed solution. The two solutions achieve similar accuracy; however, the proposed solution takes fewer epochs to achieve optimal accuracy. This means that the proposed solution has reduced processing time for training the model for the large dataset.   Table 5, in terms of accuracy and processing time. Our proposed solution results are compared with the baseline SOTA solution using data reporting and bar graphs. The results are obtained for two aspects: 1) based on the three scenarios (i.e. small dataset, large dataset, and unbalanced dataset); 2) based on the two different review types (i.e. negative and positive). Firstly, for the different scenarios, the results are divided into three categories. For each scenario, the results are shown based on two different stages, i.e. training and validation. The results for each dataset are presented in terms of accuracy and processing time. More specifically, accuracy is calculated in terms of the percentage of correctly classified samples against the total labelled samples. And the processing time is measured in terms of the number of epochs required to reach convergence. The comprehensive test was performed on 20% of the three datasets' samples (i.e. Polarity, IMDB 1, and IMDB 2). The overall average of accuracy and processing time was calculated by averaging the training stage and validation stage results for the respective datasets, which are shown in Fig.7 and 8. For baseline SOTA, the average accuracy and processing time for the Polarity dataset is 83% and 15.5 epochs, the IMDB dataset 1 88.25% and 7.5 epochs, and the IMDB dataset 2 92.42% and 11.5 epochs. For the proposed solution, the average accuracy and processing time for the Polarity dataset is 90.85% and 10 epochs, the IMDB dataset 1 90.33% and 4 epochs and the IMDB dataset 2 95.31% and 7 epochs. Secondly, the negative and positive review samples from each dataset were tested, and the results for each sample are shown in Tables 6,7,8,9,10,and 11. We have performed the tests dividing each type (i.e. positive and negative) into ten strata to obtain average class accuracy and processing time. For the polarity dataset, each of these 20 strata consisted of 20 individual samples (total 400), while for IMDB1 and IMDB2, 20 strata of 100 (2000) and 30 (600) were used, respectively.
The average class accuracy is measured in terms of the probability of each class's labelled data while the processing time is measured in terms of the execution time to classify each test sample.
The results were obtained in the classification stage of CNN. The results are analyzed based on the two aspects mentioned above. The proposed solution has improved the average class accuracy and processing time by using equation (6) to reduce the risk of gradient saturation. And the accuracy is further enhanced with the help of equation (9) to reduce the bias in the dataset. This will improve the automatic analysis of 19 | P a g e product and service reviews for business organizations.

Discussion
The results show the improvement in average class accuracy and processing time of the proposed solution compared to the baseline SOTA solution based on text emotion classification. With the modified activation function and modified loss function, the proposed solution provides an average class accuracy of 96.63%, which is more than 5% higher than the current solution [21]. The proposed model also requires less convergence time to achieve optimization with 7 epochs which are 4.5 epochs below the requirements of the current solution.
In addition, the proposed solution decreases the average processing time to 23.3 milliseconds which is 10 milliseconds less below that needed by the current solution. The accuracy for each sample is obtained using the Predict() method from the Python Keras package where true positives and true negatives are used. Microsoft Excel. The overall degree of improvement in average class accuracy and processing time is quantified by running the baseline SOTA algorithm and the proposed solution algorithm, respectively. Accuracy is calculated using equation (11) above [30].
Using the modified Leaky ReLU (MLReLU) function as the activation function in the convolutional layer prevents gradient saturation and improves the proposed system's performance. This modified feature is inspired by the work of Phillips et al. [20]. The main benefit of the MLReLU function is that its gradient will never saturate so that the parameters can be updated effectively to achieve model optimization. The use of a weighted loss function is another feature of our proposed solution, which was adapted from Kratzwald et al. [24]. It reduces the bias in the imbalanced data and provides enhanced accuracy. Thus, the proposed solution provides improved average class accuracy and reduced processing time when compared to the baseline SOTA solution. In conclusion, CNN's combination with the proposed ELReLUWL algorithm has greatly improved the online review emotion classification with enhanced accuracy of 96.63% and reduced processing time of 23.3 milliseconds.
A significant number of techniques has been implemented for emotion classification of text. These techniques' goals are consistent: to achieve higher average class accuracy while using lower processing time.
The current solution's limitations have been successfully overcome with this research, with an average accuracy of 96.63% against the current accuracy of 91.56%. This research also reduces the average processing time to 23.3 milliseconds against the current processing time of 33.2 milliseconds. This is due to improvements in the activation function to minimize the risk of gradient saturation and the weighted loss function to reduce data bias. Hence, the proposed solution has better performance across different data scenarios (i.e., small dataset, large dataset, and unbalanced dataset) and different review types (i.e., positive and negative reviews).

Conclusion
Understanding emotions from online reviews is particularly important for marketers to capture user opinions and attitudes to products and services. Accurate classification of emotion from the reviews boosts the chance of business success. Many deep learning approaches have been utilized to achieve this. However, there are limitations in terms of accuracy and processing time. A modified leaky ReLU activation function has been proposed, which reduces the risk of gradient saturation when the model is trained to improve accuracy and convergence speed. The weighted loss function is another feature adapted for this research to minimise the negative effect of bias in the imbalanced dataset and further enhance average class accuracy. Therefore, the proposed solution has reduced the convergence time by 4.5 epochs on average, improved accuracy by more than 5% on average, and decreased the processing time by 10 milliseconds on average. Because there are no publicly available labelled datasets for multi-class emotion classification, this research is limited to binary classification (i.e., positive and negative). In the future study, multi-class datasets for different emotion types (e.g., joyful, surprise, angry, sad, and so on) should be used to train and test the model to extend it to broader domains. Other enhancement methods such as a data augmentation technique could be leveraged for better feature extraction and further advancing the classification performance.