{'fraction_non_alphanumeric': 0.025477707006369428, 'fraction_numerical': 0.012848671205798374, 'mean_word_length': 4.7453207150368035, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 35, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Sentiment Analysis with Deep Learning Models: A Comparative Study on a Decade of Sinhala Language Facebook Data

The relationship between Facebook posts and the corresponding reaction feature is an interesting subject to explore and understand. To achieve this end, we test state-of-the-art Sinhala sentiment analysis models against a data set containing a decade worth of Sinhala posts with millions of reactions. For the purpose of establishing benchmarks and with the goal of identifying the best model for Sinhala sentiment analysis, we also test, on the same data set configuration, other deep learning models catered for sentiment analysis. In this study we report that the 3 layer Bidirectional LSTM model achieves an F1 score of 84.58% for Sinhala sentiment analysis, surpassing the current state-of-the-art model; Capsule B, which only manages to get an F1 score of 82.04%. Further, since all the deep learning models show F1 scores above 75% we conclude that it is safe to claim that Facebook reactions are suitable to predict the sentiment of a text.


I. INTRODUCTION
Understanding or analyzing human emotions could be quite a complex task for a computer. Nonetheless, the numerous possible applications of this task has driven researchers around the globe to attempt to tackle it through varying perspectives, causing sentiment analysis to become a leading research area in natural language processing. A wide range of fields including movie reviews [1], product reviews [2], and political studies [3] benefit from advancements in this study area.
However, a noteworthy gap in research on sentiment analysis, which is currently available, is the lack of interest in the perspective of the observer. Most research studies attempt to understand the sentiment expressed by the creator of the content. Only an extremely limited number of researchers such as Hui et al. [4] have attempted to analyze the sentiment invoked by the content in an observer. While there are instances where the emotional reaction of the creator of content is more important than that of the observer, in certain instances such as political campaigns and brand management, understanding the reaction of the observer towards content could be quite important [5].
Facebook reactions, which could be considered as user sentiment annotations for Facebook posts, contain an immense potential in the field of sentiment analysis from the perspective of the observer, due to the convenient availability of massive amounts of data. [5][6][7][8].
Through this paper, we attempt to assess the applicability of Facebook data for sentiment analysis with respect to the Sinhala language. In contrast to English, for which numerous resources as well as previous research studies are readily available for sentiment analysis, the amount of research done in Sinhala in this arena remains highly unsatisfactory [9]. The fact that most of the research conducted on the subject are outdated, incomplete, or access-restricted contribute heavily to this issue [10]. To the best of our knowledge, the reigning state-of-the-art model for sentiment analysis in Sinhala is the Sencat tool [11]. The name of the tool is an abbreviation for Sentiment Categorization. The developers of the tool have tested the performance of several wellknown sentiment analysis models for a data set consisting of over 15, 059 Sinhala news comments [12]. Using their results as a benchmark, we have tested the feasibility of Facebook data as a means for sentiment analysis in Sinhala.
Here it should be noted that our reaction sentiment data set is significantly (ten times) larger than that of Senevirathne et al.. The reaction sentiment data set contains 150, 000 Facebook posts with 235, 660 sentences as discussed in Section III-A.
Results of this study indicate that the highest performance metrics are achieved by the 3-Layer Stacked BiLSTM model, with accuracy, recall, precision, and F1 score values of 83.59, 83.59, 85.59, and 84.58 respectively. These are significantly higher than the best results obtained by Senevirathne et al. [12]. Furthermore, in contrast to the results obtained by Senevirathne et al. where the best values of different performance metrics were scattered among different models, the consistent high performance of the 3-Layer Stacked BiLSTM model across all metrics provides further reassurance regarding the ability of the model to predict the sentiments included in the reaction sentiment data set used in this study. The utilization of a larger data set is a major reason for this consistency as well as the significant increase in performance scores of all models tested.

II. BACKGROUND
Many of the currently available studies on sentiment analysis aim to understand the market for various products and services. The work of Fang and Zhan [2] is seminal on product review analysis. It introduces a procedure for sentiment polarity categorization of product reviews, in which the sentiment analysis is carried out at a phrase-level. Phrases and words that convey a sentiment are identified as sentiment tokens, for which a sentiment score is calculated following the popular star rating model. Feature vectors developed using them are then used for predicting the sentiment polarity of previously unseen product reviews. The study by De Silva et al. [20] also tackles the same problem, but at an aspect level. It calculates sentiment values separately for each aspect included in a review. An important property of this study is that it takes into account several semantic features such as negations, sentiment enhancements, sentiment shifts, and groups of words. This helps to increase the accuracy of transforming the sentiments embedded in text into mathematical values. The method proposed by De Silva et al. [20] has been tested for the corpus we derive the reaction sentiment data set from by Jayawickrama et al. [5].
Wang et al. [13] present a multitude of deep neural network architectures that combines CNNs and RNNs for sentiment analysis. These models exploit the key characteristics of both types of neural networks by using CNN layers to capture lower order features and RNN layers to capture higher order features. Though the study was originally conducted for capturing sentiments included in short text, the models included exhibit great performance in sentiment analysis even for longer text.
This study was carried out using, reaction sentiment data set which is a corpus composed of Facebook posts and their metadata. Numerous researchers have taken a similar path before, including Pool and Nissim [6] and Freeman et al. [21] who utilized Facebook data sets for sentiment detection. However, the work by Freeman et al. [21] is limited to a narrow scope of scholarly articles on Facebook. On the contrary, Pool and Nissim [6] has used a more generalized data set consisting of a wide range of sources. The authors explain the motive behind this choice as the need to choose the optimum sources to train machine learning models for each Facebook reaction. Additionally, the authors have conducted research on models with features such as TF-IDF, n-grams, and embeddings. A detailed comparison of the models is presented by them, which provides valuable insight into their use cases.
While the aforementioned studies provide useful insight into sentiment analysis, there is a major barrier that our study has to face that many of them might find unfamiliar. Most of those research work relate to the English language, which is resource-rich. In contrast, our study relates to Sinhala language, which could be considered a resourcepoor language in the NLP domain [10]. Research work related to sentiment analysis in Sinhala are highly limited, and most of the initiated research studies in this arena were abandoned or not released to the public [9].
The Sencat sentiment analyzer by Demotte et al. [11] is the current state-of-the-art model for Sinhala sentiment analysis. The tool was originally trained using a data set comprising of 15, 059 purely Sinhala news comments. The sentiment classification of news comments is done at the document-level, under the assumption that the whole document represents a single emotion. Sencat has been presented in two formats, namely:1) Binary Class Classification: Positive and Negative 2) Multi Class Classification: Positive, Negative, Neutral, and Conflict In the case of multi class classification, news comments that display neither positive nor negative emotion are assigned to the Neutral class while comments that display both positive and negative emotions are classified as Conflict. Prior to introducing sentiment categorization, Demotte et al. [11] have tested the performance of several sentiment analysis tools built using deep learning techniques against a data set composed of Sinhala news comments [12] which they published earlier.
Jayawickrama et al. [5], on the other hand, have introduced a set of simple machine learning models for sentiment analysis in Sinhala.In which they have introduced three models that have been developed specifically targeting sentiment analysis with Facebook data, and has been tested with the reaction sentiment data set which we use in this study. A simple procedure of using the average of sentiment vectors of previously seen data to predict that of unseen data is utilized in these models. All Reaction Set Model and Core Reaction Set Model attempt to predict the distribution of Facebook reactions while the Star Rating Model concatenates reactions into a single value representing where it would fall in the scale between extreme positivity and extreme negativity. The work of Medagoda et al. [22], although dated a few years back, provides an insightful exploration of the applicability of tools available for resourcerich languages to resource-poor languages such as Sinhala.

III. METHODOLOGY
In this section we discuss how we create reaction sentiment data set using the Facebook data set composed by Wijeratne and de Silva [23] and test it on the sentiment analysis models proposed by Senevirathne et al. [12] and Jayawickrama et al. [5] where the former includes the model that obtains state-of-the-art results for direct Sinhala sentiment analysis.

A. Data Set
This study is conducted using a portion of the Facebook data set composed by Wijeratne and de Silva [23]. It was selected as the most suitable for the study for a multitude of reasons. Firstly, Facebook, unlike most other social medial platforms, enables the users to annotate posts with a set of sentimental reactions that are representative of both positive and negative sentiments. At the time the data set was composed, a total of 7 Facebook reactions were available: Like, Love, Wow, Haha, Thankful, Sad, and Angry. These reactions chosen to be applied to posts by Facebook users can be considered as a user sentiment annotation for posts, eliminating the need for a separate dedicated manual sentiment annotation of data sets, which is a tedious and highly resource consuming task. These reactions have a direct correlation to the actual sentiment invoked in a user by the post, which means, the reaction given by a user can be taken as the sentiment that particular user was expressing with a high confidence as opposed to a third party guessing and tagging a data set after the fact [24]. However, due to behaviours such as sarcastic reactions [5], it should be noted that Facebook reactions may not have a straightforward one-to-one mapping to the actual sentiment experienced by the users 100% of the times. Nevertheless, it is still a more accurate measure of reaction sentiment than what a third party may tag. The deviant reactions, such as sarcastic reactions, are an interesting aspect to explore in future studies.
The original data set created by Wijeratne and de Silva [23] consists of 1, 820, 930 Facebook posts taken from pages popular in Sri Lanka over a time period ranging from 01-01-2010 to 02-02-2020. Post text as well as several metadata, including reaction counts for each of the Facebook reactions, are among the features present in the data set. For this study, a subset of 150, 000 posts, which we refer to as reaction sentiment data set has been extracted from the original corpus. The counts and the percentages of each reaction included in this reduced data set is presented in Table I.

B. Pre-processing
The version of the data set that Wijeratne and de Silva [23] had made available has been pre-processed by them to a certain extent. However, we observed that their procedure has resulted in some characters of the Sinhala language being removed as well. Therefore, we obtained the raw corpus and performed pre-processing procedures better tailored per our requirements. The corpus used needed several pre-processing steps in order to conform to the requirements of this study. The Message field contained raw, unprocessed text including nonprintable characters, special characters, and text in various languages that would not be useful within the scope.
Non printable characters belonging to the unicode categories Cc, Cn, Co, and Cs were substituted with white spaces [25]. The Cf category too was replaced in the same manner, with the exception of the character Zero Width Joiner, which often appears in the middle of Sinhala words consisting of characters such as rakāransaya, yansaya, and rēpaya shown in the Figure 1 . Singular unicode characters to represent these characters have not been introduced due to them being derivatives of existing Sinhala unicode characters. Thus a method of including the Zero Width Joiner between characters that phonetically represent the said characters is often utilized in Sinhala text to indicate their occurrence. As such, replacing the Zero Width Joiner with a space could lead to erroneous tokenization of Sinhala words. Therefore, it was replaced by a null string instead.
From the data set thus filtered, email addresses, URLs, user-tags (format: @user), and hashtags were removed. Numerical content too was filtered out as they do not possess a significant sentimental value. Since the corpus used by Senevirathne et al. [12] was composed exclusively of Sinhala text, to facilitate a fair comparison, any words that included characters which did not belong to the Sinhala language were removed from the corpus. Furthermore, the set of stop words for the Facebook data set generated by Wijeratne and de Silva [23] were removed as well, since stop words possess a rather low information value in the context of sentiment analysis [26]. As the final pre-processing step, multiple continuous white spaces were replaced by a single white space.

C. Normalization and Annotation
The set of deep learning models used by Senevirathne et al. [12] perform a binary classification of text, classifying them as either Positive or Negative based on a sentimental perspective. To evaluate the performance of our corpus against those models, the Facebook reactions needed to be integrated into a single binary value. To achieve this end, only the Love, Wow, Sad, and Angry reactions were considered, omitting the Like, Thankful, and Haha reactions.
Like and Thankful reactions could be considered as outliers in comparison to the other reactions. In the reduced data set used for this study, a total count of 38, 889, 706 Like reactions were present, which amounted to 88.4% of the total reaction count. In contrast, only 7 Thankful reactions were present in the corpus, which is a negligible value in comparison to the counts of other reactions. Inclusion of these reactions in the target label annotation would introduce a massive imbalance to the data set. The unusual behaviour of these two reactions could be attributed to the duration they were present on the Facebook platform [5]. The percentages of the remaining reactions after removing Like and Thankful is shown under the Filtered column in Table I.
Furthermore, the sentiment depicted by the Haha reaction is a controversial matter as Facebook users tend to use it both genuinely and ironically, causing the sentiment category of the reaction to vary between Positive and Negative though it was originally introduced to Facebook as a positive reaction [5]. Thus, the Haha reaction too has been excluded in the binary classification of reactions. Love and Wow are considered as Positive while Sad and Angry are considered as Negative. Figure 2 portrays the distribution of the considered reactions in the data set.
Since the focus of this study is to analyze the distribution of reactions received by a post, inclusion of the raw number of reactions could create an unwelcome bias towards posts with high reaction counts. To overcome this issue, steps were taken to normalize the reaction counts of posts. For a given Facebook post, the summation of reactions t is taken using the count of reactions Love(c l ), Wow(c w ), Sad(c s ), and Angry(c a ) as demonstrated through Equation 1. The normalized value n r of a given reaction r for a certain Facebook post is obtained by dividing the reaction count of the said reaction c r by the total reaction count t of the post.
The normalized value of positive reactions for a post is obtained using the Love and Wow reactions, as shown by This sentiment value is then utilized to determine the sentiment category to which the post is assigned. Equation 6 shows the classification process.
It should be noted that Senevirathne et al. [12] presents two classification methodologies: binary and multi-class. However, the multi-class classification does not befit our study given that their definitions of categories do not match the nature of our data set. They define the Neutral class as text that possess neither positive nor negative sentiment, and the Conflict class as text that possess both positive and negative sentiment. Only text that exclusively exhibit positive or negative emotions are classified as Positive or Negative in this classification. In our corpus, Facebook posts that have neither positive nor negative reactions would be posts that have not received any reactions from the considered classes, which depicts more of a lack of reaction data than a true sense of neutrality. Furthermore, a large proportion of posts would be allocated into the Conflict class as the same post that invokes a positive reaction in one Facebook user often has the capability to invoke a negative reaction in another user, and vice versa. An argument may be raised here to say the instances where sen value of Equation 5 equal to 0 might map to the union of Neutral and Conflict classes. However that definition is extremely narrow given that the probability of a post having exactly equal pos and neg values are extremely low. Thus, if the sen = 0 rule is used, we will be introducing an extreme minority class. As we saw on the discussion on the Thankful reaction in Jayawickrama et al. [5] as well as Section III-C, this would have resulted in an undesirable outcome.
Thus, the binary classification where text that exhibits more positive sentiment than negative is classified as Positive and vice versa is selected for this study. Upon completion of those steps, 116, 061 of the 150, 000 Facebook posts included in the data set were classified as Positive and the remaining 33, 939 were classified as Negative.

D. Model Training
Following the conventions set by the benchmark Senevirathne et al. [12], we next use the fastText [27,28] model with 300 dimensions trained using the aforementioned Sinhala News Comments data set to convert the post text included in each Facebook post to vectors. The prepared data set is then used to train each deep learning model tested by Senevirathne et al. [12] and two models introduced by Jayawickrama et al. [5]; Core Reaction Set Model and Star Rating Model.

IV. RESULTS
The results obtained from this study following the steps described in Section III is presented in Table II along with a summarized view of the results reported by Senevirathne et al. [12] for the purpose of comparison. Here, the performance measures accuracy, recall, precision, and F1 score of each model are displayed in the columns A,R,P, and F1 respectively.
Upon comparing the results of the two research studies, a clear increment in all performance metrics could be observed in the results of this study. The highest F1 score reported by Senevirathne et al. is 59.42, while the lowest F1 score received by a deep learning model in this study is 77.39. Only the simple models by Jayawickrama et al. exhibit F1 scores lower than the peak F1 score of Senevirathne et al.. It should be noted that the performance of the Core Reaction Model, though lagging behind in comparison to the results obtained through this study, lies in a comparable range with the deep learning models in the results of Senevirathne et al., with a F1 score of 49.80. The highest F1 score observed in this study reaches as high as 84.58.
In contrast to the results reported by Senevirathne et al. [12], the 3 Layer BiLSTM model exhibits the best results for every performance metric in this study, with accuracy, recall, precision, and F1 score values of 83.59%, 83.59%, 85.59%, and 84.58% respectively. The BiLSTM model which produced the best accuracy and recall values in Senevirathne et al. [12] has proved its worth with accuracy and recall values of 82.58% each, which are the second highest values observed in those metrics. However, the HAHNN model which exhibited the highest precision score in Senevirathne et al. [12] has slumped down to the 13th place with a precision value of 77.54%. This model displays the lowest increase in performance metrics upon comparing the results of the two studies. The 3 Layer Stacked BiLSTM model, which has produced the highest F1 score in the work of Senevirathne et al., has continued to excel achieving the highest values in all performance metrics in this study. The state-of-the-art model; Capsule-B highlighted by Senevirathne et al. as the best in the set, has been pushed down to the 4th rank with respect to the F1 score in the new results, outperformed by the BiLSTM model as well as the 2 and 3 Layer Stacked BiLSTM models. It is noteworthy that all BiLSTM models except for the CNN + BiLSTM model has achieved remarkable F1 scores in our experiments.
The performance of the baseline models; GRU, LSTM, and BiLSTM have experienced a drop when a CNN layer is added to the model in both experiments. This counterintuitive result may be a result of the specific configuration of CNN that Senevirathne et al. [12] proposed. However, given that we needed to do a fair comparison with the results they have reported, when used for reaction sentiment data set we did not vary the CNN in this study. We discuss this further in Section V. The lowest performance across both studies is observed in the Star Rating Model, with a F1 score of 33.77%. Though the model has achieved a fairly decent precision value of 68.96%, the rather weak recall value of 22.36% has pushed the model down to this position.
It should be highlighted is that the 10-fold cross validation method was used to obtain the performance measures reported by Senevirathne et al.. Since this study uses 150, 000 rows from the Facebook data set in contrast to the 15, 059 rows used in the work of Senevirathne et al., the excessive resource consumption of using 10-fold cross validation was too costly. Therefore, the holdout method was used with the data set split into development and test sets in the ratio 8:2, with the development set again divided into train and validation sets in the ratio 9:1. However, even with this setback to training, this study achieved better results than Senevirathne et al. [12] as shown in Table II.

V. CONCLUSION AND FUTURE WORK
The results of this study exhibit a clear improvement of performance metrics in comparison to the work done by Senevirathne et al. [12]. The differences among F1 scores  of the two studies are visualized in Fig. 3. The size of the selected data set can be postulated as a major reason for the drastic improvement of results in this study; the data set we used is almost 10 times the size of the corpus used by Senevirathne et al. [12]. As deep learning is generally data hungry [29], the 15, 059 entries in their data set may have been insufficient to properly train the models. The fact that a larger user annotated data set could produce better performance than a smaller manually annotated data set is a promising result for resource poor languages, for which manually annotated data sets are rather difficult to create or find in comparison to user annotated social media data sets. Even in the cases where manually annotated data is available, in configurations such as parallel corpora, it has been observed that the data is of poor quality for Sinhala [30].
The BiLSTM models outshine the rest of the models in the experiment, with the exception of the BiLSTM model with a CNN layer. This effect could be due to the ability of BiLSTM models to be trained in both positive and negative time directions [16] , which is countered by upper layers in the stacked models filtering out contextual information from positive and negative time sequences when the model is paired with a CNN [17].
In the work of Senevirathne et al. [12], the reason for the lack of performance of baseline models paired with a CNN layer is explained as the lack of data entries to train the model properly. Furthermore, since only one layer of one dimensional CNN layer is used, a different configuration of the CNN layer may alter the results. However, since changing the CNN layer configuration may lead to unfair comparison with the model used in the work of Senevirathne et al. no adjustment was done as discussed earlier in Section IV. The HAHNN model [18] shows the lowest improvement in the performance compared to other models. Since the HAHNN model focuses on the attention mechanism introduced by the work of Vaswani et al. [31], the lack of respect to strict grammatical structures observed in colloquial text in our corpus may have hindered the capability of HAHNN.
The Core Reaction Set model and the Star Rating model taken from Jayawickrama et al. [5], being derived from classical statistical modeling, lack the complexity to perform well in the sentiment analysis task compared to other models. Thus, further improvements will be required in order for them to become effective sentiment analysis models.
The word embedding model used in this work has been trained using the Sinhala News Comments data set. This can be replaced by a new word embedding trained using the Facebook data set itself, which could capture the relationships among words in the data set more effectively. Different word embedding models such as fastText, Word2vec [32], and Glove [33] with different dimensions could be tested to select the best embedding.
The latest model known as transformers introduced by Vaswani et al. [31] is another area to be explored. Further research is needed to determine the performance of the model with the Sinhala language. However, transformers usually tend to be more data hungry than other deep learning models [29]. The implications of this on resource poor Sinhala may be an interesting study.