{'fraction_non_alphanumeric': 0.033180374161666075, 'fraction_numerical': 0.013942816801976702, 'mean_word_length': 4.942842160461458, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
HS-BAN: A Benchmark Dataset of Social Media Comments for Hate Speech Detection in Bangla

In this paper, we present HS-BAN, a binary class hate speech (HS) dataset in Bangla language consisting of more than 50,000 labeled comments, including 40.17% hate and rest are non hate speech. While preparing the dataset a strict and detailed annotation guideline was followed to reduce human annotation bias. The HS dataset was also preprocessed linguistically to extract different types of slang currently people write using symbols, acronyms, or alternative spellings. These slang words were further categorized into traditional and non-traditional slang lists and included in the results of this paper. We explored traditional linguistic features and neural network-based methods to develop a benchmark system for hate speech detection for the Bangla language. Our experimental results show that existing word embedding models trained with informal texts perform better than those trained with formal text. Our benchmark shows that a Bi-LSTM model on top of the FastText informal word embedding achieved 86.78% F1-score. We will make the dataset available for public use.


Introduction
Hate speech (HS) has been rapidly spreading in recent years through various social media sites. HS detection in the user's comment section remains a difficult challenge due to lack of formal language syntax, spelling mistakes, and use of various slang and non-standard acronyms. [1]. Researchers have started working in this very challenging domain, but much of their efforts concentrated on English language [2]. There are almost 46 million Facebook 1 and 29 million Youtube 2 Bangladeshi users but there has been a severe lack of large, linguistically diverse Bangla HS datasets.
Existing datasets for Bangla HS have some significant drawbacks. First, as Table A.1 of Appendix A shows, the datasets are not large enough. Most of them contain less than 10k sentences. Secondly, the majority portions of those datasets come from one or two domains, which makes them domain-dependent [3], [4] and [5]. Thirdly, annotating HS is inherently a complex and challenging task [1]. [6] showed how annotator's bias on HS can influence the annotation process and subsequently affect classification task. To combat this, [7] created a stringent and detailed annotation guideline, providing specific points on what constitutes HS and what not. Although [8] followed a guideline to prepare a small Bangla HS dataset, it lacks inter-annotator scores, which makes it difficult to judge annotation quality. To the best of our knowledge, only [9] mentioned Cohen's kappa score in their paper.
In this paper, we manifest HS-BAN, the largest binary class Bangla HS dataset comprised of more than 50,000 comments crawled from Facebook and YouTube. We present a few examples from the dataset in Table 1. To ensure cross-domain generalization, we collected comments from seven categories as presented in Section 2. A strict annotation guideline was followed to reduce human annotation bias for labeling the dataset achieving an inter-annotator agreement score of 0.658. In Section 3, we present the different natural language processing approaches we used to extract linguistic features by processing noisy social media text. Finally, in Section 4, we present the benchmarking outcome for hate speech classification showing that a Bi-LSTM model on top of the FastText informal word embedding achieved 86.78% F1-score. The dataset and source code will be made publicly available to foster future research. Table 1: Examples of the annotate HS-BAN dataset Data Collection: We collected more than 100,000 comments from different YouTube channels and Facebook pages in seven categories: sports, entertainment, crime, politics, religion, celebrity, and miscellaneous. For each category except miscellaneous, we compiled a list of controversial events that happened recently (2017-2020) in Bangladesh that falls under those categories. Then public comments were extracted on those issues by using an open-source tool called Facepager 3 . Furthermore, for the miscellaneous category, we searched for videos on YouTube related to Bangla TikTok, roasting, or similar videos as the comment section in these videos tend to be very toxic. We removed duplicate, highly similar comments based on Jaccard Index > 0.8 to reduce repetitiveness and ensure a diverse vocabulary.

Dataset
Annotation: We prepared a detailed annotation guideline based on the community guidelines of Facebook and YouTube (Appendix B). We worked with 50 annotators who are undergraduate students within the age range from 20 to 25 years. They know about the sensitive nature of the task and they willingly volunteered in this research. Three annotators tagged each comment, and the majority vote decided the final decision. The inter-annotator agreement score [10] of 0.658 indicates that our dataset has moderate agreement among annotators.
Dataset Statistics: Our dataset has 50, 314 comments in total; among them, 20,209 of the dataset comments are HS and 30,105 are Not-HS, which implies that our dataset is moderately imbalanced. A more detailed statistics of our dataset is presented in Table 2. We can see that each category is also moderately imbalanced, especially the politics and celebrity categories, whereas crime has the highest hate speech percentage. Observing the average word count (AWC) for each category, we can find that, celebrity category has the highest AWC per sentence with the most standard deviation while miscellaneous has the lowest.

Experimental Setup
We removed all hashtags, numbers, and non-Bangla words. However, we kept emoji and punctuation (e&p) in order to confirm [1]'s claim whether e&p can be good indicators for HS. F1 score and Matthews correlation coefficient (MCC) were used to evaluate all models. We split the dataset into train(80%) and test(20%) datasets using stratified sampling so that each category in both dataset contains an equal ratio of hate to not-hate comments. The training dataset was further divided into train(80%) and development(20%) datasets for feature selection and hyper-parameter tuning purposes.
Machine Learning Models: We trained linear Support Vector Machine (SVM) [11] with Term Frequency Inverse Document Frequency (TF-IDF) weighted score as a feature. Regularizer C, penalty, and loss were fine-tuned to find the best hyperparameter combination. And for the deep learning model, we fine-tuned Convolutional Neural Networks (CNN) as described in [12] to get the best F1 score in the development set. We also implemented Bi-directional Long Short Term Memory (Bi-LSTM) as described in [13]. Transformer Models: We experimented with some transformer-based models: monolingual Bangla-BERT [17], multilingual BERT-cased and uncased [18]. The hyperparameters were set according to literature [9] except for mBERTuncased. In this case, the learning rate was changed from its original value of 5e-5 to 3e-5 as the validation loss during the initial training phase was terrible.

Result and Analysis
Baseline analysis: We present our experimental findings in table 3. We experimented with different combinations of char and word ngram, and only the most notable combinations were presented. It becomes apparent that in our dataset, e&p do not serve as valuable features. Word n-gram with e&p had the worst F1 score, and for char n-gram (1,6) and (2,6), when e&p are present, the F1 score drops. For this reason, in hyper-parameter tuning and neural network experiments, e&p were removed in the prepossessing step. The experimental results show that char n-gram as a feature performs better than word n-gram. Our dataset contains many spelling mistakes, which means the same words can have inconsistent and multiple spelling variations for which SVM peforms poorly [2]. The performance drops with experiments combining char n-gram with word n-gram further validated the fact. Overall, char ngram (1,6) is the best performing feature with F1 score of 85.86.
Formal vs Informal Embeddings: We observed that, SVM with char (1-6) gram as feature outperformed neural networks with FastText and BFT as embedding by a good margin. [19] found similar observations for their informal sentiment analysis dataset. They proposed that pre-trained models based on formal text do not perform well on the informal dataset. Two FastText informal embedding FT(SG) and FT(CBOW) outperformed SVM in three out of four instances. Only CNN+FT(CBOW) achieved a lower F1 score. Most notably, they were trained on a small 1.47 million informal sentences compared to embeddings such as BFT and FastText but still managed to outperform the bigger embeddings by a large margin. From the experiments, we conclude, word embedding trained from scratch on informal corpus will perform better for the informal dataset collected from social media.
According to [20], FastText is generally a better choice compared to Word2Vec for our dataset. We can also see that W2V(SG) and FT(SG) outperformed W2V(CBOW) and FT(CBOW), respectively. From this, we can summarize that word embedding created on the SG method tends to perform better than the CBOW method for our dataset. Overall, combining Bi-LSTM with FT(SG) or Bi-LSTM+FT(SG) achieved the best result with an F1 score of 86.85%.
Transformer vs Other: All BERT variants performed significantly worse compared to other models. Possible reasons could be that mBERT is under-tuned for Bangla informal text as it was trained with Bangla Wikipedia for which we observe the lower performance. The experimental success with informal word embedding over formal embedding support the last assumption.   Influence of slang words: The dataset vocabulary was extracted, and slang words were annotated with the help of a linguistic expert. These slang words were divided into two types: traditional slang (TS) and non-traditional slang (NTS). NTS is not usually used as slang words but can be interpreted as such depending on the context. Figure C.1 in Appendix C shows some examples of both types of slang words. Then we looked at how models predicted comments that contain at least one such slang word. For simplicity, we ignored comments that contained both TS and NTS words.

Model
Our initial hypothesis was that models would face more trouble detecting sentences NTS as these words are often used as HS as well as NHS. Table 5 shows this result. TS acc shows how models predicted comments containing TS and NTS acc is for NTS. We can see that models performed better when comments contained TS words rather than NTS. NTS words are often vague in meaning, and so they pose special difficulty for the models.

Conclusion
In this paper, we present HS-BN, the largest Bangla HS dataset collected from social media comments. We followed various schemes to ensure linguistic diversity and reduce repetitiveness. We found that emoji and punctuation do not affect HS detection. We also found that word embedding trained on informal text outperforms available embedding trained on formal text. We also observed that traditional slangs make the detection of hate speech much easier than that of comments with non-traditional slangs. In the future, we plan to research on the interpretability and generalizability of hate speech detection model.