{'fraction_non_alphanumeric': 0.03551195246649164, 'fraction_numerical': 0.023352217769794112, 'mean_word_length': 4.721739130434782, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
GCDH@LT-EDI-EACL2021: XLM-RoBERTa for Hope Speech Detection in English, Malayalam, and Tamil

This paper describes approaches to identify Hope Speech in short, informal texts in English, Malayalam and Tamil using different machine learning techniques. We demonstrate that even very simple baseline algorithms perform reasonably well on this task if provided with enough training data. However, our best performing algorithm is a cross-lingual transfer learning approach in which we fine-tune XLM-RoBERTa.


Introduction
In recent years, the spread of negative comments and hatred through social media has created a focus on the detection of misinformation and hate speech in the NLP community.
On the other hand, Hope Speech detection is a relatively new task. It can be employed to identify the positive aspects of large collection of social media posts, e.g. detecting pro-peace voices during politically heated situations. These detection efforts can potentially counter-act the perceived majority of hatred, and be a means to prevent harsh measures such as the disabling of internet access (Palakodety et al., 2020).
The ability to identify and therefore foster these aspects of online communication is a step towards a more positive representation of the internet. The hypothesis is: if hate-speech can incite violence, Hope Speech can ease tensions. For the first time to our knowledge, a shared task has been announced to identify Hope Speech in a multi-lingual dataset of Youtube comments. (Chakravarthi and Muralidaran, 2021)

Data
The dataset consists of three different subsets. It contains 10,705 comments for Malayalam, of which 8,564 are assigned for training, 1,070 for validation, and 1,071 for testing. For Tamil, the organizers provided 20,198 comments, of which 16,160 were designated for training, 2,018 for validation, and a test set of 2,020 comments. The English data consists of 28,451 comments in total, 22,762 for training, 2,843 for development, and 2,846 for testing.
The organizers provided annotations for the classes Hope Speech, non Hope Speech, and not-"target language", where "target language" is either Malayalam, Tamil, or English. Figure 1 illustrates how the different classes are represented in the data sets. Notably, we can observe that the Tamil data is more balanced, whereas the Malayalam and English data sets show an over-representation of non Hope Speech data. Furthermore, only 22 comments in the English training set are labelled "not-English", but 12 % of the comments in the Tamil training set are "not-Tamil", and 8 % of the Malayalam comments are "not-Malayalam".
Annotations for the Hope Speech class include utterances that convey a generally bright prospect to the future, are supportive, insightful, and promote values such as inclusiveness and equality, among others. Some of these concepts for the English training set are illustrated in figure 2. For a full description of the annotation process, see (Chakravarthi, 2020).

Baseline
Using scikit-learn (Pedregosa et al., 2011), we implemented two baseline algorithms for the task of Hope Speech detection, a Naive Bayes algorithm, and a Support Vector Machine. From the short texts, we first extracted a number of features including unigrams, bigrams, length and number of stopwords. Additionally, we assigned emotion scores for all three languages based on the NRC emotion lexicons (Mohammad et al., 2013). However, we found that solely relying on the textual data using a simple vectorizer based on tf-idf yielded better results.
As a first baseline, we implemented a Complement Naive Bayes model (Rennie et al., 2003). We iteratively found the best smoothing parameters α = 1.3 for Malayalam, α = 10.5 for Tamil and α = 1.4 for English.
For the second baseline, we used an SVM with a degree of three and linear kernel. We found the best regularization parameters c = 1.3 for Malayalam, c = 1.1 for Tamil and c = 1.4 for English.

Transformer Model
Our main model is based on XLM-RoBERTa (Conneau et al., 2019), a pre-trained neural language model that uses the transformer architecture (more specifically, we use the xlm-roberta-base version from the huggingface transformers library (Wolf et al., 2020)). It is particularly useful for this task, because it is already pre-trained on 100 different languages, including Malayalam, Tamil and English.
We preprocess the provided datasets by tokenizing the comments using the pre-trained Senten-cePiece (Kudo and Richardson, 2018) tokenizer that belongs to XLM-RoBERTa. All comments are padded or truncated to a length of 128 tokens. XLM-RoBERTa is able to implicitly infer the language used in a text based on the tokens that it contains, which enables it to deal with code-switching. This also means that we can use batches that contain multiple languages at the same time during training.
The model is fine-tuned for hope speech detection on the combined training datasets for all three languages. In order to account for the different sizes of the datasets, Malayalam and Tamil samples are assigned sample weights of 2 and 3, respectively. We use a batch size of 64 and the Adam (Kingma and Ba, 2015) optimizer with a learning rate of 2 · 10 −5 for 4 epochs. We further employ decoupled weight decay (Loshchilov and Hutter, 2019) with a decay rate of 0.01 and shrinking learning rates for early transformer layers (Sun et al., 2019) with a decay rate of 0.95. Since the datasets only contain a small portion of all possible tokens, the token embeddings are deliberately not fine-tuned. Training takes place on an Nvidia GeForce RTX 2080 Ti in mixed precision mode and takes roughly 20 minutes. We observe that even a simple algorithm such as Complement NB results in a high F1-Score of 0.91 for English. However, for Malayalam the SVM performs better with an F1-Score of 0.80. For Tamil, both baseline systems perform similarly with 0.60 (SVM) resp. 0.61 (CNB) F1-Score.   The official results on the test set (see table 3) place the model at first rank for English and Malayalam and fourth rank for Tamil in the competition.

Analysis
The improvements the fine-tuned XLM-RoBERTa model offers over the baseline models are not surprising, since large transformer models have a high capacity and are able to take the order of tokens into account. Differences between the languages are likely related to structural properties of the datasets, which is supported by the fact that the other competitors received similar results. In a similar way, imbalances in the datasets as described in section 2 can explain why the classifiers perform better on the non Hope Speech class than the other classes.

Conclusion
In this paper we presented several machine learning models trained for Hope Speech Detection in three different languages (Malayalam, Tamil and English). As baseline models we used SVM and Complement Naive Bayes classifiers for each language. The best results were achieved by a crosslingual transfer learning approach in which we finetuned XLM-RoBERTa. In the competition, this model achieved the first rank for Malayalam and English and the fourth rank for Tamil. In conclusion, we provide a cross-lingual model that is both effective and relatively fast to train.