{'fraction_non_alphanumeric': 0.0321936610930871, 'fraction_numerical': 0.010232093835787373, 'mean_word_length': 4.602156980227681, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Unsupervised Embeddings with Graph Auto-Encoders for Multi-domain and Multilingual Hate Speech Detection

Hate speech detection is a prominent and challenging task, since hate messages are often expressed in subtle ways and with characteristics that may vary depending on the author. Hence, many models suffer from the generalization problem. However, retrieving and monitoring hateful content on social media is a current necessity. In this paper, we propose an unsupervised approach using Graph Auto-Encoders (GAE), which allows us to avoid using labeled data when training the representation of the texts. Specifically, we represent texts as nodes of a graph, and use a transformer layer together with a convolutional layer to encode these nodes in a low-dimensional space. As a result, we obtain embeddings that can be decoded into a reconstruction of the original network. Our main idea is to learn a model with a set of texts without supervision, in order to generate embeddings for the nodes: nodes with the same label should be close in the embedding space, which, in turn, should allow us to distinguish among classes. We employ this strategy to detect hate speech in multi-domain and multilingual sets of texts, where our method shows competitive results on small datasets.


Introduction
In this paper we investigate an unsupervised, graphbased approach to learn embeddings for hate speech detection. According to (Fortuna and Nunes, 2018), hate speech can be defined as a language that attacks, diminishes or incites violence against groups based on specific characteristics. Accordingly, the aim of hate speech detection is to discriminate texts that contain hate from those that do not. This is a widely studied task that involves a number of challenges (Poletto et al., 2020). Specifically, we study the problem of datapoor settings that appears in low-resource domains and languages -i.e., in those settings where supervised approach are not able to generalise well. For this, we make use of Graph neural networks (GNNs). They are a framework based on deep learning to operate on graphs. They follow a recursive neighborhood aggregation scheme, called message passing, where each node aggregates feature vectors of its neighbors to compute its new feature vector (Xu et al., 2018). After a number of iterations, a node is represented by its transformed feature vector, which captures the structural information within its neighborhood. Therefore, GNNs have been effective at tasks thought to have rich relational structure since they can preserve global structure information of a graph in embeddings. (Wu et al., 2021) provide an overview of recent studies on deep learning approaches for graphs and discuss the applications of GNNs in several areas. (Zhou et al., 2020) present some GNN-based methods that have been applied to text classification, and point out that representing texts as graphs can effectively capture semantics among words. (Yao et al., 2019), for instance, use GNNs for text classification. The authors propose a strategy to represent texts as graphs and use a convolutional graph neural network to learn em-beddings of words and documents. As a result, they show that the improvement of their graph-based model over state-of-the-art models becomes more prominent as lower the percentage of training data is. Cross-lingual transfer learning is one of the strategies used to leverage already existing data from higherresource languages (Ranasinghe and Zampieri, 2020;Stappen et al., 2020;Bigoulaeva et al., 2021). However, this approach can introduce other problems related to the variability between different languages. Moreover, the resulting datasets are heterogeneous not only in terms of languages, but also in terms of domains, which can affect the learning of the models. We aim to study the performance of hate speech detection with GNNs under the motivation of improving this task in data-poor settings without the need of external data. For this, we propose a graph auto-encoder framework that allows us to learn a latent representation from a set of texts in an unsupervised way. In this representation the texts from the same class are close, hence we can use embeddings from this space to efficiently distinguish instances of different classes. Our framework builds a graph with the texts, encodes the nodes in a low-dimensional space (the latent space) and then reconstructs the original graph with a decoder. The encoder is composed of a transformer layer, which introduces an attention mechanism, and a convolutional layer for generating the embeddings. We evaluate the embeddings for hate speech detection as a classification task in small datasets. Moreover, we study the multi-domain and multilingual settings, to experimentally analyse whether graphs can jointly represent different types of information. Our contributions are the following ones 1 : • We propose a graph auto-encoder for unsupervised representation learning on graph-structured data by reconstructing the initial graph. In this framework we incorporate a self-attention mechanism that allows us to adapt the strengths of the Transformer model (Vaswani et al., 2017) in the generation of embeddings.
• We use the embeddings generated with this framework for hate speech detection and show results which outperforms state-of-the-art models in datapoor settings, without using pre-trained word embeddings.
• We investigate the performance of our approach in multi-domain settings, where the small amount of data once more highlights the potentiality of our proposal.
• We extend the analysis for multilingual hate speech detection and use a strategy to aggregate prior knowledge about the language to obtain outstanding results.

Related Work
Hate Speech Detection. Most of the works done to detect hate speech is based on the analysis of textual instances. Other works have studied the phenomenon at the author level (Rangel et al., 2021), where the idea is to analyze a set of texts published by the same author to detect possible propagators of hate on the web. In general, the techniques used for hate speech detection range from traditional machine learning models to methods based on deep learning, such as convolutional neural network and recurrent neural networks, including attention mechanisms (Badjatiya et al., 2017;Gröndahl et al., 2018;Magalhaes, 2019). Due to the nature of the task, it is worth noting the models that take into account certain keywords that may indicate hateful content. The work (De la Peña Sarracén and Rosso, 2021b) proposed an approach for keyword extraction based on the attention mechanism of BERT and a reasoning on a word graph. Experimental results highlighted some points to consider when training models based on keywords. Moreover, the work (De la Peña Sarracén and Rosso, 2021a) studied how models learn bias towards relevant words in the training data.
To extract the relevant words, the authors proposed a keyword extraction method based on the harmonic mean of relative frequencies and the discrimination between hateful and non-hateful texts. In recent years, the bidirectional encoder representations from Transformers (BERT) (Devlin et al., 2019), as well as other Transformer-based models such as RoBERTa (Liu et al., 2019) have been widely used due to their ability to capture language phenomena (Mozafari et al., 2020;Samghabadi et al., 2020). In fact, they have been used in most systems with outstanding results in shared tasks (Basile et al., 2019;Zampieri et al., 2020;Mandl et al., 2019). Moreover, (Mozafari et al., 2019) investigated the ability of BERT for detecting hateful content on social media and the results showed a considerable performance in comparison to other existing approaches. That is why we use this model to compare the results obtained with our framework.
Graph Neural Network for Abusive Language Detection. Regarding GNN-based models, the literature points out a number of strategies (Koncel-Kedziorski et al., 2019;Shi et al., 2021a). (Peng et al., 2018) proposed a graph-based deep learning model to convert texts to graphs of words, and then used graph convolution operations over the graph. (Yao et al., 2019) represented documents and words as nodes to construct a graph and used a convolutional graph neural network to learn embeddings of words and documents. As a result, the authors showed improvements over state-of-the-art models for text classification. However, very little has been studied to employ strategies based on GNNs to address the problem of hate speech detection. (Mishra et al., 2019) proposed a convolutional graph neural network for capturing the structure of online communities and the linguistic behavior of the users. They showed that the resulting heterogeneous graph significantly advanced the state of the art in abusive language detection. Thus, to the best of our knowledge, our method is the first proposal to learn embeddings in an unsupervised way for the specific problem of hate speech detection.

Graph Auto-Encoders for Hate Speech Detection
In this section, we describe the preliminaries of the framework, followed by details of our proposal.

Formalization
In this work we consider hate speech detection as a classification problem which involves the classes hate and not-hate. The data comprises N samples, where each sample is given by In order to generate these feature vectors we use Term Frequency -Inverse Document Frequency (TFIDF) representation of each text in . TFIDF generates vectors from texts in such a way that often produces lower scores for high frequency function words and increases scores for terms that are more relevant in each text, it is well suited for tasks involving textual similarity. Thus, we represent the texts as vectors with TFIDF scores, for what we do not need external sources to train the initial vectors. The set {y i } N i=1 is composed of the labels 0 and 1, which indicate the presence or not of hate in each of the texts in {t i } N i=1 . Then, the aim of hate speech detection is to assign one of the labels to each t i by using Our aim in the present work is to learn embeddings from x i in an unsupervised way to improve the performance in hate speech detection when N is small. Besides, we attempt to use this approach for multi-domain and multilingual hate speech detection, due to the suitability of graphs to jointly represent different types of information. In these cases , where m represents each of the M domains or languages and S m its size, such that N = M m=1 S m . We address the problem by using a graph auto-encoder framework. Following, we describe our framework in details.

Background: Graph Auto-Encoders
Graph neural networks (GNNs) are models based on deep learning to operate on the graph domain. In particular, Graph Auto-Encoders (GAEs) are unsupervised learning frameworks which encode nodes or graphs into a latent vector space. Therefore, they can be used to learn embeddings. In general, they are trained with the aim of reconstructing their original graph input. First, an encoder takes a graph as its input and compresses it into a low-dimensional vector. Then, a decoder takes this vector representation and attempts to generate a reconstruction of the original input. Encoder-decoder pair is designed to minimize the loss of information between the input graph and the output graph (Wu et al., 2021). Formally, let G = (V, E) be a graph, where V and E represent the set of nodes and edges respectively. Let X ∈ R |V |×d be a matrix containing the features of the nodes, such that the i-th row is a d-dimensional feature vector of the i-th node. Moreover, let A ∈ R |V |×|V | be a matrix representation with a representative description of the graph structure, such as the adjacency matrix. A GAE takes as input the matrices X and A to learn a function Z = enc(X,A) and produces a latent representation Z ∈ R |V |×d ′ (embeddings), where d ′ < d is the number of features of the nodes in the latent representation. Then, Z is used to produce an approximate reconstruction outputÂ = dec(Z) such that the error between A andÂ is minimized for preserving the global graph structure. Both functions enc(·, ·) and dec(·) are often defined through stacked layers. Graph convolutional layer (GCL) re-defines the notion of convolution for graph data and are widely used as propagation operators for GNNs in general. The main idea is to operate directly on a graph and induce the embedding vectors of nodes based on the properties of their neighbors. A GCL takes as input the matrices X and A and generates a representation H = f (X, A), where f (·, ·) is a propagation rule. (Kipf and Welling, 2017) introduced the propagation rule (1), where W is a weight matrix and σ(·) is an activation function. The matrixÃ = A + I (I is the identity matrix) contains self-connections to aggregate, for each node, not only the information from its neighbors but also the node itself. Moreover, the matrix D is the diagonal node degree matrix ofÃ, which is used for a symmetric normalization to deal with the problem of changing the scale of the feature vectors.
Graph transformer layer (GTL) adapts the multihead attention of Transformer (Vaswani et al., 2017) for graph learning. This was introduced in (Shi et al., 2021b) considering the case of edge features. Given the by calculating multi-head attention for each node i with its neighbors N (i): where c represents each head, d c its hidden size and C the total number of heads. The vectors q c,i , k c,j and v c,j correspond to the 'query', 'key', and 'value' vectors respectively, and e c,ij is a representation for the edge between i and j. W c,q , W c,k , W c,v and W c,e are the parameters in the head c. Notice that a term r i is calculated to add a gated residual connection between layers. Figure 1 illustrates our auto-encoder. In order to generate the input for the model, we build the matrix X with the set of numeric feature vectors {x i } N i=1 , such that each vector is a row in X. On the other hand, we build the edges among nodes, to generate the matrix A, based on the inner product of the feature vectors. Then, the weight of each edge is defined by the inner product between the original vectors. We only consider edges between node pairs with values higher than a threshold (positive edges). The rest of node pairs are considered as non-existent edges (negative edges). The encoder in our model stacks two layers. The first one is a GTL and the second one is a GCL. In particular, we use a GTL to enrich the node embeddings with attentive information propagation between nodes.

Auto-Encoder Architecture
Thus, the encoder uses a GTL as the first layer to determine the relevance between nodes and their neighbors by leveraging the advantages of the attention mechanism among nodes. In this sense, we adopt the proposal in (Shi et al., 2021b) by considering only nodes and using an unique head. Hence, we transform the input X matrix as (2).
The second and final layer is based on the propagation rule (1) and the ReLU as the activation function. The input of this layer is composed of the new matrixX and the matrix A. Thus, we obtain the output of the encoder as (3), where W c is a parameter matrix.
The decoder implements the idea of the GAE in (Kipf and Welling, 2016). Thus, we base on the inner product of the embeddings to generateÂ. The aim is to decode node relational information from the embeddings by reconstructing A as (4) defines. Then, the auto-encoder (5) is trained by minimizing the negative cross entropy given the real matrix A and the reconstructed matrixÂ.

Experimental Design
In this section, we present our methodology for the empirical evaluation of the capability of our framework for unsupervised learning of embeddings. We also present the used dataset and details for the reproduction of the experiments.

Dataset
We evaluate our proposed auto-encoder framework on the XHate-999 dataset (Glavaš et al., 2020), which was built for abusive language detection. This dataset is composed of large training and validation sets of English texts, and a small multi-domain and multilingual test set. The test set contains text of six typologically diverse languages: English (EN), German (DE), Russian (RU), Turkish (TR), Croatian (HR) and Albanian (SQ). For each language there are three distinct domains: Fox News (Gao) with 99 samples, Twitter/Facebook (Trac) with 300 samples, and Wikipedia (Wul) with 600 samples, for a total of 999 samples per language. We only rely on the test set since our purpose is to study the data-poor settings as well as the multi-domain and multilingual perspective.

Experimental Setup
For each experiment we set the size of the vectors generated with the GTL in the encoder to 32 and the size of the output of the GCL to 16. The auto-encoder was trained using batches of 32 instances and the Adam optimizer with a learning rate of 0.01, in 200 epochs with the strategy of early stopping with patience of 10. For the threshold used in the generation of the matrix A, we searched in {0.01, 0.1, 0.5}, but realized that a value close to the average of the weights calculated for the pairs of vectors fitted in a better way, hence we set this value to 0.07. In the evaluation, we first visualize the embeddings in the latent representation, generated with the encoder of our graph auto-encoder. We also visualize the initial vectors (TFIDF) to visually compare both representations. Secondly, we evaluate the capacity of the embeddings on the task of node classification to study the performance of hate speech detection. In this sense, we use a classifier of two fully connected layers of 32 neurons with the ReLU activation function and the softmax function in a last layer to generate the predictions. This classifier was used to obtain prediction for the texts with the initial representation and on the other hand, with the embeddings obtained with our encoder. Hence, we can compare the results of classification between them. In both cases, the classifier was trained with a size of batch 32, using the Adam optimizer with a learning rate of 0.01, in 10 epochs. For test we separate the 30% of the data and the rest was used to train. That is, we used 30% from the test set of the XHate-999 dataset for testing and the other 70% for training. We run all the experiments five times and report the average scores.

Embeddings Evaluation
In this section we analyze the mono-domain and monolingual evaluation. Therefore, we focus on each domain and each language separately.

Analysis of Latent Representation
In order to better analyze the generated embeddings by our encoder, we use t-SNE (Pezzotti et al., 2017) to visualize the initial and the latent representation of the vectors, corresponding to the hateful and non-hateful texts. As illustration, Figure 2 shows the results for the texts in English in each of the domains Gao, Trac and Wul. In each case, the representation for the initial vectors (with TFIDF) is visualized on the left, and on the right there is the latent representation (embeddings). We can see that our model can be used to distinguish both classes, since the separation between hateful texts (red points) and non-hateful texts (green points) is more evident. Then, with this representation, a simple algorithm can be used to separate both types of texts. Similar behaviour was observed for the rest of the languages.

Evaluation for Hate Speech Detection
The results for hate speech detection, using both the initial vectors and the embeddings, are summarized in Figure 3. In general, we observe an improvement by using the embeddings as the input in the classifier. Thus, we can verify the suitability of the embeddings to discriminate among classes.
Notice that the results between both variants were similar only for the case of Russian texts in the Trac domain. Figure 4 illustrates the initial and latent representation for Russian, where we can see that in the Trac domain it is more difficult to learn embeddings that allow discriminating between classes. This suggests that in this domain and language, the hateful and non-hateful texts are more similar. In future work, we will try to increase the number of convolutional layers in the encoder to make a deeper propagation and analyze if this case improves. Anyway, for Gao and Wul in this same language, we observe better performance for the embeddings.

Multi-domain Evaluation
Besides the mono-domain evaluation, we focus on the multi-domain setting. In this case, the set of texts for the input of the auto-encoder is composed of three different types of data per language i.e.
. In order to compare our classification results with a state-of-the-art model we use the multilingual BERT (mBERT) (Devlin et al., 2019) and XLM-R (Conneau et al., 2020). We used the HuggingFace Transformers framework 2 with the pre-trained models bert-basemultilingual-cased and xlm-roberta-base. For these models, the input is composed of the texts For fine-tuning we used the same setup that we presented above for the classi- fier that we use to evaluate our embeddings. The only difference is that we set the learning rate to 2e-5. Note that in (Glavaš et al., 2020), the authors reported the results on the entire test set. In our experiments, we used a part of the test set 3 times. That is why we reproduced the experiments of that paper with mBERT and XLM-R using the same data that we used to evaluate our framework. Anyway, we observed that the reported  Results and Discussion. Tables 1 and 2 summarizes the results of these experiments. The results obtained by using our embeddings are identified with the acronym GAE. In particular, Table 2 illustrates the results in the multi-domain setting per language. While Table 1 corresponds to the results obtained for each domain separately. We observe two interesting points. First, GAE seems to be more stable in data-poor settings. We verify outstanding results even in Gao, which is the domain with the least amount of data. In contrast, the smaller the dataset, the worse the results obtained with XLM-R.

GAE
Notice that for all the languages, the best results of XLM-R were in the Wul domain, which is the larger one. This confirms the findings in (Yao et al., 2019), where the authors use a model based on a convolutional graph neural network for text classification, and point out that the improvement over state-of-the-art models becomes more prominent as the percentage of training data is lower.
On the other hand, we note that XLM-R and mBERT obtain the worst results in the multi-domain setting. This suggests that heterogeneous data can affect the performance of these models. The behaviour is different for GAE. Although we do not see any gains moving from the mono-domain, we do not observe a considerable decrease. This suggests the suitability of our framework to deal not only with data-poor settings, but also with heterogeneous data.

Multilingual Evaluation
In order to evaluate the multilingual perspective, we consider the combination of all the languages. Therefore, the set of texts for the input of the autoencoder is composed of six different languages per do- , where S is the amount of samples in the specific domain and L = {EN, DE, RU, T R, HR, SQ}. Then, we use three datasets, one per domain. The size of the dataset in Gao is 594 (6*99), in Trac is 1800 (6*300), and in Wul is 3600 (6*600). For comparison we use mBERT and XLM-R as in the multi-domain evaluation. Table 3 illustrates the results of the multilingual evaluation. We observe that mBERT and XLM-R outperform the classifier that uses our embeddings. In fact, the results obtained with our framework decrease considerably. This makes sense, since no knowledge about the difference of the languages has been added in the learning of our embeddings. Whereas mBERT and XLM-R have been trained with large data collections that include the languages of the datasets. For this reason, we use a strategy to incorporate language knowledge in the input of our graph autoencoder with the embeddings from the Universal Sentences Encoder (USE) 3 (Cer et al., 2018). The results of this new variant (GAE-USE) are also shown in Table 3. In this way, the use of the embeddings once again improves the results obtained with mBERT and XLM-R.

Domain
Gao In this case, instead of using TFIDF, we use the multilingual model of the universal sentences encoder (USE) 4 . This model was trained on several data sources and tasks to dynamically adapt a wide variety of natural language understanding tasks. The input is a text of variable length and the output is a 512 dimensional vector. Figure 5 illustrates the representation of the initial vectors on the left and the latent representation (embeddings) on the right. This only corresponds to the case of Trac, but we observed similar behaviour in the other two domains. The two first Figures 5(a) and 5(b) show the representation when we add knowledge about languages with USE. In the initial representation we observe six groups (marked with red circles) that suggest the six different languages that the input contains. In the latent representation we observe a division among classes, similar to the ones obtained in the monolingual evaluation. Therefore, it seems that our framework can be useful to deal with multilingual datasets by adding prior knowledge of the languages. On the other hand, Figures 5(c) and 5(d) show the representations obtained by using TFIDF. That is, the case where no knowledge about languages is added. An interesting phenomenon in the embeddings generated with our encoder is that separated groups of points with both types of points (red and green) can be identified. It seems that the encoder has learned the difference among languages instead of the difference between the classes hate and not-hate. In this sense we note that this representation is somewhat similar to the initial representation obtained with USE. Therefore, the embeddings learned with our framework could improve by adding more layers in the encoder. Therefore, we attempt to adapt the proposal in (Li et al., 2019) as future work. In that paper, the authors present a ways to successfully train very deep convolutional graph neural network.

Conclusion and Future Work
In this work, we proposed a graph auto-encoder framework to learn embeddings of a set of texts in an unsupervised way. The auto-encoder receives an initial vector representation of the texts and the relation among them in the form of a graph, to generate a lowdimensional representation. Then, the embeddings are extracted from this latent representation. In this sense we built the encoder with a sequence of a transformer layer and a convolutional layer to consider the information of the graph structure in the learning of the embeddings. We used this framework for hate speech detection by using the embeddings as input of a classifier. In the evaluation we considered multi-domain and multilingual settings with small datasets. We observed promising results by outperforming mBERT and XLM-R, one of the state-of-the-art models in hate speech detection. We noticed that the improvement by using the embeddings generated with our auto-encoder became more notable in small data, suggesting the suitability of our proposal to deal with data-poor settings. Moreover, we observed that the use of our embeddings was more stable in multi-domain settings. While in the case of multilingual settings, we had to add prior knowledge about languages to avoid a decrease in the performance. As future work, we will extend our analysis by building a deeper encoder. The idea is to investigate if it is possible to learn the embeddings for a multilingual setting without the need of prior knowledge about the languages. Moreover, we will adapt our graph autoencoder to encode not only text, but also visual information. Thus, we will be able to deal with multimodal hate speech detection.