{'fraction_non_alphanumeric': 0.02959463824289406, 'fraction_numerical': 0.015746124031007752, 'mean_word_length': 4.745534678728833, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
On the Importance of Tokenization in Arabic Embedding Models

Arabic, like other highly inflected languages, encodes a large amount of information in its morphology and word structure. In this work, we propose two embedding strategies that modify the tokenization phase of traditional word embedding models (Word2Vec) and contextual word embedding models (BERT) to take into account Arabic’s relatively complex morphology. In Word2Vec, we segment words into subwords during training time and then compose word-level representations from the subwords during test time. We train our embeddings on Arabic Wikipedia and show that they perform better than a Word2Vec model on multiple Arabic natural language processing datasets while being approximately 60% smaller in size. Moreover, we showcase our embeddings’ ability to produce accurate representations of some out-of-vocabulary words that were not encountered before. In BERT, we modify the tokenization layer of Google’s pretrained multilingual BERT model by incorporating information on morphology. By doing so, we achieve state of the art performance on two Arabic NLP datasets without pretraining.


Introduction
Word embeddings are one of the main building blocks of most natural language processing tasks. Many word embedding techniques have been proposed (Mikolov et al., 2013b;Pennington et al., 2014;) that try to capture better word representations. Although most of the approaches are language agnostic, they were historically designed to be used with English. Nonetheless, it was shown that these techniques work well in other languages (Grave et al., 2018;Soliman et al., 2017).
One feature that is unique to Arabic, and other highly inflected languages, is the expressiveness of its words. The fact that Arabic encodes a large amount of information in its word structure leads to potential problems in learning embeddings due to the large number of forms for each word, the more likely chances of out-of-vocabulary (OOV) instances, and the increase in model size.
In this work, we propose two embedding strategies for Arabic that take into consideration its rich morphology by modifying the tokenization phase. The first technique concerns traditional embedding models and the second a contextual one. Our experiments are done on Word2Vec (Mikolov et al., 2013a) and BERT (Devlin et al., 2018) since they are the most popular traditional and contextual embedding techniques, respectively. Nonetheless, the approaches we propose are embedding-agnostic and can be applied to other embedding techniques. Figure 1 summarizes our two approaches and illustrates what happens in the training and inferences stages of each approach.
In traditional embeddings, we analyze the effect of tokenizing words into subwords by splitting their suffixes and prefixes before training an embedding model and then combining these subwords using an algorithm we propose. We show that by doing so we get: 1. Better performance: our model outperforms Word2Vec in multiple tasks.
2. Smaller size: our model is 59.6% smaller than a Word2Vec model trained on the same corpus. 3. Superior out-of-vocabulary handling: our model is able to handle some OOV instances unlike Word2Vec.
In contextual embeddings, we investigate different tokenization schemes when using BERT without needing any pretraining, in contrast to previous approaches such as Antoun et al. (2020). We simply modify the tokenization part of Google's pretrained multilingual BERT model (Devlin et al., 2018) resulting in models that: 1. Achieve state of the art results on two Arabic NLP datasets.
2. Do not require pretraining and can work on top of existing models.
The rest of the paper is structured as follows: Section 2 introduces the background and explains some of the fundamental ideas of word embeddings; Section 3 defines the process of gathering and cleaning our data; Section 4 highlights the embedding approaches we are proposing; Section 5 details the experiments and the results; Section 6 discusses related work; and Section 7 concludes by summarizing our findings and pointing to potential directions for future work.

Background
Word embedding techniques rely on the distributional hypothesis (Harris, 1954) which suggests that words that appear in similar contexts tend to have similar meanings. Mikolov et al. (2013a) popularized word embeddings when they introduced Word2Vec and showed that it produces representations that capture not only syntax but also words' semantics. Many related techniques have been produced after that (Mikolov et al., 2013b;Pennington et al., 2014;. The word vectors produced by such techniques capture interesting semantics; one popular example is how the the vectors capture relationships between them. For example, if we subtract the vector for 'man' from the vector of 'king', and then add the vector of 'woman' we get very close to the vector of 'queen'. Contextual word embeddings were proposed (Devlin et al., 2018;Peters et al., 2018;Liu et al., 2019) to tackle the issues that arise from words have multiple senses and meanings. In traditional embeddings such as Word2Vec, each word is encoded in a vector, which is a fixed representation. This may cause problems with homographs and words that have multiple senses depending on the context. For example the wear 'bear' means two very different things in the following sentences: "The right of the people to keep and bear arms shall not be infringed." and "A wild bear was seen in the city." Yet, traditional embeddings will only capture one fixed representation. Contextual word embeddings aim to solve this issue by modeling embeddings where the context of the word will affect its generated representation.

Data
In this section, we describe the process of gathering and preparing the data used for training the embeddings. We do not require any training data for contextual embeddings since we do not require pretraining: we use Google's multilingual BERT model (Devlin et al., 2018), which supports 104 languages (including Arabic) and was trained on their respective Wikipedia dumps. For traditional embeddings, we use Arabic Wikipedia as a corpus. We downloaded the Wikipedia dump from January 2018 and then cleaned it by using WikiExtractor 1 , which is a utility that generates plain text from an XML formatted Wikipedia dump. We then use a custom set of regexes to filter out all non-Arabic words, such as English words and numbers, and remove all diacritics and kashidas resulting in over 86 million tokens.

Approach
One example of Arabic's morphological complexity is in the number of verb forms it possesses which is much higher than in English as we can see in Table 1. While traditionally, this aspect of Arabic has been challenging to the natural language processing and computational linguistics communities (Farghaly and Shaalan, 2009;Al-Ayyoub et al., 2018), we asked whether we may benefit from this characteristic. Can we tokenize text differently for Arabic than we do for English, and would that result in better performance on NLP tasks? We experimented with two approaches; one applied to traditional embedding models (Word2Vec) and the other on contextual models (BERT).

Traditional Embedding Models
Traditional word embedding models are trained on a large corpus of text. The main difference in our approach is that we preprocess the text before feeding it into the embedding algorithm by splitting every word into subwords, which are its prefix(es), stem, and suffix(es) using Farasa, an Arabic segmenter, ( Abdelali et al., 2016). The effect of using the Farasa segmenter can be seen in Figure 2 2 , where each row of squares represents a vector. Then we train the resulting corpus using Word2Vec, though we note that our approach is embedding agnostic and may be used with any embedding model. Notice that our vocabulary, and the resulting vectors, will be completely different now as shown in Figure 2. It may seem at first glance that our vocabulary is increasing, but in fact it decreases as we keep adding more words as shown in Figure 3, which depicts the sizes of the vocabularies in the first million words in Arabic Wikipedia.
One question that comes to mind is how do we generate embeddings for words that were split into subwords because in most cases we want embeddings at the word level and not at the subword level. We propose the following technique for getting the embeddings of all types of words, including those with multiple subwords. For words with only one component, we just return the embeddings learnt by the model. For words with multiple components (subwords), we get the embedding of the longest subword, and the average of the embeddings of the remaining subwords. We then take a weighted average of the two values which results in our embedding as Equations 1 and 2 show.
where S is the set containing all the subwords contained in the word, M is the learnt model that contains embeddings for all subwords, and α is a parameter that decides the weights between the longest subword and the other subwords. Algorithm 1 illustrates the algorithm used in determining embeddings for all cases. This method not only allows us to generate embeddings for all words in the original corpus, but also increases its capacity to deal with out-of-vocabulary (OOV) words that the model has never seen before as shown in Figure 4. For example, in Figure 4, we see how our model can produce a representation of 'and their iPhone' which is one word in Arabic. A traditional model trained on the Arabic Wikipedia will fail to produce a representation of the word 'and their iPhone' because that word never appeared in Wikipedia. In fact, since there are many forms for each word, no matter how large the training corpus is, it is almost impossible for it to have seen occurrences of all possible forms of all words in it. Our model can tackle this problem because it operates on a subword level and has seen all the three components that make up the word: 'and', 'their' and 'iPhone' as shown in Figure 4. This procedure allows one to expand a model's vocabulary without retraining or requiring numerous examples of a given word. Of Algorithm 1 Generating embeddings of words from subwords course, not all out-of-vocabulary words will be found this way. Nonetheless, it's a cheap way to generate representations of new words that is not possible in classical approaches.

Contextual Embedding Models
Transformer-based (Vaswani et al., 2017) contextual embedding models, such as BERT (Devlin et al., 2018), often require a tokenization step that solves problems such as the out-of-vocabulary issue. Bytepair encoding (BPE) (Sennrich et al., 2015;Gage, 1994), one of the most popular tokenization methods relies on segmenting each word into the most frequent subwords. Shapiro and Duh (2018a) have shown that byte-pair encoding does not perform well for Arabic compared to other languages. One possible explanation of this is that byte-pair encoding does not include information derived from a given language's morphology. We did some experiments using BERT's pretrained tokenizer and found instances where the produced segments generated erroneous meanings. For example, the word mal\ab in most cases is a noun that refers to a stadium or field. BERT tokenizes mal\ab by segmenting it to mal (milliliter) and \ab (gulp or fill up), instead of the correct segmentation: ma (a prefix used to create nouns of place) and l\ab (play). BERT's segmentation seems to indicate that the word mal\ab is related to liquids and water (milliliter/gulp/fill up). Keep in mind that both ma and l\ab are in BERT's subword vocabulary.
We propose two methods that aim to incorporate a language's structure via better segmentations, which  we call MorphBERT (Morphology BERT) and CharBERT (Character BERT). In MorphBERT, a custom tokenizer is used to replace the default tokenizer layer as seen in Figure 5. That custom tokenizer will use a language specific segmenter, Farasa (Abdelali et al., 2016) in our case, to segment each word before processing it. We then pass each word, after segmentation, to the original model's tokenizer to make sure that the produced segments are in the model's vocabulary. Keep in mind that MorphBERT differs from AraBERTv1 (Antoun et al., 2020), an Arabic BERT model that also utilizes Farasa, in that it does not require pretraining. In addition to that, Antoun et al. (2020) preprocess the training corpus by segmenting it using Farasa before training AraBERTv1 which is not the case with MorphBERT. In CharBERT, we segment everything to characters as shown in Figure 5. The main idea behind CharBERT is to let the network learn these language structures on its own. Both of these models do not require training and can be used with any pretrained model as we see in Figure 5. This is important due to the expensive -money-wise, time-wise, and environment-wise -process of training BERT and other state of the art models. We believe that developing simpler, more sustainable, and more efficient NLP models is of an utmost importance due to the many problems that arise from computationally heavy models, (Strubell et al., 2019) which can take weeks to train on many TPUs/GPUs. Moreover, most people, especially in less developed countries, do not have the resources to train these models, which limits accessibility.

Experiments and Results
In this section we detail our experiments and highlight the results produced by the models we proposed.

Evaluation Datasets
For intrinsic evaluation of traditional embedding models we used the Arabic word analogy benchmark created by Elrazzaz et al. (2017). This dataset consists of nine relations that each consist of over 100 word pairs. We use the following datasets to evaluate our models extrinsically: 1. APMD: The Arab poem meters dataset (Alyafeai, 2020) which consists of 55,440 poem verses with each verse classified into one of the fourteen Arabic poetry meters. The data is split into training and testing sets.  2. HARD: The Hotel Arabic Reviews Dataset (Elnagar et al., 2018) consists of 93,700 hotel reviews that are classified into positive or negative according to their rating. Reviews with a rating of four or five were assigned positive, and those with a rating of one or two were labeled negative. Reviews with a rating of three were ignored. We split the data into 80% and 20% training and testing sets respectively using the script provided by Antoun et al. (2020).

LABR:
The Large-scale Arabic Book Reviews dataset (Aly and Atiya, 2013) consists of 63,000 book reviews rated between one and five. We use the unbalanced two-class dataset, where reviews with a rating of one or two are labeled negative, and those with a rating on four or five are labeled positive. The data is split into training and testing sets.

Intrinsic evaluation
Word analogies, which consists of sets of pairs that share a common relationship, are often used to evaluate different embedding techniques. For example, let's say that we have the following pairs: (king, queen), and (male, female). Both of these pairs contain a word that indicates masculinity and a second word that indicates the feminine version of the first word. A perfect word embedding representation should be able to capture this relationship, and the way to mathematically measure it is by calculating the vector king−male+female. After that, we check to see whether the resulting vector is the closest to the vector queen or not. We use the Arabic word analogy benchmark created by Elrazzaz et al. (2017) to evaluate our approach. We used Gensim's (Řehůřek and Sojka, 2010) word analogy evaluate function to evaluate the models and set the 'dummy4unknown' flag on so that all tuples of pairs that contain a word (or more) that are not in our vocabulary will get zero accuracy (instead of being skipped), similar to the procedure adopted by Elrazzaz et al. (2017). We train two models: a vanilla Word2Vec model (base model) and our proposed model on the Arabic Wikipedia dataset mentioned in Section 3. For the base model, we set the window size to be five. Since our model segments words before learning embeddings, we compute the average number of subwords per word in our corpus and adjust the window size by that number. The average number of components per word is around two (1.97); to account for that we set window size in our model to be be ten instead of five. For both, we set the number of epochs to be equal to ten. We experimented with multiple α values for our proposed model and found that setting it to 0.3 achieves the best results. Moreover, to avoid vocabulary size discrepancies, we standardize the vocabulary size before the evaluation by ensuring that our model has the same vocabulary as the base model. To do that, we create a new empty set and go through all vocabulary in the base model and if the word exists in our model (words with only one subword) then we add its representation to the set; otherwise, we decompose it and then add its representation according to equation 2. The results are summarized in Table 2. Our method performs as well, if not better, than the Word2Vec while being around 60% smaller in size. The difference in size come from the vocabulary size which is 155K for our model compared to 383K for the base model. We also notice that CBOW performs better than Skip-gram which is consistent with previous research findings (Elrazzaz et al., 2017).

OOV Handling
One important distinction that separates our model from the base model is its ability to accommodate OOV words. To test the quality of these generated OOV vectors, we go through all the OOV words in our analogy benchmark and generate vectors for the ones we can, i.e. the ones for which we have entries Accuracy (without OOV handling) Accuracy (with OOV handling) FastText 8.05% 6.44% Our model 10.10% 13.32%  for all their respective subwords. The total number of OOV words is 127 and we are able to generate representations for 70 of them covering 55.12% of all the OOV words. After that we follow a similar procedure to the one we have done in the previous subsection and check how the accuracy has been affected. We evaluated the performance of our best performing model (CBOW, dim=200) and found that adding the OOV representations increased the accuracy from 10.10% to 13.32%. To compare our OOV representations, we trained a fastText  model, a popular embedding approach that can handle OOV, and then calculated the accuracy before adding the OOV entities and after; keep in mind that the vocabulary size in both, fastText and our model, will be the same. FastText achieves 8.05% before adding the OOV entities and 6.44% after adding them as shown in Table 3. Not only did fastText achieve worse gains than our model, it actually performs worse than before which calls to question the accuracy of fastText's OOV embeddings for Arabic.

Extrinsic Evaluation
We evaluate our approach on the three datasets mentioned above. We feed the embeddings of the words to a bidirectional Gated Recurrent Units (GRU) (Cho et al., 2014) network to train it. After that, we evaluate the network on the test set. Table 4 shows the performance of our model compared to the base model. We can see that our model clearly outperforms the base model in one dataset (APMD) and performs slightly better on the two other datasets (LABR and HARD).

Contextual Embedding Models
We fine-tune the three models: the base cased multilingual BERT, MorphBERT, and CharBERT; and then compare their performances on the three datasets mentioned above. We follow the recommendations from BERT's paper (Devlin et al., 2018) in setting the fine-tuning hyperparameters. We run them all for four epochs in batches of 32 or 16 depending on the lengths of input sequences to avoid memory issues on the GPU. We optimize using the Adam algorithm with a learning rate of 2e −5 , β1 = 0.9, and β2 = 0.999. We also compare our models to AraBERT (AraBERTv0.1 and AraBERTv1) which consists of two monolingual BERT models trained on Arabic that were proposed by Antoun et al. (2020) and use the results reported by them for LABR and HARD. For APMD, we downloaded their models and fine-tuned them on the task. We also evaluate our models on the cleaned ANERcorp (Benajiba and Rosso, 2007;Antoun et al., 2020) NER dataset. For ANERcorp, we used the script provided by Antoun et al. (2020) in fine-tuning our models. Table 5 shows the performance of the five models on the downstream tasks.
It is interesting that by just changing the tokenization method we can improve BERT's performance in Arabic without retraining. As we can see in Table 5, MorphBERT and CharBERT achieve state of the art performance on LABR and APMD respectively. The previous state of the art model for LABR is the MCE-CNN model proposed by Dahou et al. (2019) which achieves an accuracy of 87.48%. Our models perform better than AraBERT in two tasks even though AraBERT was: 1) trained specifically for Arabic, and 2) trained on a larger Arabic corpus: 24GB of data for AraBERT compared to 4.3GB for the multilingual BERT. Although MultiBERT was trained on over a hundred languages, simply replacing tokenizations allowed us to add language specific information without requiring any training. While normally byte pair encoding learns representations of subwords without paying attention to their  meaning, we can utilize this procedure of breaking words into chunks that make more sense as we saw in the mal\ab example before. CharBERT in particular is interesting; one would expect that it will require more time to fine-tune since it only uses characters. Nevertheless, it achieves great performance without requiring more epochs than the other methods. One potential issue with CharBERT is that it results in very long sequences due to the character segmentation approach it follows which lead to more frequent truncations than other models. One potential way to mitigate this is by using new models such as Longformer (Beltagy et al., 2020) that allow longer sequences than BERT. Previous research (Virtanen et al., 2019;Antoun et al., 2020;Vries et al., 2019;Martin et al., 2020) has shown that a language specific BERT model performs better than a multilingual one. This is the first work, according to our knowledge, that shows that by tweaking a multilingual BERT model one can beat a BERT model trained on a specific language. Natural language processing entered a new era with the advent of pretrained models that do not need to be trained from scratch for every task but can simply be tweaked/fine-tuned instead. Our results shows that it may be possible to only have one multilingual model that can be tweaked instead of learning a pretrained model for every language.

Related Work
Many works have noted how sensitivity to Arabic's morphological complexity can result in better performance in standard NLP tasks. However, we do not know of any previous work that has specifically focused on the effect of tokenization on different Arabic embedding models. Antoun et al. (2020) trained an Arabic specific BERT model. They also trained another Arabic BERT model in which they segment the text before training the model and showed that it usually improves performance. Taylor and Brychcín (2018) analyzed morphological relations in Arabic word embeddings. They noted that some morphological features are captured in embeddings representations. Shapiro and Duh (2018b) proposed utilizing subword information in training embeddings to enrich the representations and showed that it improves the performance on word similarity tasks. Salama et al. (2018) investigated morphological-based embeddings and lemma-based embeddings. They utilized part-of-speech information to train their embeddings, similar to Trask et al. (2015), and then build lemma-based embeddings from them by aggregating on different senses of each word first and them combining words that share the same lemma. El-Kishky et al. (2019)

Conclusion
We show that tokenization that pays attention to Arabic's morphology can create better traditional and contextual embedding models. Breaking words into subwords in Word2Vec not only leads to an increase in performance and reduction in the vocabulary size and hence the model size, but also provides a simple way to produce good out-of-vocabulary representations. We also show the importance of tokenization in BERT where we were able to achieve impressive performance without requiring any pretraining. One possible future work would be to investigate tokenization's effect in other morphologically rich languages such as Hebrew and Turkish and see if our results can be generalized to other highly inflected languages.