{'fraction_non_alphanumeric': 0.03435399551904406, 'fraction_numerical': 0.01755041075429425, 'mean_word_length': 4.747019551740582, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
DiaLex: A Benchmark for Evaluating Multidialectal Arabic Word Embeddings

Word embeddings are a core component of modern natural language processing systems, making the ability to thoroughly evaluate them a vital task. We describe DiaLex, a benchmark for intrinsic evaluation of dialectal Arabic word embedding. DiaLex covers five important Arabic dialects: Algerian, Egyptian, Lebanese, Syrian, and Tunisian. Across these dialects, DiaLex provides a testbank for six syntactic and semantic relations, namely male to female, singular to dual, singular to plural, antonym, comparative, and genitive to past tense. DiaLex thus consists of a collection of word pairs representing each of the six relations in each of the five dialects. To demonstrate the utility of DiaLex, we use it to evaluate a set of existing and new Arabic word embeddings that we developed. Our benchmark, evaluation code, and new word embedding models will be publicly available.


Introduction
Word embeddings are the backbone of modern natural language processing (NLP) systems. They encode semantic and syntactic relations between words by representing them in a low-dimensional space. Many techniques have been proposed to learn such embeddings (Pennington et al., 2014;Mikolov et al., 2013a;Mnih and Kavukcuoglu, 2013) from large text corpora. As of today, a large number of such embeddings are available in many languages including Arabic. Due to their importance, it is vital to be able to evaluate word embeddings, and various methods have been proposed for evaluating them. These methods can be broadly categorized into intrinsic evaluation methods and extrinsic evaluation ones. For extrinsic evaluation, word embeddings are assessed based on performance in downstream applications. For intrinsic evaluation, they are assessed based on how well they capture syntactic and semantic relations between words.
Although there exists a benchmark for evaluating modern standard Arabic (MSA) word embeddings (Elrazzaz et al., 2017), no such resource that we know of exists for Arabic dialects. This makes it difficult to measure progress on Arabic dialect processing. In this paper, our goal is to facilitate intrinsic evaluation of dialectal Arabic word embeddings. To this end, we build a new benchmark spanning five different Arabic dialects, from Eastern, Middle, and Western Arab World. Namely, our benchmark covers Algerian (ALG), Egyptian (EGY), Lebanese (LEB), Syrian (SYR), and Tunisian (TUN). Figure  1 shows a map of the five Arab countries covered by DiaLex. For each one of these dialects, DiaLex consists of a set of word pairs that are syntactically or semantically related by one of six different relations: Male to Female, Singular to Plural, Singular to Dual, Antonym, Comparative, and Genitive to Past Tense. Overall, DiaLex consists of over 3,000 word pairs in those five dialects, evenly distributed. To the best of our knowledge, DiaLex is the first benchmark that can be used to assess the quality of Arabic word embeddings in the five dialects it covers.
To be able to use DiaLex to evaluate Arabic word embeddings, we generate a set of word analogy questions from the word pairs in DiaLex. A word analogy question is generated from two word pairs from a given relation. These questions have recently become the standard in intrinsic evaluation of word embeddings (Mikolov et al., 2013a;Gao et al., 2014;Schnabel et al., 2015). To demonstrate the usefulness of DiaLex in evaluating Arabic word embeddings, we use it to evaluate a set of existing and new Arabic word embeddings. We conclude that both available and newly-developed word embedding models have moderate-to-serious coverage issues and are not sufficiently representative of the respective dialects under study. In addition to the benchmark of word pairs, our newly-developed dialectal Arabic word embeddings will also be publicly available. 2 The rest of the paper is organized as follows. In Section 2, we describe how DiaLex was constructed. Section 3 offers our methodical generation of a testbank for evaluating word embeddings. In Section 4, we provide a case study for evaluating various word embedding models, some of which are newly developed by us. Section 5 is about related work. Section 6 is where we conclude and present future directions.

Benchmark Construction
DiaLex consists of a set of word pairs in five different Arabic dialects and for six different semantic and syntactic relations, namely Male to Female, Singular to Plural, Singular to Dual, Antonym, Comparative, and Genitive to Past Tense. We chose only this set of relations as they are standard in previous literature. In addition, they are comprehensive enough to reflect dialect specificity. A good word embeddings model should thus have close representation of word pairs for each of these relations in the embeddings space.
For each dialect, word pairs were manually generated by at least one native speaker of the dialect. Each person independently came up with the word pairs representing a given relation based on their knowledge of the dialect and while trying to include words that are typically representative and unique in that dialect. That is, to the best of our ability, the words were chosen so that they are frequently-used words in the dialect and are not the same as in MSA. One challenge we faced when generating the word pairs was orthographic variation. For example, consider the antonym of the word ("behind") in the Egyptian dialect. It can be written as or ("in front of") . We decided to include all variations of the same word in the benchmark. A second challenge was the need to present the relationship using more than one word. For instance, consider Algerian and the relationship dual, the word ("pair") is sometimes used to describe two items of something. So, for instance, for the word ("hour"), the dual can be either ("pair of hours") or ("two hours"). Again, we opted for including both variants in the benchmark. Overall, for each dialect and each relation, around 100 word pairs were generated. Table 1 shows the statistics of DiaLex's word pairs lists and Table 2 shows some example word pairs in DiaLex and their English and MSA translations. Overall, DiaLex consists of a total of 3, 070 word pairs distributed evenly among the dialects and the relations.

Testbank for Evaluating Word Embeddings and Evaluation Metric
Given the word pair lists in DiaLex, we generate a testbank consisting of 260,827 tuples. Each tuple consists of two word pairs (a, b) and (c, d) from the same relation and the same dialect. For each of Dialect antonym compara genitive-pt male-fem sing-pl sing-dual all our five dialects and for each of our six relations, we generate a tuple by combining two different word pairs from the same relation in the same dialect. Once tuples have been generated, they can be used as word analogy questions to evaluate different word embeddings as defined by Mikolov et al. (Mikolov et al., 2013a). A word analogy question for a tuple consisting of two word pairs (a, b) and (c, d) can be formulated as follows: "a to b is like c to ?". Each such question will then be answered by calculating a target vector t = b − a + c. We then calculate the cosine similarity between the target vector t and the vector representation of each word w in a given word embeddings V . Finally, we retrieve the most similar word w to t, i.e., argmax w∈V &w / ∈{a,b,c} w·t ||w||||t|| . If w = d (i.e., the same word) then we assume that the word embeddings V has answered the question correctly.
Moreover, we extend the traditional word analogy task by taking into consideration if the correct answer is among the top K, with K ∈ {5, 10}, closest words in the embedding space to the target vector t, which allows us to more leniently evaluate the embeddings. This is particularly important in the case of Arabic since many forms of the same word exist, usually with additional prefixes or suffixes such as the equivalent of the article "the" or possessive determiners such as "her", "his", or "their". For example, consider one question which asks to is like to " ?", i.e., "man" to "woman" is like "prince" to "?", with the answer being " " or "princess". Now, if we rely only on the top-1 word and it happens to be " " which means "for the princess" in English, the question would be considered to be answered wrongly. To relax this, and ensure that different forms of the same word will not result in a mismatch, we use the top-5 and top-10 words for evaluation rather than just the top-1.
Note that we consider a question to be answered wrongly if at least one of the words in the question are not present in the word embeddings. That is, we take into consideration the coverage of the embeddings as well (Gao et al., 2014).
Finally, we report the number of questions that were answered correctly over the total numbers of questions available. That is, assume the number of questions for a given dialect and a given relation is n, and assume that a given embeddings model M correctly answered m out of those n questions as explained above. Then, the accuracy of the model M will be m n .

Evaluation of Arabic Word Embeddings Using DiaLex
In this section, we demonstrate how DiaLex can be used to evaluate word embeddings across the different dialects it covers. Particularly, we evaluate two large word embeddings models based on Word2Vec (Mikolov et al., 2013b) released by Zahran et al. (2015). One model is based on skip grams (Zah SG) and the other is a continuous bag-of-words (Zah CBOW). Both of these models have a vocabulary size of 626, 3435 words. We also create four CBOW Word2Vec models, all of which have 300 word vector embedding dimensions, as we describe next.

Newly-Developed Word Embedding Models
The following are our four newly-developed dialectal Arabic word embedding models: Twitter-1B. Our first model was trained using a one billion in-house Arabic tweet collection. All tweets were crawled by putting a bounding box crawler around the whole Arab world. Since this collection Dialect male-fem sing-pl sing-dual rooster -chicken sandwich -two sandwiches shoe -pair of shoes unlucky -lucky poor -poorer slap -slapping  Twitter-1B. Our first model was trained using a one billion in-house Arabic tweet collection. All tweets were crawled by putting a bounding box crawler around the whole Arab world. Since this collection is large, we only performed light pre-processing on it. This involved removing hashtags, URLs, and reducing consecutive repetitions of the same character into only 2. We then trained a CBOW Word2Vec model using the Python library gensim. We set the minimum word frequency at 100 and a window size of 5 words. This model has a vocabulary size of 929, 803 words.
Twitter-250K-MC50. This model uses the same data as Twitter-1B, yet with stricter preprocessing. Namely, we normalize usually orthographically confused Arabic characters by converting Alif maksura to Ya, reducing all hamzated Alif to plain Alif, and removing all non-Arabic characters. We then only keep tweets with length ≥ 5 words. This gives us a total of 223, 387, 189 tweets (and hence the name Twitter-250K). We then use the same parameters as the 1B model, but we set the minimum count to 50 words (again, hence the name Twitter-250K-MC50 where MC50 meaning minimum count of 50). We acquire a model with a vocabulary size of 536, 846 words. Twitter-250K-MC100. This model is identical with the Twitter-250K-MC50 model, but uses a minimum count of 100 words when training Word2Vec. This model has a vocabulary size of 202, 690 words.
Twitter-Seeds. We use all unigram entries in our benchmark to crawl Twitter, using the search API. In order to avoid overfitting to our benchmark, we randomly sample only 400 words from it for this process. This gives us about ∼ 3M tweets. We then crawl up to 3, 200 tweets from the timelines of all ∼ 300K users who have posted the initially collected 3M tweets. The resulting collection is at 214, 161, 138 tweets after strict cleaning and removal of all tweets of length < 5 words. To train an embedding model on this dataset, we use the same Word2Vec parameters as with Twitter-250K-MC50 (i.e., with a minimum count of 50 words). This model has a vocabulary size of 773, 311 words.

Model Evaluation
We evaluate the two models from Zahran et al. (2015) and our four models described above using DiaLex.  Table 4: Evaluation across all relations for our Twitter-MC-100 model. This is the model that tends to perform best across most dialects. Values shown as "None" are for relationships where the model did not include any of the word pairs in the question tuples in the model vocabulary. Zeros mean the model includes the words in its vocabulary but no correct answers were returned in the top-K. Table 4 shows per-relation results evaluation of the Twitter-250K-MC100 model. We show allrelations results only for this model for space limitations, and we choose this model since it tends to perform well compared to other models. As Table 4 shows, the model works best on the LEB dialect (17.87 accuracy for top-10) and worst on TUN (12.14 accuracy for top-10). The table also shows that the sing-plural relationship is the one most challenging for the model. Clearly, the dual feature either involves (1) bigrams (in which case these unigram models do not work and hence the "None" values) or (2) different ways of expressing the same meaning of duality. For example, in the LEB pair -"corridor-two corridors", duality can be expressed also by the phrase , using the digit "2" instead of the dual suffix in . Overall, these results suggest that even our developed models are neither sufficiently powerful nor large enough (even with large vocabularies close to 1M words) to cover all the dialects. We also observe that models need to see enough contexts (words > 50 minimum count) to scale well. This calls for the development of more robust models with wider coverage and more frequent contexts. Our Twitter collection of 5M words can be used towards that goal, but we opted for not exploiting it for building a word embeddings model since this would be considered overfitting to our benchmark.

Arabic Word Embeddings
The number of available Arabic word embeddings is increasing rapidly. Some of these are strictly trained using textual corpora written in MSA, while others were trained using dialectal data. We review the most popular embedding models we are aware of here. Zahran et al. (2015) built three models for Arabic word embeddings (CBOW, SKIP-G, and GloVe). To train these models, they used a large collection of MSA texts totaling ∼ 5.8B words. The sources used include Arabic Wikipedia, Arabic Gigaword (Parker et al., 2009), Open Source Arabic Corpora (OSAC) (Saad and Ashour, 2010), OPUS (Tiedemann, 2012), MultiUN (Chen and Eisele, 2012), and a few others. Soliman et al. (2017) proposed AraVec a set of Arabic word embedding models. It consist of six word embedding models built on top of three different Arabic content domains; Wikipedia Arabic, World Wide Web pages, and Tweets with more than 3.3 billion word. Both CBOW and SKIP-Gram architecture are investigated in this work. Abdul-Mageed et al. (2018) build an SKIP-G model using ∼ 234M tweets, with vector dimensions at 300. The authors, however, do not exploit their model in downstream tasks. Abu Farha and Magdy (2019) built two word embeddings models (CBOW and SKIP-Gram) exploiting 250M tweets. The authors used the models in the context of training the sentiment analysis system Mazajak. The dimensions of each embedding vector in the Mazajak models are at 300.
More recently, El-Haj (2020) developed Habibi, a Multi Dialect Multi-National Arabic Song Lyrics Corpus which comprises more than 30, 000 Arabic song lyrics from 18 Arab countries and six Arabic dialects for singers. Habibi contains 500, 000 sentences (song verses) with more than 3.5 million words. Moreover, the authors provided a 300 dimension CBOW word embeddings of the corpus. Doughman et al. (2020) built a set of word embeddings learnt from three large Lebanese news archives, which collectively consist of 609, 386 scanned newspaper images and spanning a total of 151 years, ranging from 1933 till 2011. To train the word embeddings, Optical Character Recognition (OCR) was employed to transcribe the scanned news archives, and various archive-level as well as decade-level word embeddings were trained. In addition, models were also built using a mixture of Arabic and English data. For example, Lachraf et al. (2019) presented AraEngVec an Arabic-English cross-lingual word embedding models. To train their bilingual models, they used a large dataset with more than 93 million pairs of Arabic-English parallel sentences (with more than 1.8 billion words) mainly extracted from the Open Parallel Corpus Project (OPUS) (Tiedemann, 2012). In order to train the models, they have chosen CBOW and SKIP-Gram as an architecture. Indeed, they propose three methods for pre-processing the opus dataset: parallel sentences, word-level alignment and random shuffling. Both extrinsic and intrinsic evaluations for the different AraEngVec model variants. The extrinsic evaluation assesses the performance of models on the Arabic-English Cross-Language Semantic Textual Similarity (CL-STS) task (Nagoudi et al., 2018), while the intrinsic evaluation is based on the Word Translation (WT) task.
Some works have also investigated the utility of using morphological knowledge to enhance word embeddings. For example, Erdmann and Habash (2018) demonstrated that out-of-context rule-based knowledge of morphological structure can complement what word embeddings can learn about morphology from words' in-context behaviors. They quantified the value of leveraging subword information when learning embeddings and the further value of noise reduction techniques targeting the sparsity caused by complex morphology such as in the Arabic language case. El-Kishky et al. (2019) tackled the problem of root extraction from words in the Semitic language family. They proposed a constrained sequence-to-sequence root extraction method. Furthermore, they demonstrated how one can leverage the root information alongside a simple slot-based morphological decomposition to improve upon word embedding representations as evaluated through word similarity, word analogy, and language modeling tasks. In this paper, we have demonstrated the effectiveness of our benchmark DiaLex in the evaluation of a chosen set of these available word embedding models and compared them to newly-developed ones by us as we explain in the next section.

Word Embeddings Evaluation
There is a wealth of research on evaluating unsupervised word embeddings, which can be broadly divided into intrinsic and extrinsic evaluations. Intrinsic evaluations mostly rely on word analogy questions and measure the similarity of words in the low-dimensional embedding space (Mikolov et al., 2013a;Gao et al., 2014;Schnabel et al., 2015). Extrinsic evaluations assess the quality of the embeddings as features in models for other tasks, such as semantic role labeling and part-of-speech tagging (Collobert et al., 2011), or noun-phrase chunking and sentiment analysis (Schnabel et al., 2015). However, all of these tasks and benchmarks are built for English and thus cannot be used to assess the quality of Arabic word embeddings, which is the main focus here.
To the best of our knowledge, only a handful of recent studies attempted evaluating Arabic word embeddings. Zahran et al. (2015) translated the English benchmark in (Mikolov et al., 2013a) and used it to evaluate different embedding techniques when applied on a large Arabic corpus. However, as the authors themselves point out, translating an English benchmark is not the best strategy to evaluate Arabic embeddings. Zahran et al. (2015) also consider extrinsic evaluation on two NLP tasks, namely query expansion for Information Retrieval and short answer grading. Dahou et al. (2016) used the analogy questions from Zahran et al. (2015) after correcting some Arabic spelling mistakes resulting from the translation and after adding new analogy questions to make up for the inadequacy of the English questions for the Arabic language. They also performed an extrinsic evaluation using sentiment analysis. Finally, Al-Rfou et al. (2013) generated word embeddings for 100 different languages, including Arabic, and evaluated the embeddings using part-of-speech tagging, however the evaluation was done only for a handful of European languages. Elrazzaz et al. (2017) built a benchmark in MSA that can be utilized to perform intrinsic evaluation of different word embeddings using word analogy questions. They then used the constructed benchmark to evaluate various Arabic word embeddings. They also performed extrinsic evaluation of these word embeddings using two NLP tasks, namely Document Classification and Named Entity Recognition. Salama et al. (2018) investigated enhancing Arabic word embeddings by incorporating morphological annotations to the embeddings model. They tuned the generated word vectors to their lemma forms using linear compositionality to generate lemma-based embeddings. To assess the effectiveness of their model, they used the benchmark built by Elrazzaz et al. (2017). Taylor and Brychcín (2018b) demonstrated several ways to use morphological Arabic word analogies to examine the representation of complex words in semantic vector spaces. They presented a set of morphological relations, each of which can be used to generate many word analogies.
El Bazi and Laachfoubi (2017) investigated the effect of stemming on Arabic word representations. They applied various stemmers on different word representations approaches, and conducted an extrinsic evaluation to assess the quality of these word vectors by evaluating their impact on the Named Entity Recognition task for Arabic. Taylor and Brychcín (2018a) provided a corpus of Arabic analogies focused on the morphological constructs which can participate in verb, noun and prepositional phrases. They conducted an examination of ten different semantic spaces to see which of them is most appropriate for this set of analogies, and they illustrated the use of the corpus to examine phrase-building. Barhoumi et al. (2020) proposed intrinsic and extrinsic methods to evaluate word embeddings for the specific task of Arabic sentiment analysis. For intrinsic evaluation, they proposed a new method that assesses what they define as the "sentiment stability" in the embedding space. For extrinsic evaluation, they relied on the performance of the word embeddings to be evaluated for the task of sentiment analysis. They also trained various word embeddings using different types of corpora (polar and non-polar) and evaluated them using their proposed methods. To the best of our knowledge, our proposed benchmark is the first benchmark developed in various Arabic dialects and that can be used to perform intrinsic evaluation of Arabic word embedding with respect to those dialects.

Conclusion
We described DiaLex, a benchmark for evaluating dialectal Arabic word embeddings. DiaLex comes in five major Arabic dialects, namely Algerian, Egyptian, Lebanese, Syrian, and Tunisian. Across these dialects, DiaLex offers a testbank of word pairs for six syntactic and semantic relations, namely male to female, singular to dual, singular to plural, antonym, comparative, and genitive to past tense.
To demonstrate the utility of DiaLex, we used it to evaluate a set of available and newly-developed Arabic word embedding models. Our evaluations are intended to showcase the utility of our new benchmark. DiaLex as well as the newly-developed word embeddings will be publicly available. In the future, we plan to use DiaLex to conduct a a more extensive evaluation of all publicly-available Arabic word embedding models. We also plan to train more dialectal Arabic word embeddings than the ones we described here and evaluate them using DiaLex.