{'fraction_non_alphanumeric': 0.03163118593757247, 'fraction_numerical': 0.013542971105236307, 'mean_word_length': 4.849701573521433, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Morphologically Annotated Corpus of Emirati Arabic

Morphology; Natural language processing systems; Annotation; Dialect identification; Emirati; Gulf Arabic; Lemmatization; Part Of Speech; Part of speech tagging; Tokenization; Computational linguistics


Introduction
There has been an increasing number of natural language processing (NLP) efforts focusing on dialectal Arabic, especially with the increasing amounts of written material on the web. However, resources for dialectal Arabic NLP tasks such as part-of-speech (POS) tagging, morphological analysis and disambiguation are still lacking compared to those for Modern Standard Arabic (MSA). MSA is the official language in more than 20 countries, where it is used in official communications, news, and education. Yet, it is not the commonly spoken variety of Arabic; the dialectal varieties of Arabic are what is used in the day-to-day communication. Dialectal Arabic is also commonly used in written form on social media platforms, forums and blogs.
Using available resources developed for MSA such as POS taggers and tokenizers gives limited performance when used on dialectal Arabic Jarrar et al., 2014;Khalifa et al., 2016a). Many researchers moved into the direction of creating tools and resources targeting the dialects specifically. Egyptian Arabic is one of the dialects that received earlier efforts for developing tools and resources. More resources are being developed for other dialects such as Levantine, Tunisian, Moroccan and Yemeni Arabic. Gulf Arabic, as we define it to be the native spoken variety in the Gulf Cooperation Council, is still lagging behind other Arabic dialects with respect to resource and tool creation, given the considerable amount of dialectal content online.
In this paper, we present an ongoing project for creating a manually annotated corpus of about 200,000 words of the Gulf Arabic of the United Arab Emirates -Emirati Arabic. The corpus is annotated for tokenization, POS, lemmas and English glosses in addition to spelling conventionalization and dialect identification. This resource will support the development of Arabic dialect enabling technologies, such as automatic POS tagging and morphological disambiguation, which in turn will facilitate efforts on different NLP tasks such as machine translation.
The rest of this paper is organized as follows. We discuss related work on dialectal corpora in Section 2. In Section 3. we describe the corpus used in this effort. We then present the annotation guidelines that are used to annotate the corpus in Section 4. We discuss the annotation process and the annotation quality results in Section 5.

Related Work
In this section we review a number of efforts on Arabic corpus creation, that significantly supported research and tool development for Arabic NLP.

Modern Standard Arabic Resources
The Penn Arabic Treebank (PATB) (Maamouri et al., 2004) has been a central resource for developing MSA resources. It was developed at the Linguistic Data Consortium (LDC), and it mainly consists of newswire text from different news sources. The PATB corpus is annotated for tokenization, segmentation, POS tagging, lemmatization, diacritization, English gloss and syntactic structure. The PATB has 12 parts of more than 1.3 million words. The annotated data has been a backbone of many state-of-the-art tools such as analyzers and disambiguators including MADAMIRA (Pasha et al., 2014) and its predecessor MADA (Habash et al., 2009), in addition to YAMAMA (Khalifa et al., 2016b), and most recently a neural morphological disambiguatior (Zalmout and Habash, 2017) and a fine grained POS tagger (Inoue et al., 2017). In addition, the PATB guidelines ) have inspired the creation of similar guidelines for the dialects including our own.

Dialectal Arabic Resources
In the scope of dialectal Arabic, there have been many recent contributions to the development and creation of resources. Below, we discuss the highlights of those contributions.
Egyptian Arabic Resources Egyptian Arabic (EGY) was one of the first dialects that received the attention of the NLP community. The earliest effort, to the best of our knowledge, is the Egyptian Colloquial Arabic Lexicon (ECAL) (Kilany et al., 2002) which was developed as part of the CALLHOME Egypt corpus (Gadalla et al., 1997). The ECAL served as the seed to the EGY morphological analyzer (CALIMA) (Habash et al., 2012a). Later on, the Egyptian Arabic Treebank (ARZATB) (Maamouri et al., 2012a;Maamouri et al., 2014) was created by the LDC using CALIMA to provide analysis options for the annotation process. The ARZATB has currently 400,000 words in eight parts annotated in a similar fashion to the PATB. The annotation guidelines for the ARZATB (Maamouri et al., 2012b) followed that of the PATB with decisions specific to the dialect. Since the release, the ARZATB has been used extensively for developing EGY resources such as the EGY part of MADAMIRA, MADA and YAMAMA, in addition to a noise-robust morphological disambiguator for EGY . Other developed corpora and POS taggers for EGY include the work of Al-Sabbagh and Girju (2012) where they created their own POS tagset and corpus with the intention to facilitate certain NLP applications like subjectivity and sentiment analysis.
Levantine Arabic and and Other Dialectal Arabic Resources Levantine Arabic (LEV) received some notable efforts including the Levantine Arabic Treebank (LATB) of Jordanian Arabic (Maamouri et al., 2006) which contains around 27,000 annotated words in a similar fashion to ARZATB. A more recent resource is the annotated corpus of Palestinian Arabic (Curras) (Jarrar et al., 2014;Jarrar et al., 2016). ARZATB and Curras were used to create morphological analyzers and disambiguators . Other dialects such as Yemeni and Moroccan Arabic followed the same approach (Al-Shargi et al., 2016). In addition to the dialects mentioned above, there were recent efforts on creating corpora for other dialects, namely Tunisian and Algerian (McNeil and Faiza, 2011;Masmoudi et al., 2014;Zribi et al., 2015;Smaïli et al., 2014). Other works targeted multi-dialect corpora (Diab et al., 2010;Zaidan and Callison-Burch, 2011;Diab et al., Forthcoming 2013;Bouamor et al., 2014;Cotterell and Callison-Burch, 2014), and, most recently, the ongoing Multi Arabic Dialect and Application Resources project (MADAR)  which includes corpora for 25 different city dialects.
Gulf Arabic Resources As far as Gulf Arabic (GLF) is concerned, the only existing annotated corpora include the Emirati Arabic Corpus (EAC) (Halefom et al., 2013) and the Emirati Arabic Language Acquisition Corpus (EMALAC) (Ntelitheos and Idrissi, 2017) that were created by linguists with emphasis on the phonological and morphosyntactic phenomena of Emirati Arabic. We recently collected a large-scale corpus of Gulf Arabic (Khalifa et al., 2016a) containing more than 100 million words covering six Gulf Arabic varieties. In regards to other tools and resources, we recently developed a morphological analyzer for Gulf Arabic verbs (CALIMA GLF ) (Khalifa et al., 2017). We are also aware of the previously developed rule-based stemmer for Arabic Gulf dialect (Abuata and Al-Omari, 2015).
In this work, we use about 200,000 words from the Emirati Arabic portion of the Gumar corpus to manually anno-tate for tokenization, POS tagging, lemma, English gloss and dialect identification. Additionally we conventionalize the spelling in accordance with the Conventional Orthography for Dialectal Arabic (CODA) rules Habash et al., 2018).

Annotating the Gumar Corpus
We discuss next the Gumar Corpus and the portion of it we use to annotate in this effort.

Gumar Corpus
The Gumar corpus is a large-scale corpus of Gulf Arabic containing more than 100 million words. The corpus consists mainly of documents of long conversational novels also known as 'Internet Novels'. This type of literature is very popular among female teenagers in the Gulf area. These novels are written mostly in dialectal Arabic, where the lengthy conversations between the characters of the story are in the dialect and the narration in between the conversations can sometimes be in MSA.
The writers of the novels remain anonymous and use noms de plume. The novels are publicly available online, where most of the writers ask for their pen name to be mentioned if the novel is to be published in a different platform than the original. The genre of the novels is mainly romantic, but also features tragedy and drama. The corpus can be browsed online, 1 it is currently annotated using MADAMIRA in EGY mode.
On the document level, Gulf Arabic text makes up more than 90% of the corpus, the rest of the corpus consists of other Arabic dialects in addition to MSA. Emirati Arabic text covers around 11% of the Gumar corpus.

The Annotated Gumar Corpus
We chose a set of 200,110 word tokens for the annotation task. The text consists of the first 25,000 words (rounded up to the nearest full sentence) from eight different novels by eight different authors. This allows us to cover different writing styles. The text is comprised of 15,277 sentences with an average of 13 words per sentence. Table 1 shows the list of the novels from which the text is selected. We name this subset of the corpus the Annotated Gumar Corpus. In the future we plan to continue adding annotations to it from other Gumar novels including different dialects.
Additionally, a total of about 12,000 words -1,500 words from each of the eight parts rounded up to the nearest full sentence -are chosen to evaluate Inter-Annotator Agreement (IAA) throughout the annotation process. Thus, the total number of words to be annotated is about 212,000 words.

Parts Tokens Sentences
Title

Annotation Guidelines
In this section, we present the guidelines with examples for each of the different annotation tasks. The annotation contains six different tasks: spelling conventionalization according to CODA, tokenization, POS tagging, lemmatization, English glossing and dialect Identification on the word level.

CODA Spelling Guidelines
Emirati Arabic is similar to other dialects where there is no standard orthography. For example the word for 'hunger' may be spelled phonetically Alywς 2 or using the MSA cognate Aljwς. Hence, there will always be inconsistencies between different writers or even within the same writer . In this annotation effort we follow the newly revised set of CODA* guidelines which include consonant mapping, vowel spelling and affixation and cliticization rules .

Tokenization Guidelines
Previous efforts used different tokenization and segmentation schemes depending on the goal of the task. In this annotation task we use the D3 tokenization (Habash, 2010 2 Arabic transliteration is presented in the Habash-Soudi-Buckwalter scheme (Habash et al., 2007): (in alphabetical order) Â b t θ j H x d ð r z s š S D TĎ ς γ f q k l m n h w y and the additional symbols: ' , Â ,Ǎ ,Ā ,ŵ ,ŷ ,h , ý .

POS Guidelines
In this work, we opted to use a new POS tagset -CAMEL POS. CAMEL POS is inspired by the ARZATB tagset and guidelines (Maamouri et al., 2012b) which is based on the PATB guidelines . The CAMEL POS is designed as single tagset for both MSA and the dialects with the following goals in mind: (a) facilitating research on adaptation between MSA and the dialects, and among the dialects; (b) supporting backward compatibility with previously annotated resources; and (c) enforcing a functional morphology analysis that is deeper and more compatible with Arabic morphosyntactic rules than formbased analysis (Alkuhlani and Habash, 2011). The CAMEL POS tags and features are the union of those in MSA and the dialects. Features are available to use when needed. For example case and state features are used more often in MSA; but on the other hand, dialects tend to have many more clitics than MSA, including non-MSA ones.
One of the main differences between CAMEL POS and ARZATB is that the morphological features of both gender and number of nominals are annotated functionally (Alkuhlani and Habash, 2011;Smrž, 2007). This decision allows us to assign the features to the baseword without the need to specify the surface form affixes that mark form gender and number. This is not the case in ARZATB, where broken plural nouns are tagged singular because they do not use the sound plural affixes.
The other main difference is that we omit case and state features for nominals, and voice and mood for verbs as the dialects have almost lost them completely, except for some high frequency fossilized MSA forms, such as TabςAã 'of course' which retains an indefinite ending.
The main part of the word, that is the baseword, is tagged in the following format: 'POS.features', where 'POS' is the core POS tag and 'features' is the possible feature combination that goes with the POS tag, a '.' separates the POS from the feature combination. Proclitics, however, get only a 'POS' tag since they have no features. However, pronom- <POS>.<A><P><G><N>.<S><C><V><M> The second period is not necessary if none of the last four features is specified. Table 2 shows the list of POS tagset used in this annotation effort compared with the ones used ARZATB. The tagset is divided into three categories according to the tokenization scheme we follow: proclitics (14 tags), enclitics (2 tags) and baseword (39 tags). Together with the features, CAMEL POS tagset maps to ARZATB and retains backward compatibility. It also offers an intuitive Arabic scheme that is suitable to use for annotation.
For a subset of POS tags in the baseword category, each POS tag has a limited number of possible feature combinations that is paired with it. Below is the list of the POS tags that take features and their possible ordered combination.
• NOUN, NOUN_*, ADJ, ADJ_* All nominals take the combination of Gender, Number. In cases where a feature is not present, such as gender in verbs of first person inflections, the gender feature is simply dropped and does not require a placeholder since the possible feature values are ordered and unique. For example the imperfective 1st person verb Aqwl 'I say' will be tagged as VERB.I1S

Lemma Guidelines
The lemma is the citation form of the the word. We follow the same guidelines of the lemma specification from Graff et al. (2009)  (VERB.I3FP) is šAf . For all other tags (i.e. particles, adverbs, ... etc) the lemma is the same form of the baseword. In this annotation effort, the lemma is the only form we require to be manually diacritized.

English Gloss Guidelines
The English gloss in this context refers to the semantic translation of the Arabic lemma. For nominals we use the singular form, and for verbs we use the infinitive form. An Arabic lemma could have multiple synonymous English glosses. For example kbyr would have the following English glosses 'large; great; important; major; senior'.

Word Level Dialect Identification
Dialect identification is the task of tagging a certain context with a given dialect tag. Deciding the dialect tag depends on the context of the sentence and/or the document. This can be challenging since many words in their written form may be shared by many dialects and MSA. Additionally, it is not uncommon to find dialect code switching between MSA and a dialect, and even a dialect with another dialect (less commonly) (Elfardy and Diab, 2012). Hence we tag per word, but rely on the context of the sentence and even the document to identify the dialect.
In Table 3 we show an example of a fully annotated sentence and the POS tag in ARZATB for comparison. For full description of each of the annotation tasks and examples, the full guidelines can be accessed online.

Annotation Process
In this section, we discuss the annotation process details, the tool we used, and some annotation quality evaluation results.

MADARi Interface
We used a newly developed interface for morphological annotation and spelling correction called MADARi . MADARi is a web-based interface that supports joint morphological annotation (tokenization, POS tagging, lemmatization) and spelling correction at any point of the annotation process, which minimizes error propagation. English glossing and dialect identification are also supported in the interface. MADARi assigns initial answers to the new text using MADAMIRA in EGY mode, whose databases we extended with CALIMA GLF for more coverage. MADARi has many utilities to facilitate the annotation process that we utilize for more efficiency, of which examples are discussed in the next subsection. Figure 1 shows a screenshot of the annotation view in MADARi.

Manual Annotation
The annotator starts on an automatically pre-annotated document. They carefully examine the spelling of each word and all its analysis choices in context with reference to the raw text at all times. For each word the annotator faces one of the following scenarios: • All annotation tasks are correct: the annotator has to only validate the answer. • Correct analysis but wrong spelling: the annotator has to adjust the spelling and then validate the answer. • Wrong analysis (wholly or partially) but correct spelling: the annotator can manually adjust the analysis or can use the 'analysis search' utility provided by MADARi to get an analysis for a word with similar structure and then they would only have to change the lemma and the gloss entries. Finally they validate the answer. • Wrong analysis and spelling: the annotator has to adjust the spelling and follow the previous step.
At any point of the annotation process, the annotator is able to apply mass changes to spelling and/or analysis across the document they are working on. However, the annotator must insure that all the words affected by the change are in similar contexts. The annotator can also modify their answers any time during the annotation through feedback they get if they have any inquiries. This allows the annotator to skip over words they are not confident about and leave the answer unvalidated.
Once the annotation task is fully completed, the annotator may 'submit' the finished document to be later exported. This will allow all the analyses made by the annotator to be accessible to all the other annotators when they look up the analysis for similar words.

Inter Annotator Agreement
We evaluated the quality of the annotation using the Inter Annotator Agreement (IAA) measure between two annotators on a selected text of 1,500 words. We measured the agreement on: (i) word boundary, that is the agreement on whether word boundaries are the same (no splits/merges); (ii) CODA spelling; (iii) baseword form; (iv) baseword POS; (v) baseword features; (vi) clitic form (averaged across all clitic positions) and (vii) clitic POS (averaged across all clitic positions). To align the pair of annotations, we perform a word level alignment within the sentences. We use a weighted Levenshtein distance to maximize alignment, where insertions and deletions are weighted as 1 and substitutions are weighted as follows: Above, t 1 , t 2 are the two word tokens, and Lev is the Levenshtein distance at the character level. We employ this character-based weighing scheme to encourage the alignment of words with spelling changes. Using the same IAA measure, we measured the similarity between each annotator and the initial answers from the CALIMA GLF -extended MADAMIRA.
The results are presented in Table 4 Table 3: An annotation example in the CAMEL POS scheme showing the different entries per word, in addition to the annotations in the ARZATB tagset for comparison. While Arabic is written from right to left, the tags above are displayed from left to right. Figure 1: Example of the annotation step using the MADARi interface. The top gray box shows the raw sentence; next are the word tokens reflecting any spelling changes made. The section below shows all the fields required to annotate; they are initially populated using MADAMIRA. This example is of a manually annotated entry following the discussed guidelines.  Table 4: Percentages of agreement between two annotators (i.e. A1 and A2) and between each annotator and the extended MADAMIRA (i.e. M) initial answers.
MADAMIRA's analysis choice. For each aligned pair of annotations, we compute the number of agreements for the considered categories (i-vii). The IAA score across the various categories ranges from 94.7% on CODA to over 99% on clitic annotations. Moreover, the measures between the annotators and MADAMIRA's answers show that both annotators changed many of the initial answers and their change was consistent to a large extent. 3

Conclusion and Future Work
We presented an ongoing project for creating a manually annotated corpus of about 200,000 words of Emirati Arabic -the Annotated Gumar Corpus. We discussed the full guidelines for the different annotation components that include spelling adjustments, tokenization, POS tagging, lemmatization, English glossing and dialect identification. We used a newly developed interface for morphological annotation and spelling correction. We described the manual annotation process and finally measured the quality of the annotation through an IAA measure that found agreements ranging between 94.7% to more than 99% for different annotation tasks. In the future, we plan to expand the annotated text to include other genres and dialects. We are also interested in using the annotations to improve the quality of Arabic dialect POS tagging and morphological disambiguation.