{'fraction_non_alphanumeric': 0.03657061152661681, 'fraction_numerical': 0.01627804663440387, 'mean_word_length': 4.873708010335918, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
An Arabic Dependency Treebank in the Travel Domain

In this paper we present a dependency treebank of travel domain sentences in Modern Standard Arabic. The text comes from a translation of the English equivalent sentences in the Basic Traveling Expressions Corpus. The treebank dependency representation is in the style of the Columbia Arabic Treebank. The paper motivates the effort and discusses the construction process and guidelines. We also present parsing results and discuss the effect of domain and genre difference on parsing.


Introduction
Treebanks, or annotated corpora, are essential for Natural Language Process (NLP) tasks. Such tasks include building lexicons, inferencing grammars, and creating computational analyzers, which can all be improved by creating treebanks with different kinds of linguistic annotations (Abeillé, 2012). Treebanks with rich annotation and good quality are very expensive resources to create. They require a large number of man-hours to create and audit. Treebanks can be in multiple genres, or genre-specific. 1 However, there is a tradeoff between the cost of the size, the diversity of a corpus, and having enough content in one genre or domain to be able to make generalizations. As a result, many treebanks tend to be predominantly of one specific genre, but may add some samples of other genres. For example, the Hindi/Urdu Treebank (Bhat et al., 2017) is predominantly in the news domain with 85.3% of its sentences coming from news articles, and only 14.7% from other domains (9.7% from conversations, and 5% from the travel domain). Webber (2009) shows that the Penn Treebank (Marcus et al., 1994) consists of 90.1% news articles, 4.9% essays, 2.6% summaries, and 2.4% letters, and it is still considered to be a news domain treebank. Similarly, Maamouri et al. (2010) demonstrate that the Penn Arabic Treebank (PATB) (Maamouri et al., 2004) consists of 39.9% newswire text, 28.2% broadcast news, 18.6% broadcast conversation in both Standard and Dialectal Arabic, and 13.3% web texts. In this paper we describe a small Modern Standard Arabic (MSA) treebank, created using a travel corpus. This treebank will be the seed of a larger multi-genre, and multidialect Arabic treebank. The corpus we are using is part of an MSA translation by Eck and Hori (2005) of the Basic Travel Expression Corpus (BTEC) (Takezawa et al., 2007), henceforth MSABTEC. As far as we know, there is no treebank based on this corpus. 1 The terms domain, genre, topic and style have been discussed a lot in the field (Lee, 2002;Van der Wees et al., 2015;Ide and Pustejovsky, 2017), and many authors discussed their ambiguous and overlapping use. For the rest of this paper we use the term travel domain, following Takezawa et al. (2007) whose corpus was the basis for the translated corpus we treebank.
In Section 2, we discuss related work followed by a description of the corpus we annotate in Section 3. In Section 4, we discuss the annotation format; and in Section 5 the annotation process. Finally, we present some results on benchmarking parsing on our corpus and a comparison with a major news-domain Arabic treebank in Section 6.

Related Work
BTEC is a collection of conversational phrases that cover various situations in the travel domain in Japanese, and their translations into English and Chinese (Takezawa et al., 2007). The sentences in the corpus were collected from bilingual travel experts, and were based from their experience rather than being transcribed. The corpus was later translated into more languages including Arabic (Eck and Hori, 2005), where it was used for evaluating machine translation systems. Another treebank that included phrases from the travel domain is the Hindi/Urdu treebank (Bhat et al., 2017). Even though the majority of the treebank comes from news sources, it contains 15K words, making up 1,058 sentences relating to heritage and tourism. This part of the data was specifically added to counteract the bias that could result from using data in one specific domain, news in this instance. The treebank contains dependency, phrasestructure, and PropBank-inspired (Kingsbury et al., 2002) annotations. The Penn Treebank is a well known resource, that contains phrases mostly from the news domain. The treebank was annotated for genres as part of the Penn Discourse Treebank (Miltsakaki et al., 2004), andWebber (2009) shows that the different genres can have different characteristics. The Penn Arabic Treebank (PATB) is the primary treebank for work on Arabic syntactic analysis. It uses a phrasestructure representation; but has been converted to other dependency formalisms Taji et al., 2017). The PATB contains various parts that come from different domains and resources. PATB comes in 12 parts (Diab et al., 2013), that are mostly from news or web sources (Maamouri et al., 2010). Other related treebanks were developed by the Linguistic Data Consortium (LDC) in various dialects such as Egyptian (Maamouri et al., 2012), and Levantine (Maamouri et al., 2006), where the data came from transcribing recorded conversations. The first dependency Arabic treebank was the Prague Arabic Dependency Treebank (PADT) (Hajič et al., 2004). It employed a multi-level description scheme for functional morphology, analytical dependency syntax, and tectogrammatical representation of linguistic meaning. Another large Arabic treebank is the Columbia Arabic Treebank (CATiB) ). CATiB has around 250K words that were annotated directly in it in addition to the full converted PATB. CATiB focuses on news domain text in Standard Arabic. Most recently, Taji et al. (2017) converted the PATB into the formalism of the universal dependency (UD) project (Nivre et al., 2016) via an intermediate step of mapping to CATiB dependencies. The Quran Corpus is another important Arabic syntactic corpus of the very specific genre of holy scripture (Dukes and Buckwalter, 2010). It has its own representation scheme which is a hybrid dependency and constituency. In this work, we annotate in the format of the CATiB treebank and compare to UD representations. And we present a comparison with the news domain as captured in the PATB.

Our Corpus
For our corpus, we selected the MSA translation of BTEC (Eck and Hori, 2005).Our selection contains 2,000 sentences making a total of 15,929 words (7.9 words/sentence). The sentence choice overlapped with the test set used in another project that focuses on machine translation and language identification (Anonymous, under review). The text of the corpus, coming from BTEC, is full of travel related expressions such as inquiring about the prices of hotel rooms, asking for directions, requesting help, ordering food, etc. Being conversational, it also has a high percentage of first and second person pronouns and conjugations.

Annotation Format
To maximize compatibility with previous efforts, we followed the Columbia Arabic Treebank (CATiB) ) annotation guidelines and tokenization schemes used by previous Arabic treebanks. We chose this format because it uses traditional Arabic grammar as the inspiration for its relational labels and dependency structure , making it intuitive for Arabic speakers, and allowing for faster annotation. In addition, this format can be automatically enriched with more morphological features (Alkuhlani et al., 2013), and converted into other dependency formats such as the Universal Dependency format (Taji et al., 2017). Except for a number of minor specifications for some new syntactic constructions, there was no change to the guidelines for tokenization, partof-speech (POS) tag set and relations.

Tokenization
The tokenization followed in the treebank creation is the same tokenization scheme used in PATB. This scheme tokenizes all the clitics, except for the definite article + Al+ 'the' (Pasha et al., 2014). The 2,000 sentences in our corpus consist of 18,628 tokens (manually checked).

Annotation Scheme
For our treebank, we followed the CATiB dependency annotation scheme. This scheme is designed to be speedy for annotation, and intuitive for Arabic speakers. We also used the guidelines that were prepared for the CATiB annotation project ).

POS Tags
The CATiB annotation scheme uses six POS tags which are NOM for all nominals excluding proper nouns; PROP for proper nouns; VRB for active-voice verbs; VRB-PASS for passive-voice verbs; PRT for particles, which include prepositions and conjunctions; and PNX for punctuation marks.

Relations
There are eight relations used in the CATiB scheme: SBJ for the subjects of verbs and the topics of simple nominal sentences; OBJ for the objects of verbs, prepositions, or deverbal nouns; TPC for the topics of complex nominal sentences which contain explicit pronominal referents; PRD for the complements of the extended copular constructions; IDF for marking the possessive nominal construction (idafa); TMZ for marking the specification nominal construction (tamyiz); MOD for general modification of verbs or nominals; and, finally, -for marking flat constructions such as first-last proper name sequences.

Syntactic Structures
Since the original CATiB treebank, as with the Penn Arabic treebank, was focused on the news genre, there were many syntactic constructions that MSABTEC introduced that needed special attention. In particular, there was an abundance of interrogatives, and first and second person statements in MSABTEC compared to CATiB. To address these constructions, additional guideline specifics and clarifications were added. All of these extensions followed naturally from the spirit of the original guidelines. For example, an interrogative pronoun such as man 'who/whom' is often sentence-initial, but it can be the subject or the object of a verb: man samiςa +ka? 'who heard you?' versus man samiςta? 'whom did you hear?'. Similarly, in Figure 1 (C), the interrogative adverb Âyn 'where' is treated as the predicate head of a copular sentence since that is the syntactic role of the answer to the question. For another common example in this genre, single word interjections such as Ā sf 'sorry' or škrAã 'thanks' are treated as independent sentence trees that attached directly to the main root of the sentence they appear in.

Interface
The annotation was done using the TrEd annotation interface (Pajas, 2008), which was also used by  for CATiB annotation. Figure 1 illustrates the annotation scheme of three examples from the MSABTEC Treebank in the CATiB format in which they were annotated. We also provide, for comparison, the analysis in the increasingly popular Universal Dependency representation (Nivre et al., 2016;Taji et al., 2017).

Annotation Process
The annotation process we followed in the preparation of this treebank is the same process described by , which consisted of the following steps: (a) Automatic Tokenization and POS Tagging, (b) Manual Tokenization Correction, (c) Automatic Parsing, and (d) Manual Annotation. In this section, we discuss what we did for these steps as well as report on annotator(s), speed and inter-annotator agreement.

Annotator(s)
Due to the relatively small size of our treebank, we had only one annotator working on the task. Our annotator is an educated native Arabic speaker, who was trained on the CATiB scheme and the use of TrEd as part of her work on the original CATiB project . To evaluate inter-annotator agreement, we worked with a second annotator who was asked to annotate a small part of the treebank (see below).

Automatic Tokenization and POS Tagging
We used MADAMIRA (Pasha et al., 2014) to tokenize and POS tag the input sentences. We used MADAMIRA's configuration for PATB tokenization and CATiB POS tags.

Manual Tokenization Correction
Our annotator then manually checked and fixed all of the tokenization errors. This also included the correction of typos and spelling changes resulting from wrong automatic analysis. Overall there were 2.8% tokenization errors, which is higher that MADAMIRA's reported tokenization error rate (around 1.1%). The increase is most likely due to the difference in genre between the data used to train MADAMIRA and our corpus.

Automatic Parsing
We ran the data with the fixed tokenization through the CamelParser (Shahrour et al., 2016), which is trained on the gold CATiB representation of the training data from the PATB parts 1, 2, and 3 according to the splits proposed by Diab et al. (2013). We present automatic parsing quality results in Section 6.2.

Manual Annotations
The output of the automatic parsing was given in TrEd's .fs format to the annotator to manually fix the POS tags, the relation labels, and the syntactic structures of the trees.

Annotation Speed
The manual fixing of the tokenization took the annotator 10 hours of work at the speed of 1,593 words/hour. The manual correction of the parsed trees (POS, relations, and structure) took 40 hours of work at the speed of 466 tokens/hour (398 words/hour). This number is comparable to the speed reported by  (540 tokens/hour). The sentences in their treebank were of the same genre as the data used to train the automatic parsers unlike our case; furthermore, their sentences are much longer than ours (32.0 words/sentence compared to our 7.9 words/sentence). These two issues may explain part of the difference in speed. The end-to-end speed (from raw words to fully corrected trees) is 319 words/hour.

Inter-Annotator Agreement
To check the consistency of our annotations, we had another person with previous experience in dependency annotation annotate a subset of 100 sentences from this treebank. The second annotator started from the CamelParser output on the same corrected tokenization produced by the first treebank annotator. The inter-annotator agreement scores are 98.7% on POS agreement, 96.1% on label agreement, 90.6% on attachment agreement, and 89.7% on labeled attachment agreement. This is close to the highest average pairwise inter-annotator agreement number reported on the creation of the CATiB Treebank ).

Results
We present next a comparison between our treebank and the Penn Arabic Treebank, followed by benchmark results of the performance of a state-of-the-art parser on our corpus.

Comparison with Penn Arabic Treebank
Our corpus is from the travel genre, which has some characteristics that are different from those of the news genre. For example, the average sentence length in MSABTEC is 9.31 tokens per sentence, as opposed to PATB's average of 37.57 tokens per sentence. Over 40% of MSABTEC sentences contained a question, while in PATB this percentage did not exceed 2.6%. This is expected as travel corpora are more likely to include questions and answers by travellers. Moreover, the most frequent words in both corpora vary distinctly. MSABTEC's most frequent verb is yumkin 'can', which is often used when asking for help. In PATB, however, the most common verb is qAl 'said', which is commonly used for reporting news. In addition, question words such as kam 'how much', hal 'do/does', and Âyn 'where' appear in the set of the most frequent 50 words in MSABTEC, whereas no question words appear in the set respective to PATB. Frequent nouns in MSABTEC include faĎl 'favor/please', raqam 'number', and γurfah 'room'. In PATB, the most frequent nouns include raŷiys 'president', lubnAn 'Lebanon', Alyawm 'today', and Almut∼aHidah 'the united'. Another phenomenon that differentiates MSABTEC and PATB is the pronoun frequencies. On the one hand, the most frequent pronouns appearing in MSABTEC are + +k, which is the second person singular pronoun in accusative, and + +y and + +ny, which are the first person singular pronouns in genitive and accusative case, respectively. On the other hand, the most frequent pronouns appearing in PATB are + +h and + +hA, which are the masculine and feminine third person singular pronouns, respectively. This leads to the obvious conclusion that MSABTEC mostly contains conversational text that refer to the speaker or the listener, whereas PATB's most dominant style is that of reporting in the third person, which is expected of a news genre corpus.

Automatic Parsing Quality
We parsed our corpus using CamelParser (Shahrour et al., 2016), which was itself trained and optimized on the PATB. Table 6.2. shows the difference in the parser's performance on PATB data, on which it is trained, versus on MSABTEC data. For the PATB, we report on the test set used by Shahrour et al. (2016). The evaluation of the parser was done using the gold annotations of the MSABTEC data. The error increase in the results of MSABTEC from the results of PATB for the Labeled Attachment Score (LAS), Unlabeled Attachment Score (UAS), and Label selection is 64%, 70% and 39%, respectively. This shows that the genre difference between the training data and the testing data significantly affects the performance of the parser. The previously described characteristics that differ between PATB and MSABTEC (sentence length, prevailing person, and different frequent words) can explain this decline in performance. The large performance drop highlights the need for creating treebanks in less-studied genres to support research on them.

Conclusion and Future Work
We presented a small dependency treebank of travel domain sentences in Modern Standard Arabic. The text comes from a translation of the English equivalent sentences in the Basic Traveling Expressions Corpus. The treebank dependency representation is in the style of the Columbia Arabic Treebank. Our parsing evaluation of the constructed treebank confirms the need for more treebanks in different genres and domains to support research on multi-domain, multi-genre parsers.
In the future, we plan to expand our annotation efforts to other genres and domains as well as to other Arabic dialects. We are also very interested in using the created corpus in improving Arabic syntactic parsing. Since the data we created is small in size compared to the large dominant treebanks, we plan to pursue the genre and domain adaptation research direction. We also plan to make this resource publicly available to support research on Arabic syntactic parsing.