{'fraction_non_alphanumeric': 0.05704248139416632, 'fraction_numerical': 0.008644200989229418, 'mean_word_length': 5.0616419165032225, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Standard Arabic Formalization and Linguistic Platform for its Analysis

From the beginning of the sixties, and starting with the first automatic analyzer proposed by David Cohen, one of the first theorists of NLP [1], research has continued with natural language processing and especially the automatic treatment of the Arabic language. In 1983, with a minimalist morphological analysis, based on the theory that any Arabic form is generated using root and pattern, researchers developed the first twolevel morphological analyzer for Arabic (Koskenniemi 1983); this work was included within the project ALPNET (Beesley and Buckwalter 1989) using finite-state technology allowing only the concatenation of morphemes in the morphotactics. Since 1996, the Xerox research centre has enhanced this system using an algorithm of automatic combination between roots and patterns generating stems; this research is based on the ALPNET’s dictionaries which were, considerably rebuilt using the Xerox finite-state technology (Beesley 2001). This technology is computationally very efficient for natural-language-processing; it’s used within the developmental environment NooJ (Silberztein 2006). The use of finite-state machines within NooJ was extremely attractive, they are used to generate and analyse several thousands of words per second. This linguistic platform will be described inside this paper as the tool used for vocabulary formalization and analysis of standard Arabic language.


INTRODUCTION
From the beginning of the sixties, and starting with the first automatic analyzer proposed by David Cohen, one of the first theorists of NLP [1], research has continued with natural language processing and especially the automatic treatment of the Arabic language. In 1983, with a minimalist morphological analysis, based on the theory that any Arabic form is generated using root and pattern, researchers developed the first twolevel morphological analyzer for Arabic (Koskenniemi 1983); this work was included within the project ALPNET (Beesley and Buckwalter 1989) using finite-state technology allowing only the concatenation of morphemes in the morphotactics. Since 1996, the Xerox research centre has enhanced this system using an algorithm of automatic combination between roots and patterns generating stems; this research is based on the ALPNET's dictionaries which were, considerably rebuilt using the Xerox finite-state technology (Beesley 2001). This technology is computationally very efficient for natural-language-processing; it's used within the developmental environment NooJ (Silberztein 2006). The use of finite-state machines within NooJ was extremely attractive, they are used to generate and analyse several thousands of words per second. This linguistic platform will be described inside this paper as the tool used for vocabulary formalization and analysis of standard Arabic language.

ARABIC LANGUAGE DESCRIPTION
The Arabic language is a Semitic language showing two great characteristics which have been the subject of much research: agglutination and non vocalization. In fact, most forms [2] in Arabic writing can correspond to a succession of one or more prefixes, a radical and one or more suffixes. Radicals themselves are forms which have been inflected or derived from lemmas. However, non vocalisation is due to a lack of short vowels in usual texts from which a high degree of ambiguity ensues. When they are present, short vowels are represented by diacritics which appear above or below the consonants that they follow. In theory, only the Coran, and children's book are fully vowelled; the automatic analysis of Arabic must be able to parse fully vowelled, partially vowelled and unvowelled texts.
To resolve these problems, we use finite state machines which we associate to dictionaries of lemmas.

NOOJ AND ARABIC ANALYSIS
NooJ is a linguistic developmental environment which can analyze texts of several million words in real time. It includes tools to construct, test and maintain largecoverage lexical resources, as well as morphological and syntactic grammars. Dictionaries and grammars are applied to texts in order to locate morphological, lexicological and syntactic patterns, remove ambiguities, and tag simple and compound words. NooJ recognizes all Unicode encodings and the runtime code that applies lexical transducers to input strings is completely language independent. Thus the code that runs the Arabic morphological analyzer is exactly the same code that processes a dozen languages, including some Roman, Germanic, Slavic, Semitic and Asian languages, etc. NooJ can build lemmatized concordances of large texts from Finite-State or Context-Free grammars, and can accordingly perform cascading transformation operations on texts, in order to annotate the text, or to generate paraphrases. The NooJ lexical module that will be used throughout this paper relies on operators performing transformations inside strings, and morphological graphs describing grammatical rules for morphological analysis. Generally, transformations inside strings are based on use of some generic predefined commands: • <B>: keyboard Backspace, Although these commands are defined in advance into NooJ, we can make meaning restatement or add new commands; for Arabic, as a highly inflectional and derivational language, we had to define three new operators such as the <T> operator that checks if the last consonant within a noun is a ‫"ة"‬ (T -Teh marbouta) to replace it with a ‫"ت"‬ (t -Teh maftouha) in some inflexional or derivational descriptions. For example, when performing the dual form from "َ ‫َﺔ‬ ‫َﺳ‬ ‫ْر‬ ‫َﺪ‬ ‫"ﻣ‬ (madorasaTa -a school), we have to substitute the ‫"ة"‬ (T -Teh marbouta) with a ‫"ت"‬ (t -Teh maftouha) before suffixing with "ِ ‫ْﻦ‬ ‫"ﻳ‬ (yoni) to obtain "ِ ‫ْﻦ‬ ‫َﻴ‬ ‫َﺘ‬ ‫َﺳ‬ ‫ْر‬ ‫َﺪ‬ ‫"ﻣ‬ (madorasatayoni -two schools). The two other added operators <M> and <Z> will be described in section 4.1 These instructions can be associated with two argument types; either a number (e.g. <L3>: go left 3 times) or a "W" (e.g. <LW>: go to beginning of word). These commands operate on a letter pile, they require a O(n) transformation time. So, they guarantee a correspondence between lemma and corresponding inflected form in a linear time.
Morphological graphs have a great interest in our researches; so we will use the NooJ graph editor to construct either morphological or syntactic grammar. These grammars are used to extract sequences of interest in texts, but also to describe various linguistic phenomena. They represent a group of input sequences and associate them with some output.
Grammars are depicted as finite-state transducers (FST) [4]. A morphological FST represents sequences of letters (that spell a word form), and then produces some kind of lexical information (part of speech, a set of semantic codes, etc.). A syntactic FST represents word sequences, and then produces some kind of linguistic information (its syntactic structure for example).
Considering that certain linguistic phenomena descriptions require the construction of very complicated grammars; they would be simplified and built using dozens of elementary graphs. At the same time, most of these elementary graphs ("local grammars") can be re-used in different contexts, for the description of many different linguistic phenomena. This re-use of constructed elementary graphs, by mentioning them in other graphs, will permit users to construct virtual libraries of graphs, and in so doing, will cover increasingly complex phenomena from morphology to syntax. NooJ [5] is used as a linguistic platform, an information retrieval system, to teach second languages, as a terminological extractor, as well as to teach computational linguistics to students. (Silberztein 2005_1) and (Silberztein 2005_2). Given the explosion of Arabic resources, especially on-line, with more than 20 000 Arabic sites on the Web and more than 300 million users, we recognized the necessity of developing an Arabic component for NooJ platform, which would allow us to process and take advantage of this readily available data. We started building Arabic NooJ module with the purpose of providing automatic analysis of texts written in standard Arabic. This module will be used to a better understanding of the Arabic language based on description of its vocabulary and its transformational syntax according to the theory of Chomsky (1971) and Harris (1985).

ARABIC LEXICON FORMALISATION
The linguistic analysis must go through a first step of lexical analysis, which consists in testing membership of each word of the text to the Arabic vocabulary (Revuz 1991). So, we must begin on formalization of the Arabic vocabulary. This study started with the formalization of three sets: verbs, nouns and particles.

Verbs
The dictionary of verbs contains 10 000 fully vowelled entries [6]. Since automatic combination between roots and patterns leads to virtual lemmas generation or leaves a large number of lexical entries unrepresented and considering that each Arabic root can combine, legally, with a subset of the potential patterns (Dichy 2001), we chose to build a lemma dictionary to avoid such problems evoked within the Xerox lexical analyser. In our case, each entry represents a third person, singular, masculine, perfect verb. These verbs are associated to an inflectional description (among 130 hand-encoded inflexional paradigms for the totality of the verbs) [7]. By inflectional description we refer to the set of all possible transformations which allow us to obtain, from a lexical entry, all inflected forms. These inflexional descriptions include the mood (indicative, subjunctive, jussive or imperative), the voice (active or passive), the gender (masculine or feminine), the number (singular, plural or dual) and the person (first, second or third). On average, there are 122 inflected forms per lexical entry.

Example
The verb "َ ‫َﺄ‬ ‫َﺑ‬ ‫"و‬ (ŵabaǍa -to have a catastrophe) can be associated to two inflexional descriptions to give the two possible forms: • "ُ ‫َﺄ‬ ‫ْﺑ‬ ‫َﻮ‬ ‫"ﻳ‬ (yaŵobaǍa) with simple concatenation of "َ ‫"ي‬ (ya) at the head of verb • "ُ ‫َﺄ‬ ‫َﺒ‬ ‫"ﻳ‬ (yabaǍa) with concatenation of "َ ‫"ي‬ (ya) at the head of the verb and the suppression of "َ ‫"و‬ (ŵa) The first version of inflexional descriptions included more than 130 paradigms to describe all verbal inflexional classes, but the possibility of defining new morphological operators (<Z> and <M> operators) enabled us to gather some descriptions where there wasn't a great inflexional difference. So, these operators allow us to reduce the number of inflexional descriptions to have only 130.
• The morphological <Z> operator is used to describe the two verbs "َ ‫َﺐ‬ ‫َﺘ‬ ‫"آ‬ (kataba -to write) and "َ ‫َﺖ‬ ‫َﺒ‬ ‫"ﺛ‬ (Ťabata -to be proved) within the same paradigm in spite of requiring two different transformations when giving the past tense. In fact, the first verb needs to delete the last vowel and add the letter succession "ُْ ‫"ت‬ (otu) to obtain "ُ ‫ْﺖ‬ ‫َﺒ‬ ‫َﺘ‬ ‫"آ‬ (katabotu -I wrote) and the second verb needs to add Arabic shadda [11] "ّ " (w) and the final vowel "ُ" (u) after deleting the last vowel to obtain ‫ﱡ"‬ ‫َﺖ‬ ‫َﺒ‬ ‫"ﺛ‬ (Ťabatwu -I was proved) ; this was the issue given by the definition of a specific transformational morphological command <Z> which verify if the last consonant of the verb is a ‫"ت"‬ (t) to process inflection in the right way. • The <M> morphological command is used when gathering the two verbs "َ ‫َﺐ‬ ‫َﺘ‬ ‫"آ‬ (kataba -to write) and "َ ‫َﻦ‬ ‫َه‬ ‫"د‬ (dahana -to smear) in spite of having two different transformations for past form in the first person, plural when producing " ْ ‫َﺒ‬ ‫َﺘ‬ ‫آ‬ ‫َﺎ‬ ‫ﻨ‬ " (katabonA -we wrote) and ‫ﱠﺎ"‬ ‫َﻨ‬ ‫َه‬ ‫"د‬ (dahannA -we smeared).

Nouns
The nouns are described in three different ways: • We built a dictionary which contains 15 000 entries in the form of primitive nouns [12], such as "ّ ‫ِﻲ‬ ‫ْﺳ‬ ‫ُﺮ‬ ‫"آ‬ (korsiyy -a chair). Each entry represents a singular noun form deprived of its final vowel. • We added into the same dictionary some plural forms which do not have a singular correspondent form used, such as " َ ‫ﻣ‬ ‫ِف‬ ‫َﺎو‬ ‫ﺨ‬ " (MakhAŵif -dangers, perils).
These nouns, deprived of their final vowel, are associated to inflexional descriptions to generate all inflected nominal forms labelled with some linguistic information such as gender (masculine, feminine or neuter), number (singular, dual and plural) and case (nominative, accusative or genitive). In addition, to perform plural forms from nominal entries, we had to develop about 125 paradigms when describing masculine regular plural, feminine regular plural and broken plural. These paradigms were carefully developed in order to treat certain specificities of Arabic plural such as the difference between plurals of small number and collective plurals such as the case of the form ‫ْﺮ"‬ ‫َﻬ‬ ‫"ﺷ‬ (chahor -a month) which can have two plural forms: ‫ُﺮ"‬ ‫ْﻬ‬ ‫َﺷ‬ ‫"أ‬ (Ǎachohur -less than 12 months) and ‫ُﻮر"‬ ‫ُﻬ‬ ‫"ﺷ‬ (chuhUr -12 months and more). The formalized inflection of verbs, primitive nouns and deverbals allows recognition of all the corresponding inflected terms; the lookup algorithm of NooJ uses finite-state machines, which make possible simultaneous recognition (i.e. without additional computing) both of vowelled, partially vowelled or unvowelled forms. In fact, the omission of diacritics in a written form can lead to numerous distinct fully vowelled words. For example, the unvowelled form "ktb" is supposed to have multiple vocalised annotations, our lookup algorithm based on finite state machines is able to return, at the same time, fifteen fully vowelled forms : • ‫ُﺐ"‬ ‫ُﺘ‬ ‫"آ‬ (kutub -books) with the five possible final vowels, • ‫ْﺐ"‬ ‫َﺘ‬ ‫"آ‬ (katob -a writing) with the five possible final vowels, • "َ ‫َﺐ‬ ‫َﺘ‬ ‫"آ‬ (kataba -he writes) , • "َ ‫ِﺐ‬ ‫ُﺘ‬ ‫"آ‬ (kutiba -it was written) , • "َ ‫ﱠﺐ‬ ‫َﺘ‬ ‫"آ‬ (kattaba -he makes write) , • "َ ‫ﱢﺐ‬ ‫ُﺘ‬ ‫"آ‬ (kuttiba -it was made write) , • "ْ ‫ﱢﺐ‬ ‫َﺘ‬ ‫"آ‬ (kattibo -make write).
Moreover, each recognized form is associated by the lookup algorithm of NooJ a set of linguistic information: lemma, grammatical category, gender and number, syntactic information (e.g. +Transitive) and distributional information (e.g. +Human).

MORPHOLOGICAL ANALYSIS AND GRAMMATICAL RULES DEFINITION
The Arabic language is a strongly agglutinant language; its morphological analyzer should separate and identify the component morphemes of the input word, labelling them somehow with sufficient information to be useful for the tasks at hand.

FIGURE 1: Chain of a text form morphological analysis
We start our analysis with application of a decomposition system, implemented via a NooJ morphological grammar, to each word of the text to identify its radical and affixes.
In the second step, grammars (finite-state transducers) produce lexical constraints checking the validity of segmentation thanks to a dictionary lookup. So, these grammars associate the recognition of a word to lexical constraints, working only with valid combinations of the various components of the form. Typically there are several output strings, each representing a possible analysis of the input word. We continue the description of the morphological analysis of Arabic within the linguistic platform NooJ by detailing the different lexical constraints implemented inside morphological grammars.

Morphological constraints
Morphological constraints follow from the distortion of some radicals by agglutination with prefixes or suffixes. They enact restoration of the initial form such as it appears in the lexicon. We proceed by the application of some morphological transformations (addition of letters, deletion, substitution, etc.)

Linguistic informations associated to recognised forms
Applying morphosyntactic rules
In fact, there are 14 Arabic consonant among the 28 letters in Arabic alphabet that requires fitting Arabic shadda "ّ " into forms when defining them. When present at the beginning of the form, these cases need the same handling as described processing. The described transformations (letter addition, substitution or deletion) can be combined together to deal with more complex morphological phenomena. This first type of constraints considers morphological incompatibilities which would have to be generated from a direct decomposition.

Constraints on the syntactic properties of verbs
These constraints verify the mark "+Transitive" of verbs in the dictionary. Indeed, the transitivity of verbs enables us, generally, to decide possibility of verb suffixation. Such agglutination will be only permitted for direct transitive verbs and indirect transitive ones conjugated at the singular third person (Achour 1998).

Orthographical constraints
These constraints look at letters which have orthographical variation during agglutination. We can expose the case of the letter "T" which can be written in two different orthographies with the same pronunciation. According to the nature of the form (noun, adjective or verb) and the position of the letter in this form (at the beginning, the middle or the end of the word), we can find either of the two spelling forms ‫"ت"‬ (t-Teh maftouha) or ‫"ة"‬ (T -Teh marbouta). This second spelling form can never be found in a verbal form and in case of its appearance at the end of a noun or an adjective it has to be converted into a ‫"ت"‬ (t-Teh maftouha) during word suffixation; thus the reverse operation is required before the dictionary lookup to associate corresponding linguistic information.

Phonological constraints
These constraints, generally combined with morphological ones, maintain a consonance inside agglutinated forms. They deal with the compatibility of the declension of radical and attached suffix.

Example
For morphological analysis of the word "ِ ‫ِﻪ‬ ‫َﺎﺑ‬ ‫ِﺘ‬ ‫"آ‬ (kitAbihi -his book), we start with a first step of segmentation which gives "ِ ‫َﺎب‬ ‫ِﺘ‬ ‫آ‬ " (kitAbi -book in the genitive case) + hi (personal pronoun in the genitive case), in a second step, we apply grammatical rules to validate agglutination of the noun (kitAb -book) to the personal pronoun since they are both declined in the genitive case.
However, the concatenation of the same nominal form (kitAb -book) declined in the accusative case with the same personal pronoun, in the genitive case, makes a prohibited agglutination. So, the agglutinated form "ِ ‫َﻪ‬ ‫َﺎﺑ‬ ‫ِﺘ‬ ‫"آ‬ (kitAba + hi) is refused because of the declension incompatibility between the accusative case of the noun ‫َﺎب"‬ ‫ِﺘ‬ ‫"آ‬ (kitAb -book) and the genitive case of the pronoun as a suffix "ِ ‫"ﻩ‬ (hi -his).

TEST AND EVALUATION
The test of the lexical coverage of our Arabic module is evaluated on lexical analysis of the corpora of the LASELDI collected from Internet. These corpora are composed of journalistic articles of the newspaper "Le monde diplomatique" [17] for five years (2001 -2005), which include about 150 000 different terms. The lexical analysis, of these corpora, shows coverage of about 93% by our lexical and morphological resources. The unrecognized forms are classified in 4 subsets: • 7 000 transliterated named entities : proper names of person such as ‫َاك"‬ ‫ِﻴﺮ‬ ‫"ﺷ‬ (chIrAk -Chirac) with some derived forms such as ‫ﱠﺔ"‬ ‫ِﻴ‬ ‫َاآ‬ ‫ِﻴﺮ‬ ‫"ﺷ‬ (chIrAkiyya -Chiraquism), cities such as ‫َﺎ"‬ ‫ِﻴ‬ ‫ِﻴﻠ‬ ‫ْﺳ‬ ‫َﺮ‬ ‫"ﻣ‬ (marsIliyA -Marseille) and organisations such as "ْ ‫ِﺖ‬ ‫ُﻮﻓ‬ ‫ُوﺳ‬ ‫ْﺮ‬ ‫ِﻜ‬ ‫َﺎﻳ‬ ‫"ﻣ‬ (mAyicrUsUfit -Microsoft), • About 2000 borrowing terms such as " ِ ‫ﻣ‬ ‫َﺎ‬ ‫ِﻳﻘ‬ ‫ِﻴﺰ‬ ‫َﺎﻓ‬ ‫ﻴﺘ‬ " (mItAphIzIkA -metaphysics), • 1 400 spelling mistakes. The most common mistake is the misplacement of the last vowel such as the form "ً ‫َﺪا‬ ‫َﺑ‬ ‫"أ‬ (Ǎabadan -never) where the last vowel "ً " (an) is misplaced, • Some irregular plural forms not yet covered by our study. Actually, we are working on coverage enhancement; on the one hand, the majority of the unrecognized forms are named entities (names of people, organizations or localities) so we are implementing a module of named entity recognition using the NooJ syntactic module (Mesfar 2006); on the other hand, we noticed that some spelling mistakes are caused by badly placed vowels so we are applying morphological grammars with low priority (i.e. applied just in case of an unrecognised form) to start removing the last vowel of the form and if the form remains unrecognized we remove all vowels. We are also checking that the description of the recognized forms is correct using local grammars built with the syntactic module of NooJ.

ENDNOTES
[1] NLP: Natural Language Processing [2] A form is a sequence of graphemes delimited with two white spaces or punctuations in a text. The terms form and word will be used interchangeably.
[3] <N> and <P> commands are used for compound-words transformations [4] We give an example of a grammar represented as a finite state transducer in FIGURE 2 (subsection 5.2. Constraints on syntactic properties of verbs) [5] More informations about the linguistic platform NooJ are available, on-line, at: www.nooj4nlp.net [6] The list of verbs was built by Slim Mesfar and Ibtihal Farawi during their researches on Arabic in the LASELDI.
[7] The classification of verbs on flexional models is described in the book of Abou Il Azm (2003).
[8] FLX: introduces the functionality which describes all the potential verbal forms from a verbal lemma [9] <LW>, <R> and <S> are commands, among ten, preset within NooJ