{'fraction_non_alphanumeric': 0.034415658743179825, 'fraction_numerical': 0.015529016749971383, 'mean_word_length': 4.757908611599297, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 4, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Unified Model for Arabizi Detection and Transliteration using Sequence-to-Sequence Models

While online Arabic is primarily written using the Arabic script, a Roman-script variety called Arabizi is often seen on social media. Although this representation captures the phonology of the language, it is not a one-to-one mapping with the Arabic script version. This issue is exacerbated by the fact that Arabizi on social media is Dialectal Arabic which does not have a standard orthography. Furthermore, Arabizi tends to include a lot of code mixing between Arabic and English (or French). To map Arabizi text to Arabic script in the context of complete utterances, previously published efforts have split Arabizi detection and Arabic script target in two separate tasks. In this paper, we present the first effort on a unified model for Arabizi detection and transliteration into a code-mixed output with consistent Arabic spelling conventions, using a sequence-to-sequence deep learning model. Our best system achieves 80.6% word accuracy and 58.7% BLEU on a blind test set.


Introduction
The term Transliteration generally refers to the process of mapping the orthographic symbols used in one script into another. The types of mapping schemes can range from strict one-to-one transliterations (Beesley, 1997), to named-entity transliteration which can be partially constrained by spelling conventions of the target language (Al-Onaizan and Knight, 2002;Rosca and Breuel, 2016), and all the way to the spontaneous romanizations used in many places around the world, in competition with local languages and spelling traditions. In the Arab world, the latter of these phenomena is often called Arabizi, 1 Since it is used primarily in social media (SMS and chat), it's notoriously noisy and inconsistent although there are some common conventions. And because it is written in the Roman script, it encourages written linguistic code mixing between Arabic and English (or French). Furthermore, since it is not an official orthography, Arabizi is a resource poor language form compared to Modern Standard Arabic and some of its dialects, where there are more resources written in Arabic script. As such, many researchers work on mapping Arabizi to Arabic script as part of general text normalization. This task includes two components: identifying the language, and mapping it to the target script accordingly. For example, the codemixed sequence w aletli Tuesday 'and she-said-to-me Tuesday' would be mapped to Tuesday wqAlt ly Tuesday 2 'and-she-said to-me Tuesday'. While the input and output here have the same number of words, they are not aligned one-to-one as Arabic spelling rules define words boundaries differently, an additional complexity of the task.
There have been a number of efforts in natural language processing (NLP) that worked on this interesting phenomenon using a range of techniques from classical machine learning and n-gram language models (Darwish, 2014; to sequence-to-sequence neural models (Guellil et al., 2017;Younes et al., 2018;Younes et al., 2020). The various previous solutions differ in (a) whether and how they modeled the subtask of Arabizi detection (which words are Arabic written in Roman script, and which are English/French), (b) what amount of context they use, and (c) how they define the Arabic script target (whether to allow token merges and splits or follow Arabizi word boundaries). In this paper we present the first effort on a unified model of Arabizi detection and transliteration into a code-mixed output with consistent Arabic spelling conventions, using neural sequence-to-sequence models. Our best system achieves 80.6% word accuracy and 58.7% BLEU score on a blind test set -9.9%, and 15.9% absolute improvements, respectively, over a simple but robust baseline. Our system's code is public. 3 But the data must be acquired from the LDC (see Section 5).
We discuss related work in Sections 2, and Arabizi challenges and task definition in Section 3. We present our system architecture and approach in Section 4, and our experiments and results in Section 5.

Arabizi Data Collection and Annotation
The various efforts working on Arabizi collected, tagged and mapped different sizes of corpora for different Arabic varieties including Egyptian (Bies et al., 2014;Tobaili, 2016;Chen et al., 2017), Lebanese (Tobaili, 2016), Tunisian (Younes et al., 2015;Masmoudi et al., 2019), andAlgerian (Guellil et al., 2017). We focus here on the work of Bies et al. (2014) since it is the largest by far, and because it targets a wellformed conventional orthography for dialectal Arabic, henceforth ARABIZICORPUS. Their data includes over 287K Egyptian Arabizi SMS and chat words that are automatically transliterated and manually validated. The corpus words are tagged for being Foreign, Name, Punctuation, sound, emoji/emoticon or the default Arabic. The Arabic text written in Arabizi is mapped to Arabic script using the conventional orthography for Dialectal Arabic (CODA) Eskander et al., 2013;. The CODA convention is close to Standard Arabic orthography while maintaining some of the unique morphological and lexical features of Dialectal Arabic. In addition to using different characters, CODA and Arabizi often use different word boundaries. Moreover, Arabizi is noisy and spontaneous while CODA is intended to be conservative and systematic. In addition to the parallel component, the ARABIZICORPUS includes about 1M Arabizi words that are not tagged or transliterated. An earlier version of this data set was used by  and . Unfortunately, since the versions used by these earlier efforts do not correspond to the public version of the data set, and are not determinable from it, we cannot compare to them directly. We report in this paper on the latest public version of the data (Chen et al., 2017), and we describe in detail the splits we follow to enable future comparisons with our results (see Section 5.1).

Pre-neural Models for Arabizi Processing
Chalabi and Gerges (2012) presented a hybrid approach for Arabizi transliteration using manual and learned character mapping rules with a language model for ranking hypotheses. Their work does not address the detection of English words, punctuation, emoticons, and so on. Voss et al. (2014) focus on classifying tokens in Arabizi as Arabic or not. They work on a three-way classification of Moroccan Arabic, French and English. Darwish (2014) was the first effort to model the detection of Arabizi and non-Arabizi words before transliterating the Arabizi text. He employed a two-step system for identification and then conversion. For identification, he used word and sequence-level features with CRF modeling to identify three tags: Arabic, foreign and others. For transliteration, he learned character level mappings from a small parallel corpus and used them to generate alternative mappings, which are filtered using a word trigram language model.  extended the work of Darwish (2014) on transliteration. They trained a finite state transducer at the character level to generate all possible transliterations for the input Arabizi words to output Arabic words. They then filtered the generated list using a dialectal Arabic morphological analyzer. They picked the best choice for each input word using a word 5-gram language model.

Neural Model for Arabizi Processing
Deep learning models, specifically sequence-to-sequence (Seq2Seq) RNN models have shown a lot of success in the task of character-based transliteration for a number of languages (Rosca and Breuel, 2016;Kundu et al., 2018;Dershowitz and Terner, 2020).
In the context of Arabizi, three efforts are particularly notable (Guellil et al., 2017;Younes et al., 2018;Younes et al., 2020). Guellil et al. (2017) and Younes et al. (2018) worked on mapping Algerian Arabizi and Tunisian Arabizi, respectively, to Arabic script. Both used Seq2Seq models at the character level. Younes et al. (2020) redefined the problem as a sequence labeling task using BiLSTM with CRF decoding. All of these approaches were focused on word-level transliteration and did not address issues of code-mixing and Arabizi identification automatically, although they acknowledge these issues. In this paper we present a single unified model that addresses the issues of Arabizi identification and conversion together using a Seq2Seq model.
Although not Arabizi, a recent effort on automatic conversion of Judeo-Arabic text written in Hebrew script to Arabic script is relevant; Dershowitz and Terner (2020) also used a Seq2Seq RNN model and described a number of interesting tricks for addressing length mismatches and forgetfulness in the network. We refer to specific insights from their work in the paper.
Also not Arabizi, but relevant, is the work of Watson et al. (2018), who achieved the current state-ofthe-art on Arabic spelling correction against a standard set using Seq2Seq models. Our baseline Seq2Seq set up is inspired by their work.

Arabizi Challenges and Task Definition
Our task of mapping code-mixed Egyptian Arabizi and English input into code-mixed Egyptian Arabic and English poses a number of challenges.
Arabizi Noise and Ambiguity While there are some common conventions for Arabizi-to-Arabic mapping, they are not strictly followed by far. This is further exacerbated by typical noisy spelling in spontaneous social media text. As a result, we have many-to-many mappings at the word level. For example, the Arabic word Hbyby 'my beloved' appears in the ARABIZICORPUS paired with 24 different Arabizi spellings: (in order of frequency) habibii, hbebe, habibi, 7abiby, 7abibi, hbebee, 7abeby, 7biby, habbii, 7bibi, 7abibii, hbybyy, hbbee, hbbebe, habiibii, habbyy, bbe, 7apipy, 7abiibii, 7abibyy, 7abebe, 7abby, 3abiibii, and 3abibi. Similarly, the Arabizi word arfo is paired with three different Arabic words: ςArfh'I know it', qrfwA 'they made me loathe [something]' and qrfh 'its nastiness'. We mitigate the noise and variability in modeling the Arabizi-to-Arabic transliteration as a character level sequence-to-sequence process. We also make use of word-level embeddings using FastText (Bojanowski et al., 2017) which models subword units within a bigger word-based context. Additionally, we handle common variations in social media text such as inconsistent capitalization and emphatic repetitions through global lower casing and repetition elision to two characters, e.g., Arabizi Habiiiiiibiiiiii is preprocessed to habiibii.
Arabizi-Arabic Mis-alignment The CODA convention we target for Egyptian Arabic does not align word-to-word with Arabizi as explained above. There are both splits and merges of words. In Figure 1, word [5] ageblk 'I bring to you' is split into Ajyb lk, and words [6] and [7] el 'the' swr 'pictures' are merged into one: AlSwr 'the pictures'. We model these two issues in our work by learning to explicitly map the merge ([+]) and split ([-]) special characters used in the ARABIZICORPUS annotations.
Code-switching and Emoji Within our task, English (or other foreign words) are simply to be mapped to the output, without any attempt to normalize or modify their spelling. As such the task is more oriented towards modeling identification of foreign words -or from a different perspective, detection of Arabizi words. This is not always simple given that some Arabizi words are ambiguous with English words, e.g. the Arabizi word men can be the English word 'men' or the Arabic words mn 'from' or myn 'who'. We model this subtask as mapping to a special symbol (#) used to identify which input words to copy to the output in a postprocessing step.
We handle emojis (e.g., ) and emoticons (e.g., :-P) in a comparable way to foreign words. Although for a small set of commonly used emoticons, we use a dictionary and preprocess them into the (#) symbol.
Defining the Task Target The ARABIZICORPUS provides a set of manually assigned tags per each Arabizi word specifying if it is Arabic, Foreign, Name, Sound or Punctuation. In the corpus, all of the words are automatically transliterated and manually validated except for the Foreign word transliterations into Arabic which are not manually validated. This is why  did not do a joint final evaluation but rather evaluated only on the manually validated words and treated identification of foreign words as a separate problem. In our work, we take a different approach where we define the task target to be a code-mix of Arabic (transliterated from Arabizi) and Foreign words in Roman script. We construct this target reference using the Arabizi word tags and the validated transliterations, by reinserting the foreign input word in place of its automatic non-validated transliteration. We also reinsert Emojis which are mapped to a special token (#) in the target side in the ARABIZICORPUS.

Approach
Due to the aforementioned complexities and a high number of out-of-vocabulary words, a word-level neural model cannot be used as an end-to-end solution. So, we opt for a character-level approach using sequence-to-sequence models to capture the complexity of the noise, variations and mistakes.

Data Preprocessing and Postprocessing
Input-side Preprocessing Each Arabizi input utterance goes through the following preprocessing steps: all letters are lowercased, repetition of more than two characters in a word are reduced to two, accented charactes are converted to their unaccented versions in the standard 7-bit ASCII, and all freestanding emojis, emoticons and punctuation are converted to hashtags. These steps can be seen in Figure 1 (ML Input row).
ML Output-side Preprocessing During training only, we convert all the foreign-tagged words to hashtags on the machine learning output side. Since our training input and output are aligned, the conversion ensures that the model learns to identify foreign words and convert them to hashtags. Identical to the is preprocessed before being input into the network. c i is the character embedding, and w j is the word embedding.
input-side preprocessing, the ML output-side free-standing emojis, emoticons and punctuation are converted to hashtags during both training and prediction. We illustrate these steps in Figure 1 (ML Output row). We do not apply AY normalization, i.e., the conversion of different versions of Alif and Ya to simple Alif and Ya, respectively (Habash, 2010;El Kholy and Habash, 2012), because it lowered the normalized accuracy by 0.1% absolute on average when we used it.
ML Output-side Postprocessing On the ML output side, we add a postprocessing step that converts hashtags back to the words that were in the source Arabizi. This is only possible if the input and output are aligned, so this step is applied before removing the [+] and [-] tokens. Moreover, to go to the final output, words with the [+] token are merged to the next word and [-] tokens are replaced with white space to split a word into multiple words. These steps can be seen in Figure 1.

System Architecture
Encoder-Decoder Model We use a character-level sequence-to-sequence architecture following Watson et al. (2018) to model P (y|x) given an input x and a target y. See Figure 2. The encoder consists of two gated recurrent unit (GRU) layers (Cho et al., 2014) with only the first layer being bidirectional following Wu et al. (2016). The decoder also has two GRUs along with the attention mechanism proposed by Luong et al. (2015). The initial states for the decoder layers are learned with a fully-connected tanh layer from the first encoder output.
The model uses scheduled sampling (Bengio et al., 2015) with a constant sampling probability, feeding character embeddings concatenated with embeddings of the words the characters appear in at every time step. The model uses dropout (Srivastava et al., 2014) for both encoder and decoder recurrent neural networks on the non-recurrent connections during training. A final softmax layer receives the decoder output to give the final output sequence y. The loss function is the cross-entropy loss per time step averaged over y i . We use beam search during inference with a fixed beam width to predict candidates with the highest log-likelihood at each step. We pick the individual beam with the highest overall log-likelihood as our prediction. As a final step in inference, we reduce repetitions of six or more text sequences to five repetitions. This addresses rare cases where the decoder misbehaves and produces non-stop repetitions of text.

Model Settings
We used a learning rate of 0.0001, batch size of 1024 and 2048 for different settings (see LINE2LINE vs WORD2WORD below), batch and character embedding sizes of 256, hidden layer size of 256, two RNN layers, dropout probability of 0.1, decoder sampling probability of 0.35, and gradient clipping with a maximum norm of 11.5. We used the Adam optimizer (Kingma and Ba, 2015) with a learning rate of 0.0005, = 1 * 10 −8 , β 1 =0.9, and β 2 =0.999, and trained the model for 40 epochs. When running all the trained models during inference, we used a beam width of 5.
All the word embeddings were trained using Fasttext and had dimension 300. They were trained on the unannotated Arabizi text that was part of the ARABIZICORPUS. In the experiments that included preprocessing, we also preprocessed the input-side Arabizi before training Fasttext. All Fasttext hyperparameters were kept to the default except the context window and minimum n-gram size which were both kept to two.
The network hyperparameters above were empirically identified in a series of ∼80 experiments starting with the hyperparamters of Watson et al. (2018) and tuning to achieve the best accuracy and BLEU score on the dev set.

Line2Line vs Word2Word
During initial experiments, we observed that the model was rather forgetful and not well performing when fed complete Arabizi input utterances (henceforth, we will refer to this Seq2Seq setting as the LINE2LINE setting). So, we considered the option of working at a WORD2WORD level similar to Younes et al. (2018) and Guellil et al. (2017), with the exception of adding a small context window similar to Mubarak et al. (2019), which is not itself mapped, but only provides contextual information. Contextual tokens such as beginning of sentence <bos>, end of sentence <eos>, beginning of word (under focus) <bow>, and end of word (under focus) <eow> were added to aid the model in this setting. We experimented with context windows of size three, two, and one words, and found +/-one word to be the best.
For example, the three-word Arabizi input Alf salama hahaha 'A thousand times safe haha', which is paired with the Arabic Alf slAm hhh, is turned into the following three separate training input-output pairs: For WORD2WORD experiments, we use batch size of 2048 because the input sequences are shorter than the LINE2LINE setup for which we used batch size 1024. For both settings, we use the same character-based Seq2Seq architecture discussed above. The only difference is the size of the input in terms of words and whether context is used.

Experimental Settings
Data We use the BOLT Egyptian Arabic SMS/Chat and Transliteration corpus (ARABIZICORPUS) (Chen et al., 2017;Bies et al., 2014). 4 We split the corpus following the suggestions given by Diab et al. (2013) for Arabic corpora. The documents were first sorted by filename alphabetically. Then, the divisions were applied on the file level, with the divisions being ∼80% training, ∼10% development and ∼10% test, from both chat and SMS subsets of the corpus. The details of the divisions can be seen in Table 1. 5 Metrics We use two metrics to evaluate our setup. The first, harsher metric, measures word-level accuracy of our Seq2Seq model. It compares the source aligned output of the system, which contains the [+] and [-] separation tokens, to the source aligned gold. In this regard, we measure both the exactmatching accuracy and the AY-normalized accuracy. After we remove the separation tokens, the number of words in the output may differ from the input; this is why we use BLEU (Papineni et al., 2002), which is generally used for translation evaluation, to measure the quality of the transliteration at the end. We measure the BLEU score for both the exact final output and the AY-normalized final output.
Baseline For a baseline, we use Maximum Likelihood Estimate (MLE) to predict the most likely output (Arabic word or hashtag) as seen in training given an input word. This simple baseline experiment is conducted with and without preprocessing.

Experimental Results
Development Results Results for the development set are shown in Table 2. We see that the WORD2WORD setup with a context window of +/-one word outperformed the LINE2LINE setup and the baseline MLE. In all setups, preprocessing always helped the model. The best setting (WORD2WORD +/-1 Context + Preprocessing) improves over the strong MLE baseline (with Preprocessing) by 7.5% absolute accuracy and 14.5 BLEU points (in exact matching space).

Blind Test Results
For the blind test set, Table 3 presents the results on our best model determined above as and the MLE baseline. The results are consistent with the observations seen in development. The best model improves over the strong MLE baseline by 9.9% absolute accuracy and 15.9 BLEU points (in exact matching space).

Speed vs Quality
We would also like to comment on the trade off between speed and quality. In our experiments, we saw that the faster systems had lower quality and vice versa. The MLE system was extremely fast but had much lower quality compared to the WORD2WORD system which was quite slow. Upon a closer look at the results, we saw that the best MLE setup did almost as well on words seen in training as the best Seq2Seq setup, and Seq2Seq easily outperformed MLE on unseen words. This led us to the idea of combining the two systems to create a hybrid system where the MLE model would predict words seen in training and the Seq2Seq model would predict unseen words. This did not affect training time because we used the pre-trained models; however, it decreased prediction time from 31 minutes to 4.5 minutes on the development set. The accuracy and BLEU score of this hybrid system was lower than the WORD2WORD model but higher than the baseline MLE model (See Table 4).

Error Analysis
We manually classified a sample of 50 development set utterances (50 lines, 205 Arabizi words) from the best MLE model and from the best Seq2Seq model. We grouped the output words into Acceptable and Unacceptable sets. The Acceptable set includes correct matches, as well as acceptable transliteration variants and minor CODA errors; while the Unacceptable set includes Arabizi, and wrong implausible outputs.

Negative Result Experiments
We report next on three different streams of experiments that gave us negative results.
Synthetic Data Inspired by successful results in machine translation, we considered the use of additional synthetic data to mitigate the lack of parallel training data (Sennrich et al., 2016). We initially considered a reverse transliteration model that would allow us to take large amounts of Egyptian Arabic text and map it to Arabizi space. However this was not simple and we quickly noticed a number of issues: (a) most of the available data is not in the chat and SMS genre, and (b) working from Arabic text eliminates examples of Arabizi style code-switching and tokenization. Instead, we opted to use the additional unannotated data we have from the ARABIZICORPUS (∼242K Arabizi utterances containing ∼1M words). We used our best model to generate a synthetic Arabizi-Arabic parallel training set, and add it to our initial corpus. The results were negative on all used metrics. We hypothesize that the high degree of noise in the input may have resulted in weak or worse, very noisy, models. This may be consistent with the limited benefits of using the additional corpus as part of FastText pre-trained word embeddings.

Foreign Language Embeddings
Intuiting that the quality of Arabizi vs Foreign detection can be improved using additional models of English text as was successfully shown by  in their non-neural models. We first took about 1M words (to match the size of data we have for Arabizi) from an English chat and SMS corpus (Chen et al., 2018) and trained another set of word embeddings using FastText; these word embeddings were concatenated to the already existing character embeddings and Arabizi word embeddings in the WORD2WORD setup. This did not show an improvement in terms of the used metrics. In a different experiment, we inserted a random sample of 21K words (or about 10% of training data, to avoid biasing the model too much) into random locations throughout the existing training data and paired those additions with the proper # symbol on the target side. This resulted in a slight decrease in scores, so we did not use it.

Modeling Different Input Sizes and Context Windows
Since sequence-to-sequence models are known to struggle with very long input sequences, we considered a solution similar to Dershowitz and Terner (2020), where we break the input and target into smaller sequences of size N words during training. During inference, we do the same for the input, then concatenate the predicted output before evaluation. We considered different sizes of N and found the minimal limit of N = 1 words to be the best performing system.

Conclusion and Future Work
We presented a character-level Seq2Seq model for Arabizi detection and transliteration together into a code-mixed output with a consistent Arabic spelling convention. Our best system improves over the baseline by 9.9% in word accuracy and 15.9 BLEU points on a blind test set. We report on a number of experiments including some negative results.
In the future, we plan to integrate our best model in an open source toolkit for supporting Arabic NLP. Furthermore, we want to work on Arabizi text from a range of Arabic dialects. Finally, we want to integrate our system into other Arabic NLP applications, such as machine translation from Arabizi to demonstrate the relevance of Arabizi-to-Arabic transliteration.