{'fraction_non_alphanumeric': 0.026446400753541283, 'fraction_numerical': 0.011013295656269247, 'mean_word_length': 4.647299509001637, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Language Informed Modeling of Code-Switched Text

Code-switching (CS), the practice of alternating between two or more languages in conversations, is pervasive in most multi-lingual communities. CS texts have a complex interplay between languages and occur in informal contexts that make them harder to collect and construct NLP tools for. We approach this problem through Language Modeling (LM) on a new Hindi-English mixed corpus containing 59,189 unique sentences collected from blogging websites. We implement and discuss different Language Models derived from a multi-layered LSTM architecture. We hypothesize that encoding language information strengthens a language model by helping to learn code-switching points. We show that our highest performing model achieves a test perplexity of 19.52 on the CS corpus that we collected and processed. On this data we demonstrate that our performance is an improvement over AWD-LSTM LM (a recent state of the art on monolingual English).


Introduction
Code Switching (CS) is a widely studied linguistic phenomenon where two different languages are interleaved. This occurs within multilingual communities (Poplack, 1980;Myers-Scotton, 1997;Muysken, 2000;Bullock and Toribio, 2009). Typically one language (the matrix language) provides the grammatical structure for CS text and words are inserted from another language (the embedded language).
However, CS data is quite challenging to obtain because this phenomena is usually observed in informal settings. Data obtained from online sources are often noisy because of spelling, script, morphological, and grammatical variations. These sources of noise make it quite challenging to build robust NLP tools (Çetinoğlu et al., 2016). Our goal is to improve LM for Hindi-English codemixed data (Hinglish) where similar challenges are apparent. The task of language modeling is very important to several downstream applications in NLP including speech recognition, machine translation etc., This task also aids in multi-task learning where parameters are shared between the language model and the other task. This is particularly important in domains such as codeswitching, that lack annotated data, where the necessity to leverage unsupervised techniques are a must. In this report, we present our attempt to this problem using two techniques: (1) Factored language model that contains the factors of word and language given to a stacked LSTM based architecture, and (2) Posing this as a multi-task learning (MTL) problem with a dual objective including predicting the next word, and predicting the language of the next word. Learning to predict the language of the next word allows the model to switch points between languages at global level. In the path towards building the MTL architecture, we attempted to explicitly learn the language of the current word without predicting the language of the next word. This is one of the model in our ablation studies, which is similar to and is inspired from the factored LM.
In addition to the techniques used for monolingual language modeling, providing information about the language is a key component in CS domain. Our main goal in this paper is to examine the effect of language information in modeling CS text. We approach this systematically by experimenting with ablations of encoding and decoding language id along with the word itself. In this way, the model implicitly learns the switch points between the languages. We achieve the least perplexity score in combination with a language informed encoder and a language informed decoder among the ablation of multiple models.
The current material begins with a review of LM techniques for CS text in section 2. Then we describe our data collection and processing steps in Section 3 and model architecture in Section 4. We present a brief quantitative and qualitative discussion of our observations in Section 5. Finally in section 6 we propose promising directions for future work and conclude in section 7.

Related Work
Early efforts to develop computational framework for CS data include Joshi (1982) in the 1980s and Goyal et al. (2003); Sinha and Thakur (2005); Solorio and Liu (2008a,b) in the 2000s. However, these methods are quite limited in their applicability to the kind of data we see on the Internet and social media where code-switching data is usually found. This has led to an increased attention of the NLP community in the areas of Language Modeling Fung, 2013, 2014;Adel et al., 2015Adel et al., , 2013aGarg et al., 2017), POS tagging (Vyas et al., 2014;Jamatia et al., 2015;Çetinoğlu and Çöltekin, 2016), language identification (King and Abney, 2013) prediction of code-switch points (Das and Gambäck, 2014), sentiment analysis (Rudra et al., 2016) and also certain meta level studies that include understanding metrics to characterize code-mixing (Patro et al., 2017;Guzmán et al., 2017).

Neural Language Models
Neural Language Models have a limitation in capturing only a finite context. This was overcome in Recurrent Neural Network (RNN) based Language Model (Mikolov et al., 2010). The RNN has an input layer x, hidden layer s (also called context layer or state) and output layer y. What allows for capturing of an infinite context is that the input to the network at time t i.e. x(t) is concatenated with s(t − 1) i.e. the output of hidden layer at time t−1 to compute the current state s(t) and output y(t) which is probability of the next word given the context. Several word-level tasks can be tackled better at the constituent character-level (Zhang et al., 2015;Chung et al., 2016). Character-Level Language Models (Kim et al., 2016) represent a word w as a collection of the embeddings of the constituent characters C w = [c 1 , c 2 , · · · , c l ] where l is the length of w. C w is then passed through a Convolution Neural Network(CNN) with varying filter widths to capture different lengths of character n-grams. Essentially, if there are h filters, then the feature mapping for w is y w = [y 1 , y 2 , · · · , y h ]. This feature mapping instead of word embeddings is fed to the RNN-LM as described above, and rest of the procedure remains the same. Adel et al. (2015) suggest using syntactic and semantic features found in CS text into a factored language model(FLM) that are indicative of a switch e.g. trigger words, trigger POS tags, brown cluster of function and content words, resulting in significant reduction in perplexity. (DLM) (Garg et al., 2017) combines two monolingual language models by using a probabilistic 'switch' between them.

Code-Switched Language Models
There has been some recent focus on adapting existing language models for CS text. Fung, 2013, 2014) use a translation model together with the language model of the matrix language to model the mixed language. The search space within the translation model is reduced by linguistic features in CS texts like inversion constraint and functional head constraint (Sankoff and Poplack, 1981).
In another approach Adel et al. (2015), use a Factored Language Model (FLM) that includes syntactic and semantic features found in CS text that are indicative of a switch e.g.

Train
Dev  (Garg et al., 2017), combines two monolingual language models by introducing a 'switch' token common to both languages. Predicting this word in either languages acts a proxy to the probability of a switch and the next word is then predicted using the LM of the language that was switched to.
Among neural methods, (Adel et al., 2013a) use of a Recurrent Neural Network based LM to predict the language of the next word along with the actual word to model CS text. Following on these intuitions, our models are built on top of the AWD-LSTM LM (Merity et al., 2017) that was chosen due to its accessibility and high performance (recently State of the Art) on the Penn-Tree Bank and Wikitext-2 dataset (Merity et al., 2016). Extensive work has been done on this model through investigation on relative importance of hyper-parameters (Merity et al., 2018).

Data Collection and Analysis
To the best of our knowledge, there is no standard dataset to evaluate LM in codeswitched texts. In this section, we will describe about our data collection and a brief analysis of it.

Data Collection
Curating a reasonable dataset for CS text is an important challenge for researchers in this domain. To the knowledge of the authors, there is no benchmark CS corpus for language modeling as there is for English (Merity et al., 2016;Marcus et al., 1994). CS is commonly observed in informal settings and in casual conversations. Hence the two potential source choices to gather data include social media (such as Twitter and Facebook) and blogging websites. We decided to go with the latter due to comparatively lesser noise and availability of more descriptive text. Our CS LM data was collected after having crawled eight blogging Hinglish websites 1 that were returned by popular search engines (such as Google and Bing) with simple code-switched queries in the domains of health and technology. These CS texts cover several different topics, primarily technical reviews of electronic and general e-commerce products as well as several health related articles. These texts were all tokenized at the sentence level and lexical language identification was performed. All the sentences that did not have at least one word each from both languages were discarded to channel our problem towards tackling intra-sentential code-switching. This resulted in a total of 59,189 unique sentences. The data needed extensive cleaning due to a lot of hyperlinking text.

Data Analysis
To estimate the quality and extent of mixing and frequency of switching in our data, we measured Multilingual index (M-Index), Figure 1: Architecture of our baseline Factored LM for Hinglish data Language Entropy and Integration index (Iindex) that were introduced in the domain of CS by (Guzmán et al., 2017). A multilingual index of 1 indicates that there is equal extent of mixing from both the participating languages. As we can observe, the mixing is nearly in the range of 0.8 which indicates that both Hindi and English are participating in the ratio of 4 is to 5 respectively. The metric itself does not reveal about the which is the embedded language and which is the matrix language. These metrics along with other n-gram statistics over our data are presented in Table 1. Note that the CS metrics for each of the train, validate and test splits of the data are almost the same, indicating a similar extent of mixing in them.

Pre-processing
One of the important characteristics of codeswitched text in Hinglish when written in Roman script is the non-standard representations of the words. This is very commonly observed since there are strict guidelines that monitor the correctness of an approximately phonetically represented word. There is no standard one to one correspondence of syllables in Hindi when written in Roman script, that universally everyone would follow in informal settings. Hence the same word could be written in multiple representations based on the idiosyncrasies of the individual and perception of Romanization of a syllable to that specific person.
For both the Factored LM and language informed encoder models, we perform lexical language identification for each of the words and annotate them with this information.
The non-standardized representations of the words are dealt in the following two ways: • Soundex Encodings: We use soundex encodings of the words as another factor. This way, the idiolectic representational variations with an additional 'h' for aspirated sounds and single vs multiple vowels for long and short syllables etc, would be mapped to the same space. This is not a solution to normalizing the representational variations but we hypothesize that this feature helps in learning context around variants of the same word better.
• Transliteration: Along the similar lines of standardizing the multiple representations of the same word, the words that are identified as Hindi at the lexical level are transliterated into their Devanagari representation and the most likely Devanagari spelling in a preexisting Devanagri dictionary is chosen according to soundex encoding.

Models and Experiments
In this section, we describe the process of step by step building of our final model to perform language modeling for codeswitched text that performs comparatively better than the remaining models. The path to this is roughly along the lines of incorporating the language information in the model along with the word information to improve the performance overall.
• With this intuition, we start with a factored LM that takes both language and words as factors to an LSTM based architecture of a language model.
• We then move onto improving the core of the LSTM by adapting the state-ofthe-art model (Merity et al., 2018).
• We improve this model by incorporating lexical level language information before encoding the input and decoding the output for the next word. This is similar to an MTL setting where the two tasks are the predicting the next word and predicting the language of the next word. Both the losses back propagate. Note that only the loss at the word decoder is used to calculate perplexity of the model. We also perform ablations with the combinations of giving language information of the current word and predicting the language of the next word.

Stage 1: Factored Language Model
Each word in the vocabulary V is represented as embeddings of 3 factors: token, soundex encoding, language information. We build an LSTM based LM with long and short term dependencies between switch points and partially normalized tokens. The model trained uses 3 embedding layers: one learned from pure vocabulary (dimension of 128), one learned from the language label (dimension of 2) and one learned from the soundex labels (dimension of 64) at training time these embeddings are concatenated and fed into an single layered LSTM. A linear decoding layer is then used to transition from the LSTM's representation into the vocabulary space (Figure 1). The model is trained using the Categorical Cross Entropy Loss and the ADAM optimizer (Kingma and Ba, 2014).

Stage 2: Improving the core of LSTM architecture
In Stage 1, the model has a single layer LSTM where the input representation from the concatenation of individual factors are fed as input. In the second stage, we improve the inner LSTM model itself based on SOTA for WikiText-2 (Merity et al., 2018). We have built a word level language model based on (Merity et al., 2017). Although the huge number of parameters in neural models enable them to learn high degree of nonlinearities present in the data, they affect the generalization capabilities of the model. In spite of techniques such as batch normalization and dropout that are generally used to regularize training in neural networks, the improvement in performance has not been substantial in the case of sequential units such as recurrent models. The nuances that this model addresses are as follows: First, coming to the problems of a default usage of dropout to the recurrent units that inhibits maintenance of longer term dependencies. One solution for this is to use the same dropout mask at each time step, known as Locked Dropout. Another solution is the application of dropout for specific network units such as certain gates or states rather than the entire unit. Second kind of regularization techniques involve different kinds of normalization that directly impacts the training process due to the induction of more trainable parameters.
Finally an effective optimization algorithm is needed in synergy with when the dropout is applied. This paper uses ASGD (Averaged Stochastic Gradient Descent) that returns an average across the iterations that have crossed a specified loss threshold value, which is decided using a non-monotonic criterion.
The distinguishing factors with this model are the regularization and optimization strategies that are applied on the LSTM based models built upon their core form.

Stage 3: Language Informed Reading and Decoding
This is built upon the idea of Stage 1, where we factorize the language id of the input word and decode the next word. This technique is incorporated with the model described in Stage 2 with AWD-LSTM. In addition to this, we pose this as a multi-task learning problem, with shared layers except for the decoder. The 2 tasks are the following: 1. Predict the next word.
2. Predict the language of the next word.
There are a number of ways to frame the desire for humans to switch between languages (Skiba, 1997;Moreno et al., 2002), however, we view the human desire as out of scope for this work. Instead, our focus is on how we can incorporate linguistic information while training a statistical model for code-switched text. We discuss two main choices as to where we can introduce this information: either at the input stage or at the decoding stage of an RNN language model. Given a CS sentence X cs = (x 1 , x 2 ... , x n ) which has lexical level language sequence L cs = (l 1 , l 2 ... , l n ), our model has to predict the word at the next time step. Note that this vector l i is the language of the ith lexical item and is represented as a vector of length sixteen which is trained in concert with the model. This allows our model to encode the distributional properties of the language switching. We experimented with encoding and decoding the word and language embeddings for this task. θ E X , θ E L , θ D X and θ D L are the parameters for the word encoder, language encoder, word decoder and language decoder respectively.
We identify four different model architectures ( Figure 2) that could be useful in training code switched language models. In the first model, our baseline, we have a sequence of words and we are trying to predict the following word. This model is identical to running a traditional RNN language model on CS text. For our baseline model we adapt the stateof-the-art language model, the AWD-LSTM, for this domain. This model is a 3 layered stacked LSTM trained via Averaged SGD with tied weights between the embedding and the softmax layer. There are several other important elements of this model, all of which are detailed in (Merity et al., 2017). The next word in this model is given by: (Xcs, θE) In our second model we extend our baseline such that we have a sequence of words  and their language IDs and we are trying to predict the following word. This model can be seen as a factored language model operating with code switched data. So, the next word in this model is given by: In our third model we take a sequence of words as an input and attempt to predict both the language and the value of the following word. The next word in this model is given by: In our fourth model we take a sequence of words and their corresponding language IDs as input and attempt to predict both the language and value of the subsequent word. In our third and fourth models we operate with two loss values being calculated for (one for the word error, and one for the language error multiplied by 0.1) and gradients for both losses are propagated through the network and are used to update the weights.

Results and Error Analysis
We trained 4 different models based on the description in Section 4. The results of these experiments are presented in Table 2. We observe that the Language Aware Encoding and Decoding with the AWD-LSTM gives the least perplexity. This aligns with our hypothesis that providing language information of the current word before encoding and enabling the model to decode the language of the next word allows the model to learn a higher level context of switch points between the languages.

Qualitative Analysis
We also did a qualitative analysis of these LMs with the sentences they generate, some generation examples of the baseline AWD-LSTM are shown in table 3. We observe them to be quite consistent in generating meaningful code-mixed n-grams of length 10 and above.
However, the sentences that got generated were not very coherent. Part of the reason for this is training sentences themselves did not have proper punctuation and endmarkers, which are issues to take care of during pre-processing. Notice that the in the first two examples, the first word is connective (in specific a conjunction), and hence expects their respective sub-ordinate and coordinate parts of the sentences that are missing.
Also, as seen in the second word of the last sentence, which is the transliterated form of the word for 'mei' in Hindi (which is the Hindi word for 'in'). In this context, the word that is closest in lexical form should have been 'main' (which is the Hindi word for 'I'). Using transliteration as a proxy to standardize the representations of the words is the reason for the model to make such errors. Character level models have been working well for spell corrections and normalization and incorporating character level convolution as input along with the word embeddings could potentially help solve this problem.
In this perspective, we have attempted character level CNN based model along with an LSTM at the top but this did not perform very well. Further experimentation of this model with MTL set up by decoding the language is work in progress. Let c 1 , c 2 ... c p be the padded sequence of characters of a word. Each of the characters has a 50 dimensional embedding (let it be e). Let c i:i+j be अगर आप android lollipop और android phone से connected ह gloss: if you android lollipop and android phone to connected are trans: if you are connected to android lollipop and android phone और आप us android phone के बारे मे share कर रहे ह gloss: and you that android phone regarding share doing are trans: and you are sharing regarding that android phone आज मे अपने blog के लए ब त ही फाय डम ड option ले कर आया hun जस पर सभी high quality backlink का post लखी gloss: today I my blog for very beneficial option bring here on which all high quality backlink 's post written trans: today I bring a very beneficial option for my blog on which all high quality backlink's post written Table 3: Sample Code-Mixed generation by AWD-LSTM along with gloss and translation (text in red are Hindi words that weren't identified correctly in the LID step. ) the concatenation of characters from i to j. Next we take 200 filters of size sXe, where s is the window size which in our case is 3. The feature in the CNN layer is computed and activated using a Relu which gives a feature map of size p − s + 1 over which mean pooling is performed. After the end of 3 convolution layers, each word is represented in a 200 dimensional hidden convoluted character space which is concatenated with the corresponding 200 dimensional word embedding. This latent representation is used to decode the next word.
In another qualitative analysis we observe the t-sne plots of the Hindi and English words from the embedding layer separately. For the English terms, the content words like infographics, click, cyber, blueborne are related to blogs and tend to be grouped closer. While in case of the Hindi terms, verbs like banaya(made),aaoge(-come), karte hain(are doing) and function words are grouped closer. This is understandable because of the matrix language being in Hindi. So, the Hindi words made up bulk of the syntax contained in these sentences with English content words sprinkled in between. Also, because the English content words are separated by a long distance, they do not influence the meanings of one another in the context. For the same reason we did not find any noticeable correlation between the meanings of Hindi and English word embeddings because they played very different roles in sentences.

Challenges: Discussion
Robustness of the language model also depends on the diversity of context in which the words co-occur. Since most of the arti-cles belong to the topics of e-commerce, latest technology and health, this may be affected. Hence, we plan to use pre-trained word embeddings based on large monolingual corpora after aligning the embedding spaces of both the participating languages such as MUSE embeddings (Conneau et al., 2017). However, due to the non-standardized spellings in the romanized Hinglish text, most words that are incorrectly transliterated are not found in the MUSE embeddings. This implies that the errors from transliteration are propagated through the subsequent parts of model. To avoid this, we plan to extend this work by using character encodings in future.

Future Work
There are a number of different avenues that could be explored to improve this work. While we aimed primarily to incorporate language information along with the word input in this work, further work could be a done to incorporate other factors such as parts of speech, & root words etc,. Additionally, our baseline model achieved its high results as a result of significant hyperparameter tuning to the Wikitext2 dataset. This tuning was not performed on this data and could represent a significant avenue for improvement. Lastly, and arguably most importantly, the collection of additional CS data would be a significant contribution to this work. Much of the work involved in this project was to properly clean, parse, and represent the CS data that was scraped from the online sources discussed above. These sources remain limited in topic and variation and additional sources of CS data would be the best way to improve how well our model can generalize. We would also explore the effect of including sentence level features composed of corresponding code-switching metrics at the sentence level which conditions the model to generate CS text.
Additionally, the probabilities given by the language decoder could be tuned towards using as bias to the word decoder model to predict the next word. These probabilities can be multiplied with the probabilities of the vocab which can be used as bias term for predicting the word. This cross product as bias term reduces the probabilities of the Hindi word if the language probability of English is higher and vice versa.

Conclusion
We hypothesize that incorporating the information of language aids in building more robust language models for code-switched text. In this context, we substantiate our hypothesis by experimenting with different combinations of providing the language of the current word as input and decoding the language of the next word along with th word itself. We conclude that we are able to improve the State-of-The-Art language model for monolingual text by both explicitly providing the language information and decoding the language of the next word to perform this task for CS domain. We treat this problem as a multi-task learning problem where the same embedding and LSTM layers are shared. These two comparable tasks are predicting the next word and predicting the language of the next word. So far, our best test perplexity is 18.51 on development and 19.52 on test sets. This is in comparison to the baseline model which is 19.73 and 20.92 on dev and tests respectively. We are of the opinion that further research can be done to not only improve perplexity, but to also improve the quality of the training and testing dataset. Language models are a core element in multiple generation tasks, from speech recognition to machine translation, we believe that this work will support future research into the development of such NLP tools for CS domain.