{'fraction_non_alphanumeric': 0.03292236824278553, 'fraction_numerical': 0.011177347242921014, 'mean_word_length': 4.730978260869565, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Combination of Recurrent Neural Networks and Factored Language Models for Code-Switching Language Modeling

In this paper, we investigate the application of recurrent neural network language models (RNNLM) and factored language models (FLM) to the task of language modeling for Code-Switching speech. We present a way to integrate partof-speech tags (POS) and language information (LID) into these models which leads to significant improvements in terms of perplexity. Furthermore, a comparison between RNNLMs and FLMs and a detailed analysis of perplexities on the different backoff levels are performed. Finally, we show that recurrent neural networks and factored language models can be combined using linear interpolation to achieve the best performance. The final combined language model provides 37.8% relative improvement in terms of perplexity on the SEAME development set and a relative improvement of 32.7% on the evaluation set compared to the traditional n-gram language model.


Introduction
Code-Switching (CS) speech is defined as speech that contains more than one language ('code'). It is a common phenomenon in multilingual communities (Auer, 1999a). For the automated processing of spoken communication in these scenarios, a speech recognition system must be able to handle code switches. However, the components of speech recognition systems are usually trained on monolingual data. Furthermore, there is a lack of bilingual training data. While there have been promising research results in the area of acoustic modeling, only few approaches so far address Code-Switching in the language model. Recently, it has been shown that recurrent neural network language models (RNNLMs) can improve perplexity and error rates in speech recognition systems in comparison to traditional n-gram approaches (Mikolov et al., 2010;Mikolov et al., 2011). One reason for that is their ability to handle longer contexts. Furthermore, the integration of additional features as input is rather straightforward due to their structure. On the other hand, factored language models (FLMs) have been used successfully for languages with rich morphology due to their ability to process syntactical features, such as word stems or part-of-speech tags (Bilmes and Kirchhoff, 2003;El-Desoky et al., 2010). The main contribution of this paper is the application of RNNLMs and FLMs to the challenging task of Code-Switching. Furthermore, the two different models are combined using linear interpolation. In addition, a comparison between them is provided including a detailed analysis to explain their results.

Related Work
For this work, three different topics are investigated and combined: linguistic investigation of Code-Switching, recurrent neural network language modeling and factored language models. In (Muysken, 2000;Poplack, 1978;Bokamba, 1989), it is observed that code switches occur at positions in an utterance where they do not violate the syntactical rules of the involved languages. On the one hand, Code-Switching can be regarded as a speaker dependent phenomenon (Auer, 1999b;. On the other hand, particular Code-Switching patterns are shared across speakers (Poplack, 1980). It can be observed that part-of-speech tags may predict Code-Switching points more reliable than words themselves. The authors of (Solorio et al., 2008a) predict Code-Switching points using several linguistic features, such as word form, language ID, part-of-speech tags or the position of the word relative to the phrase (BIO). The best result is obtained by combining those features. In (Chan et.al., 2006), four different kinds of n-gram language models are compared to predict Code-Switching. It is discovered that clustering all foreign words into their part-of-speech classes leads to the best performance. In the last years, neural networks have been used for a variety of tasks, including language modeling (Mikolov et al., 2010). Recurrent neural networks are able to handle long-term contexts since the input vector does not only contain the current word but also the previous hidden layer. It is shown that these networks outperform traditional language models, such as n-grams which only contain very limited histories. In (Mikolov et al., 2011), the network is extended by factorizing the output layer into classes to accelerate the training and testing processes. The input layer can be augmented to model features, such as partof-speech tags (Shi et al., 2011;. In , recurrent neural networks are applied to Code-Switching speech. It is shown that the integration of POS tags into the neural network, which predicts the next language as well as the next word, leads to significant perplexity reductions. A factored language model refers to a word as a vector of features, such as the word itself, morphological classes, POS tags or word stems. Hence, it provides another possibility to integrate syntactical features into the language modeling process. In (Bilmes and Kirchhoff, 2003), it is shown that factored language models are able to outperform standard n-gram techniques in terms of perplexity. In the same paper, generalized parallel backoff is introduced. This technique can be used to generalize traditional backoff methods and to improve the performance of factored language models. Due to the integration of various features, it is possible to handle rich morphology in languages like Arabic or Turkish (Duh and Kirchhoff, 2004;El-Desoky et al., 2010).

Motivation
Since there is a lack of Code-Switching data, language modeling is a challenging task. Traditional n-gram approaches may not provide reliable estimates. Hence, more general features than words should be integrated into the language models. Therefore, we apply recurrent neural networks and factored language models. As features, we use part-of-speech tags and language identifiers.

Using Recurrent Neural Networks As Language Model
This section describes the structure of the recurrent neural network (RNNLM) that we use as Code-Switching language model. It has been proposed in  and is illustrated in figure 1. Figure 1: RNNLM for Code-Switching (based upon a figure in (Mikolov et al., 2011)) Vector w(t), which represents the current word using 1-of-N coding, forms the input of the recurrent neural network. Thus, its dimension equals the size of the vocabulary. Vector s(t) contains the state of the network and is called 'hidden layer'. The network is trained using backpropagation through time (BPTT), an extension of the back-propagation algorithm for recurrent neural networks. With BPTT, the error is propagated through recurrent connections back in time for a specific number of time steps t. Hence, the network is able to remember information for several time steps. The matrices U 1 , U 2 , V , and W contain the weights for the connections between the layers. These weights are learned during the training phase. Moreover, the output layer is factorized into classes which provide language information. In this work, four classes are used: English, Mandarin, other languages and particles. Vector c(t) contains the probabilities for each class and vector y(t) provides the probabilities for each word given its class. Hence, the probability P (w i |history) is computed as shown in equation 1.
It is intended to not only predict the next word but also the next language. Hence according to equation 1, the probability of the next language is computed first and then the probability of each word given the language. Furthermore, a vector f (t) is added to the input layer. It provides features (in this work part-of-speech tags) corresponding to the current word. Thus, not only the current word is activated but also its features. Since the POS tags are integrated into the input layer, they are also propagated into the hidden layer and backpropagated into its history s(t). Hence, not only the previous feature is stored in the history but also features from several time steps in the past.

Using Factored Language Models
Factored language models (FLM) are another approach to integrate syntactical features, such as part-of-speech tags or language identifiers into the language modeling process. Each word is regarded as a sequence of features which are used for the computation of the n-gram probabilities. If a particular sequence of features has not been detected in the training data, backoff techniques will be applied. For our task of Code-Switching, we develop two different models: One model with only part-of-speech tags as features and one model including also language information tags. Unfortunately, the number of possible parameters is rather high: Different feature combinations from different time steps can be used to predict the next word (conditioning factors), different backoff paths and different smoothing methods may be applied. To detect useful parameters, the genetic algorithm described in (Duh and Kirchhoff, 2004) is used. It is an evolution-inspired technique that encodes the parameters of an FLM as binary strings (genes). First, an initializing set of genes is generated. Then, a loop follows that evaluates the fitness of the genes and mutates them until their average fitness is not improved any more. As fitness value, the inverse perplexity of the FLM corresponding to the gene on the development set is W t-1 P t-1 P t-2 W t-1 P t-2 W t-1 P t-1 P t-2 W t-1 P t-1 unigram Figure 2: Backoff graph of the FLM used. Hence, parameter solutions with lower perplexities are preferred in the selection of the genes for the following iteration. In (Duh and Kirchhoff, 2004), it is shown that this genetic method outperforms both knowledge-based and randomized choices. For the case of part-of-speech tags as features, the method results in three conditioning factors: the previous word W t−1 and the two previous POS tags P t−1 and P t−2 . The backoff graph obtained by the algorithm is illustrated in figure 2. According to the result of the genetic algorithm, different smoothing methods are used at different backoff levels: For the backoff from three factors to two factors, Kneser-Ney discounting is applied. If the probabilities for the factor combination W t−1 P t−2 could not be estimated reliably, absolute discounting is used. In all other cases, Witten-Bell discounting is applied. An overview of the different smoothing methods can be found in (Rosenfeld, 2000).

POS Tagger for Code-Switching Speech
To be able to assign part-of-speech tags to our bilingual text corpus, we apply the POS tagger described in (Schultz et al., 2010) and . It consists of two different monolingual (Stanford log-linear) taggers (Toutanova et al., 2003;Toutanova et al., 2000) and a combination of their results. While (Solorio et al., 2008b) passes the whole Code-Switching text to both monolingual taggers and combines their results using different heuristics, in this work, the text is splitted into different languages first. The tagging process is illustrated in figure 3. Mandarin is determined as matrix language (the main language of an utterance) and English as embedded language. If three or more words of the embedded language are detected, they are passed to the English tagger. The rest of the text is passed to the Mandarin tagger, even if it contains foreign words. The idea is to provide the tagger as much context as possible. Since most English words in the Mandarin segments are falsely tagged as nouns by the Mandarin tagger, a postprocessing step is applied. It passes all foreign words of the Mandarin segments to the English tagger in order to replace the wrong tags with the correct ones.

Evaluation
For evaluation, we compute the perplexity of each language model on the SEAME development and evaluation set und perform an analysis of the different back-off levels to understand in detail the behavior of each language model. A traditional 3gram LM trained with the SEAME transcriptions serves as baseline.

LM Performance
The language models are evaluated in terms of perplexity.  It can be noticed that both the RNNLM and the FLM model outperform the traditional 3-gram model. Hence, adding syntactical features improves the word prediction. For the FLM, it leads to no improvement to add the language identifier as feature. In contrast, clustering the words into their languages on the output layer of the RNNLM leads to lower perplexities.

Backoff Level Analysis
To understand the different results of the RNNLM and the FLM, an analysis similar to the one described in (Oparin et al., 2012) is performed. For each word, the backoff-level of the n-gram model is observed. Then, a level-dependent perplexity is computed for each model as shown in equation 2.
In the equation, k denotes the backoff-level, N k the number of words on this level, w k the current word and h k its history. Table 3 shows how often each backoff-level is used and presents the leveldependent perplexities of each model on the development set.  In case of backoff to the 2-gram, the FLM provides the best perplexity, while for the 3-gram and backoff to the 1-gram, the RNNLM performs best. This may be correlated with the better over-all perplexity of the RNNLM in comparison to the FLM. Nevertheless, the backoff to the 2-gram is used about twice as often as the backoff to the 1-gram or the 3-gram.

LM Interpolation
The different results of RNNLM and FLM show that they provide different estimates of the next word. Thus, a combination of them may reduce the perplexities of table 2. Hence, we apply linear interpolation to the probabilities of each two models as shown in equation 3.
The equation shows the computation of the pobability for word w given its history h. P M 1 denotes the probability provided by the first model and P M 2 the probability from the second model.  to the 3-gram model. While the RNNLM performs better for the 3-gram and for the backoff to the 1gram, the FLM performs the best in case of backoff to the 2-gram which is used more often than the other levels (table 3).

Conclusions
In this paper, we presented two different methods for language modeling of Code-Switching speech: Recurrent neural networks and factored language models. We integrated part-of-speech tags and language information to improve the performance of the language models. In addition, we analyzed their behavior on the different backoff levels. While the FLM performed better in case of backoff to the 2-gram, the RNNLM led to a better over-all performance. Finally, the models were combined using linear interpolation. The combined language model provided 37.8% relative improvement in terms of perplexity on the SEAME development set and a relative improvement of 32.7% on the evaluation set compared to the traditional n-gram LM.