{'fraction_non_alphanumeric': 0.038564850345356866, 'fraction_numerical': 0.0173317984139166, 'mean_word_length': 4.641053391053391, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Internal Language Model Estimation based Language Model Fusion for Cross-Domain Code-Switching Speech Recognition

Internal Language Model Estimation (ILME) based language model (LM) fusion has been shown significantly improved recognition results over conventional shallow fusion in both intra-domain and cross-domain speech recognition tasks. In this paper, we attempt to apply our ILME method to cross-domain code-switching speech recognition (CSSR) work. Specifically, our curiosity comes from several aspects. First, we are curious about how effective the ILME-based LM fusion is for both intra-domain and cross-domain CSSR tasks. We verify this with or without merging two code-switching domains. More importantly, we train an end-to-end (E2E) speech recognition model by means of merging two monolingual data sets and observe the efficacy of the proposed ILME-based LM fusion for CSSR. Experimental results on SEAME that is from Southeast Asian and another Chinese Mainland CS data set demonstrate the effectiveness of the proposed ILME-based LM fusion method.


Introduction
Code-switching (CS) refers to a discourse containing more than one language in terms of written or spoken form. On utterance level, one can further classify CS into two categories. One is there is different language alternation within an utterance, normally called as intra-sentential code-switching, here intra-CS for brevity. Another is language alternation occurs across neighboring utterances, here called as inter-CS. CS ASR [1][2][3][4][5][6][7][8][9] is more challenging when it is compared with monolingualbased speech recognition due to data sparsity issue. This is particularly true for the intra-CS case. In such a case, the code-switching is not only culture-dependent it is also personal peculiarity-dependent. As a result, how code-switching occurs is full of uncertainty, and such an uncertainty results in data sparsity, and hence poor recognition results.
To achieve an improved code-switching speech recognition performance, the first strategy one can easily think of is to employ data augmentation approaches [10][11][12]. For instance, one can employ off-the-shelf text-to-speech system to generate more audio-text paired CS data to train ASR models [11].
However, the difficulty lies in a text-to-speech system capable of synthesizing spontaneous code-switching data itself doesn't easily come by. Likewise, another easier avenue is to generate more CS text to boost language modeling performance. [13] employs Machine Translation (MT) and word/phrase substitution methods to generate more CS text. The difficulty is how to get more paired text data to yield a sensible MT model. Recently, [14] proposed to use CycleGan [15] to produce CS text data without paired text data to train the sequence-to-sequence model. Similarly aiming to alleviate CS data sparsity issue, [16] translates CS recognition results to monolingual text to score.
Except for explicit data augmentation method addressing data scarcity issue, one can aim to approach the problem from model optimization perspective. [17] proposed to employ multilingual data to train CS E2E ASR model by penalizing the divergence of ASR output distributions of different languages, while [12,18] employed the similar idea to train neural network CS language model with monolingual text data. Also through model optimization efforts, [5] proposed to employ LID classification results to fine-tune the output of a CTC-based E2E CS ASR score, yielding improved performance on intra-CS recognition. More recently, [19] proposed an language-dependent self-attention mechanism that is to reduce cross-lingual context confusion and achieved improved results. The underlying motivation of [20] is to fully take advantage of monolingual data and make what is learned from monolingual data be transferable to CS ASR models. Similar idea is also reflected in [21], where two monolingual encoders are employed to pretrain the ASR model, and less CS data is employed to fine-tune the pretrained model, attaining improved CS ASR results.
In this paper, we are interested in cross-domain CS ASR instead. Concretely, we improve cross-domain CS ASR performance by internal language model estimation (ILME) [22][23][24][25][26][27] based language model (LM) fusion [28][29][30][31][32] method. This is inspired from the prior works where ILME method has significantly improved both intra-domain and cross-domain ASR performance in diversified monolingual ASR tasks. As mentioned, cross-domain CS ASR is much more challenging in contrast with monolingual ASR, since CS is not only culture-dependent it is also personal peculiarity-dependent, especially for the intra-CS case. For instance Chinese mainland English-Mandarin CS is remarkably different from Southeast Asian CS in terms of accent, as well as style. Chinese mainland CS is a mandarinpredominant CS with single English content word in utterance in most cases, while in Singapore of Southeast Asia, English and Mandarin word ratio is more diversified in daily conversa-tion. Besides, there are frequent connective words appearing in utterance, such as "Then 我就去canteen 吃饭了(Then I went to canteen for meal)".
Specifically, in this paper we examine the efficacy of the proposed ILME-based LM fusion method [33] for cross-domain CS ASR performance between SEAME [34] and a mainland Chinese-dominated CS data set that is release by DataTang for ASRU 2019 English-Mandarin CS ASR Challenge [3]. Our contributions are mainly reflected in the following aspects: 1. To the best of our knowledge, this is the first pioneering work showing the efficacy of the ILME-based LM fusion for CS ASR, in terms of both intra-domain and crossdomain scenarios under E2E ASR framework. 2. More importantly, we attempt to employ monolingual data to train multilingual ASR for CS recognition, employing ILME-based LM fusion, leading to significant performance improvement compared with conventional shallow fusion. 3. We elaborate how to perform ILM estimation using Transformer decoder.

ILME-based LM fusion
In this paper, we use Conformer for E2E ASR modeling. With the trained conformer, the ASR inference process can be described as follows:Ŷ where X is input acoustic features, and Y is inferred word sequence, while P (Y |X) is a joint posterior probability.
Give external text data that is employed to train a LM, the E2E ASR inference is formulated as: where PLM(Y ) is the joint probability estimated from external LM. What Eq. 2 states is not harmony with Bayes probability theory for conventional hybrid DNN-HMM ASR, where it states:Ŷ To realize LM fusion as in conventional DNN-HMM ASR under E2E ASR framework, we approximate log P (Y |X) in Eq. 2 as log P (Y |X) := log P (X|Y ) + log P (Y ) using Bayes posterior probability. Replacing term log P (X|Y ) in Eq. 3, we have Eq where P (Y ) is estimated with training transcripts, as a result we called as internal LM (ILM), i.e., PILM(Y ). Besides, we normally need to rescale it, and Eq. 4 is turned into: where λILM is ILM scaling factor. Now, we can apply Eq. 5 for E2E ASR LM fusion that is in harmony with Bayes theory for ASR, and the overall focus is on how to estimate the internal LM score log PILM(Y ) with the existing ASR framework.

Transformer-decoder-based ILME
To realize Transformer-decoder-based ILME, we need to re-  visit cross-attention modules in each decoder block as indicated in Figure 1.
For each cross-attention, we have the following operations: where Layernorm and MHCA refers to layer normalization and multihead cross-attention operations respectively. For ILME, we need to make the decoder act as an LM model, and the parameters of such an LM model should be maximally shared with existing decoder. Following what is proposed in [33], we employ two methods to estimate ILM. We name the first method as One-time Context-vector Learning (OTCL). Here, the contextvector refers to x i−1 in Eq. 7. By OTCL-based IMLE, it is estimated as follows: where b is a learnable bias in Eq. 8. For simplicity, it is shared across different decoder blocks and it is learned with overall training transcripts by fixing the other learned parameters in decoder.
The obvious limitation of Eq. 9 is that the bias cannot remember the history context, besides, once it is learned it cannot change. As a result, [33] proposes another IMLE method, denoted as label synchronous context-vector learning (LSCL). Specifically, we introduce a lightweight Feed Forward Network (FFN), realizing: Likewise, the FFN is shared by different decoder blocks.

Data
We select two Mandarin-English code-switching ASR data sets for cross-domain CS ASR performance study. One is SEAME [34], a conversational Mandarin-English corpus from Southeast Asia, i.e., Malaysia and Singapore, another is a Mandarin-English CS data set from Chinese Mainland, released by Datatang for an open Mandarin-English CS ASR challenge in ASR 2019 [3]. For brevity, we name it as ASRU in what follows. Though both data sets are Mandarin-English CS, they are hugely different. They are from different areas that means CS influenced with different cultural background. More importantly, SEAME is conversational speech, while ASRU is reading speech, and hence much simpler. Table 1 reports overall speech data distributions in details. Since we are not only inter- To this end, we make a new "Multi-lingual" data set by merging two monolingual data sets in Table 1.
To be specific, "Man/Eng" in Table 1 refers to average ratios of Mandarin characters to English words. From Table 1, we can see that ASRU CS data is much more predominated with Mandarin. Actually, majority of sentences are dotted with only a single English word. Besides, our SEAME data set definition follows [6], while ASRU also follows official data set definition [3]. Finally, For "Multilingual " corpus, we get majority of English speech data from IMDA corpora [35], while Mandarin data is from extended DataTang release [3].
Language modeling is decisive in this paper. To realize a comprehensive study of how ILME-based LM fusion works under different CS scenarios, we define 6 data sets to build diversified LMs as indicated in Table 2, where "SEAME-CS" and "ASRU-CS" are corresponding training transcripts, while "SEAME-CS-extra" includes more IMDA [35] CS data and "ASRU-CS-extra" includes 500 hours of Mandarin transcripts provided by DataTang as well as 960 hours of transcripts from Librispeech, and finally, the "SEAME-ASRU-*" refers to the text data by corresponding data combinations.

Models
The ASR models are built with Conformer [36] and LMs are built with Transformer using Espnet toolkit [37]. For ASR  [38] with dropout rate of 0.1 to all of our models, and the final models are obtained by averaging the parameters of the 10 models that have the best accuracy on validation set. For inference, the weighting factor of CTC is increased to 0.4, and all results are reported with Mix Error Rate (MER), that is, character for Chinese while word for English. Table 3 presents our perplexity results on overall test sets using all our LMs in Table 2. We observe from Table 3 that crossdomain perplexity is quite high, implying significant difference between two CS data sets. Besides, LM5&LM6 by merging two CS text data leads to remarkable perplexity drop for two CS test sets. It is noteworthy it gets significantly better perplexity when more English and Mandarin data is merged with ASRU transcript data (see LM4 versus LM3).  Test   LM1  52  78  190  256  249  LM2  47  57  119  145  142  LM3  388  28128  31  399  4607  LM4  704  686  38  236  228  LM5  82  108  43  172  168  LM6  49  58  44  105  102   Table 4 reports our overall CS cross-domain ASR results. We have trained 3 ASR models using SEAME, ASRU, as well as SEAME + ASRU speech data respectively. For all IMLEbased LM fusion methods, we have the following configurations to mention. In S5&S6 of the SEAME ASR group, we apply LM2 on SEAME test sets, and apply LM4 to ASRU test sets. The same LM2 and LM4 are also applied for A5&A6 of the ASRU ASR group respectively. For C4&C5 of the SEAME+ASRU ASR group, LM6 are applied for both. For experiments with SEAME ASR, we observe the proposed LSCL-based IMLE fusion method makes consistent better recognition results compared with shallow fusion methods in both intra-domain (for SEAME) and cross-domain (for ASRU) CS ASR tasks. The MER improvement in intra-domain is minor. It can be seen when S6 is compared with S3, where the MER improvement is from 15.7% to 15.4% for DevMan, and 21.9% to 21.7% for DevSge. This is understandable since our external LM text data is rather limited and majority are monolingual data. The MER improvement for the cross-domain is slightly better as is seen by comparing S6 with S4.

Results
For experiments with ASRU ASR, table 4 shows the proposed method also brings performance improvement over the shallow fusion method in cross-domain experiments though very minor. However we haven't observed any improvement over shallow fusion in ASRU intra-domain case.
Finally, Table 4 also reports various recognition results from ASR models trained with SEAME and ASRU combined data. Surprisingly, we get best baseline ASR results on overall test sets of two corpora. More importantly, both IMLE-based fusion methods are effective for MER reduction compared with shallow fusion method.
In Table 4, we have demonstrated the efficacy of the proposed methods in terms of cross-domain CS ASR, and all ASR models are trained with CS data. However, CS data is hard to access in reality. Now we want to examine how an ASR model that is trained with combined monolingual data performs for CS ASR, particularly with the help of ILME-based LM fusion method. Table 5 reports the MER results with the multilingual ASR models using ILME-based LM fusion method. We notice that in all cases the proposed method makes significant MER reduction over the shallow fusion method. This is especially remarkable in the ASRU case, and we get up to 32.06% WERR. 5. Analysis Table 5 shows us that CTC-Attention joint learning based multilingual ASR models can also recognize code-switching speech, which beyond our previous knowledge [7], so we are curious about which part of the ASR model helps to do so. Figure 2 plots the Loss versus epoch for CTC-only criterion on the left and the Attention-only criterion on the right. The "Train-*" and "Valid-*" refer to multilingual and CS speech respectively. We find that for CTC-only system, losses for Train and Valid descend following the same tendency, probably thanks to the non-auto-regressive property of CTC. In contrast, Attentiononly system shows a diverged losses on validation data instead, which suggests attention has difficulty to perform CS ASR if CS utterances has never been seen during training process.

Conclusion
In this paper, we proposed an ILME-based LM fusion for crossdomain CS ASR. We have demonstrated the effectiveness of the proposed method on SEAME and a so-called ASRU Mandarin-English code-switching corpora. We also found the proposed IMLE-based LM fusion method is also very effective on the ASR models that are obtained by combining two data sets, in terms of intra-domain CS ASR. More importantly, we have also attempted to train a multilingual ASR system by merging two monolingual data, to perform CS ASR. With the help of the proposed method, as well as the joint CTC-attention training criterion, the performance improvement is very significant, achieving up to 32.1% mixed error rate reduction compared with conventional shallow fusion method.