{'fraction_non_alphanumeric': 0.03289217594394706, 'fraction_numerical': 0.01725703905540418, 'mean_word_length': 4.850094876660341, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Monolingual Data Selection Analysis for English-Mandarin Hybrid Code-switching Speech Recognition

In this paper, we conduct data selection analysis in building an English-Mandarin code-switching (CS) speech recognition (CSSR) system, which is aimed for a real CSSR contest in China. The overall training sets have three subsets, i.e., a code-switching data set, an English (LibriSpeech) and a Mandarin data set respectively. The code-switching data are Mandarin dominated. First of all, it is found using the overall data yields worse results, and hence data selection study is necessary. Then to exploit monolingual data, we find data matching is crucial. Mandarin data is closely matched with the Mandarin part in the code-switching data, while English data is not. However, Mandarin data only helps on those utterances that are significantly Mandarin-dominated. Besides, there is a balance point, over which more monolingual data will divert the CSSR system, degrading results. Finally, we analyze the effectiveness of combining monolingual data to train a CSSR system with the HMM-DNN hybrid framework. The CSSR system can perform within-utterance code-switch recognition, but it still has a margin with the one trained on code-switching data.


Introduction
Recently, code-switching speech recognition (CSSR) has drawn increased attention in automatic speech recognition (ASR) community [1,2,3,4,5]. Here, code-switching means a linguistic phenomenon that one speaks different languages within an utterance or between consecutive utterances. Intuitively, we can build a CSSR system simply by means of merging two languages. However, such a CSSR system can rarely produce optimal recognition results. This is because there is no within-an-utterance code-switching samples, and the resulting ASR system frequently fails to recognize those code-switching utterances [5,6].
In reality, we usually have very limited code-switching training data, but more monolingual data [5]. As a result, we treat CSSR as a low-resource ASR problem [7,8,9]. The question is how to fully exploit those "unlimited" monolingual data to boost the CCSR performance.
In this paper, we report our efforts in terms of data selection 1 [10,11,12] for an English-Mandarin code-switching speech recognition contest sponsored by DataTang in China 2 . It drew over 70+ participants from world wide. The organizer released two training data sets, namely, 200 hours of English-Mandarin code-switching data, and 500 hours of Mandarin data of which there are about 15 hours of similar codeswitching data. Additionally, 960 hours of LibriSpeech English data [13,14] is allowed to use. There are three tracks, and one of the tracks is for the hybrid DNN-HMM based ASR system competition. Except for the above-mentioned 3 training data sets, a trigram language models are given as well, and the only flexibility is the participants are allowed to use their own lexicon to build their ASR system. In this work, all the ASR results are based on the DNN-HMM system [15].
The motivation of this work lies in the observation that we end up with worse results when using overall three data sets. This motivates us to answer how to exploit the two monolingual data sets reasonably. To begin with, we first add Mandarin data incrementally, and we found consistent performance improvement on the Mandarin part recognition, with insignificant performance drop on the English part. Nevertheless, we found that there is a limit point, where more data degrade results over the point. We do the same experiments for LibriSpeech English data, however we observe consistent performance drop. We conjecture this is due to data mismatch problem. To look further into the problem, we categorize the test utterances into different subsets according to how many English words each utterance contains, then analyze the performance change on those subsets respectively.
Moreover, we are interested in a condition where no codeswitching data is available at all. As a result, one has to utilize monolingual data to train a code-switching system. Particularly, we are interested in how much effectiveness can be achieved by merging pure monolingual data, compared with that we have code-switching data. We remove all the code-switching data from the 500 Mandarin data set, and incrementally merge the LibriSpeech English data into it to train a CSSR system. To our surprise, using more English data is consistently beneficial to both language performance improvement. We notice that this is one of the significant differences of the DNN-HMM ASR system versus End-to-End ASR [16,17] one. By merging monolingual data, it is observed end-to-end CSSR system fails to recognize those utterances containing code-switching words [5,6].
The paper is organized as follows. Section 2 describes the overall data for the experiments that are followed. Section 3 reports the experimental setup. Section 4 describes the development of our baseline system. Section 5 analyzes the effectiveness of using more Mandarin monolingual data, as well as more English data. Section 6 reports the effectiveness of using monolingual data to train CSSR system. Finally we concludes in Section 7. arXiv:2006.07094v1 [eess.AS] 12 Jun 2020

Data description
The entire data sets that are released by the organizer are reported in Table 1. Each participant has 3 training data sets and 3 test data sets, two dev sets, and one evaluation set that is released after the contest.
The code-switching data are Mandarin-dominated. Figure 1 reveals the English word distribution in T1, and E1 to E3 four code-switching data sets in Table 1. As is shown in Figure 1, majority of utterances contain only single English words. Besides, English word distributions are very similar between the training data T1 and the dev set E1, while the cases of E2 and E3 are more alike.
eval 20 Figure 1: English word distribution of the code-switching data sets, including T1, E1 to E3 in Table 1.

Experimental setup
All experiments are conducted with Kaldi 3 . The acoustic models are trained with the Lattice-free Maximum Mutual Information (LF-MMI) criterion [18] over the Factorized Timedelay Neural Network (TDNNf) [15]. The acoustic feature is the concatenation of 40-dimensional MFCC features and 100dimensional i-vectors [19,20]. The TDNNf is made up of 15 layers, and each layer is decomposed as 1536x256, 256x1536, where 256 is the dimension of the bottleneck layer. Besides, the activation function is Rectified Linear Unit (ReLU) [21]. To train TDNNf, data augmentation is employed [22]. The vocabulary is 200k, of which Mandarin and English words are 121k and 79k respectively. We use language dependent phone set, and there are 210 Mandarin initials and finals [2], as well as 42 English phones. As mentioned, the language models are released by the organizer, and it is a trigram language model that includes 813k Mandarin words and 116k English words. 3 https://kaldi-asr.org/

Baseline
We report our efforts on building the baseline system. Since the data and language models are fixed, our efforts are mainly focused on choosing an appropriate lexicon. Initially, we use our in-house lexicon. As there is mismatch between the language models and the lexicon in Chinese word segmentation, we decompose all out-of-vocabulary Chinese words into characters, yielding our initial lexicon, denoted as L0 here. After submitting our evaluation results, we realized there are many entries with incorrect pronunciations for the Chinese part in L0. We conducted manual checking and updated L0 to L1. Simultaneously, we also attempt to use a grapheme lexicon for English words, and this is inspired from [23]. Table 2 reports our baseline results in terms of Meta-data Error Rate (MER) using only CS200 code-switching training data. By MER, we mean a token error rate that is a mixture of Chinese character error rate and English word error rate. From Table 2, The lexicon after manual checking gets the best results. Therefore, we use L1 for the remaining experiments in this paper. Table 3 reports experimental results using various data combination recipe. 7.07 6.28 7.08 +CS15 6.93 6.14 6.87 +Man500 6.87 5.91 6.63 +Libri 7.58 6.67 7.50 Table 3 suggests employing more Mandarin data works, yielding significant MER reduction, however, employing more Libri English data degrades results. These suggest there are data selection issues concerned, and it is worthwhile to see what details look like.

Mandarin data
To begin with, we fix the CS200 code-switching data, incrementally increasing monolingual Mandarin data from the Man500. We are interested to see how the recognition results will be changed for each individual language, as well as for the two languages combined. Figure 2 plots the various change of the recognition results versus incremental increase of Mandarin data usage. We notice that we select the data as follows. We first ensure each selection covers the overall speakers, and then determine how many utterances are to be selected, and finalize by random selection. From Figure 2, what we observe can be briefly summarized in the following aspects. First, employing more Mandarin data generally produces improved results on the Mandarin part as shown in Figure 2(b). Secondly, more Mandarin data does not significantly affects English recognition results, as indicated in Figure 2(c). Thirdly, more Mandarin data over a certain point can hurt the overall performance, as can be seen in Figure 2(d). The best performance is achieved when about 400 hours of Mandarin data is employed on the eval test set.
First of all, we attribute the improvement to the close similarity of the Man500 and the CS200 code-switching data in terms of Mandarin part. Actually, they are all read speech released by the organizer. After that, we think there is a balance point, and more monolingual Mandarin data can divert the models in terms of code-switching capability. However, the abovementioned guesses need further support from more details. Combined what are shown in Figures 2 and 3, the recognition performance gains are mainly from the utterances with single English word. From this perspective, data matching guess are supported here. However, it does limited or even negative help on the performance improvement for those utterances containing more English words. We achieved overall performance improvement as shown in Figure 2, only because the single English utterances are dominated, as shown in Figure 1.

English data
In this section, we are interested to see how the English-Mandarin code-switching ASR system is affected by using more English monolingual data. Here, the English data is Librispeech, as shown in Table 1. It is worth a mention that we do this over the ASR system that is trained with CS200 plus Man500, instead of using the data by directly combining CS200 and Librispeech. Figure 4 plots the recognition performance change versus incrementally introducing Librispeech data. Figure 4 clearly reveals recognition performance degradation as more English data are merged, on either Mandarin or English part of the codeswitching utterances. Intuitively, we attribute this to the data mismatching problem. That is, Librispeech is mismatched with the English that are spoken by Chinese Mainland speakers, and such a mismatch can be enlarged for the case when those single English words are mixed with Chinese words. To verify our guess, we also need further details. Figure 5 shows the performance change details on 4 groups of utterances as shown in Figure 1. From what Figure 5 shows, we cannot simply attribute performance degradation only to data mismatching reason. With Mandarin dominated utterances as seen in Figure 5(a)-(b), introducing English data does not help, and more English data degrades results. However, for utterance less dominated with Mandarin words as shown in Figure 5(c)-(d), where there are more than 3 English words in the utterances, using more English data does not necessarily lead to performance drop. This suggests data mismatch is one reason, code-switching pattern is another season. Different codeswitching patterns are affected by monolingual data usage differently. However, we really need English data from Mainland Chinese to confirm a solid conclusion.

Pure monolingual data
Code-switching ASR system is much more flexible compared with monolingual ASR one. However, real code-switching data is a low-resource, and hard to access. One natural consideration is to build a code-switching ASR system by merging related monolingual data. This has been extensively studied un- der End-to-end (E2E) ASR framework[]. Unfortunately, stateof-the-art E2E ASR system almost completely fails to recognize those utterances mixed with word from different language (within-utterance code-switching) [], when it is learned with monolingual data. This is because the E2E system has never been learned with utterances containing code-switching word sequence, and so does the decoder fail to predict such a sequence. Here, we examine if such a disastrous case can happen under the HMM-DNN ASR framework. Specifically, we do as follows. We first make a subset of Man500 set, by removing all code-switching utterances from it. We fix the Mandarin subset and keep incrementally merging the English data from Librispeech. Figure 6 draws recognition results with a codeswitching ASR system trained with monolingual data. From Figure 6, we see consistent performance improvement by merging the two monolingual data sets. By looking into Figure 6 ical in code-switching scenario. Improving one language can benefit another language's recognition performance. However, there is a balance point, and it should be related to specific codeswitching pattern as shown in Section 5. By looking into Figure 6(c), we see sharp WER drops for the English part. However, the mismatching guess in Section 5.2 is observable here. The WER drop "stops" over 50%, which is much higher than what is shown in Figure 2(c), though the overall WER decrease has not been fully saturated. Finally in Figure 6(d), we compare the performance of the two systems on eval set, one trained with monolingual data (though English part is not matched), and another trained with CS200 code-switching data. The gap is still remarkable, and this should be further studied under a fair comparison scenario.

Conclusion
In this work, we conducted a thorough study on the codeswitching system performance with diversified data selections. First we analyzed the code-switching pattern, it is Mandarindominated code-switching data sets, both for training and testing. We also found the monolingual data helps, but data matching is crucial. Besides, there is a balance point for how much monolingual is employed, which is dependent on a specific code-switching environment. Finally, we performed an analysis on a code-switching system performance assuming no real code-switching data available, under the HMM-DNN modeling framework. We found the HMM-DNN ASR system still well performs for within-utterance code-switching recognition. Our observation is completely different from what have been observed under the end-to-end ASR framework.

Acknowledgements
The computational work for this paper is partially performed on the resources of the National Supercomputing Centre (NSCC), Singapore (https://www.nscc.sg).