{'fraction_non_alphanumeric': 0.03137679199350825, 'fraction_numerical': 0.013674751299852734, 'mean_word_length': 4.98560892246807, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Semi-supervised acoustic modelling for five-lingual code-switched ASR using automatically-segmented soap opera speech

This paper considers the impact of automatic segmentation on the fully-automatic, semi-supervised training of automatic speech recognition (ASR) systems for five-lingual code-switched (CS) speech. Four automatic segmentation techniques were evaluated in terms of the recognition performance of an ASR system trained on the resulting segments in a semi-supervised manner. The system's output was compared with the recognition rates achieved by a semi-supervised system trained on manually assigned segments. Three of the automatic techniques use a newly proposed convolutional neural network (CNN) model for framewise classification, and include a novel form of HMM smoothing of the CNN outputs. Automatic segmentation was applied in combination with automatic speaker diarization. The best-performing segmentation technique was also tested without speaker diarization. An evaluation based on 248 unsegmented soap opera episodes indicated that voice activity detection (VAD) based on a CNN followed by Gaussian mixture modelhidden Markov model smoothing (CNN-GMM-HMM) yields the best ASR performance. The semi-supervised system trained with the resulting segments achieved an overall WER improvement of 1.1% absolute over the system trained with manually created segments. Furthermore, we found that system performance improved even further when the automatic segmentation was used in conjunction with speaker diarization.


Introduction
Code-switching is the alternation between two or more languages by a single speaker during discourse, and is a common phenomenon in multilingual societies. In South Africa, for example, 11 official and geographically colocated languages are in use, including English which serves as the lingua franca. Here, speakers frequently codeswitch between English, a highly-resourced language, and their Bantu mother tongue, which is in comparison highly under-resourced. The automatic recognition of code-switched speech has become a topic of growing research interest, as reflected by the increasing number of language pairs that have recently been studied. While English-Mandarin has received extensive attention (Li and Fung, 2013;Zeng et al., 2018;Vu et al., 2012;Taneja et al., 2019), other language pairs such as Frisian-Dutch (Yılmaz et al., 2016;Yılmaz et al., 2018), Hindi-English (Pandey et al., 2018;Emond et al., 2018;Ganji et al., 2019), English-Malay (Ahmed and Tan, 2012), Japanese-English (Nakayama et al., 2018) and French-Arabic (Amazouz et al., 2017) have also attracted interest. We have introduced the first South African corpus of multilingual code-switched soap opera speech in (van der Westhuizen and Niesler, 2018). For code-switched speech, the development of robust acoustic and language models that are able to extend across language switches is a challenging task. When one or more of the languages are under-resourced, as it is in our case, data sparsity limits modelling capacity and this challenge is amplified. Acoustic data that includes code-switching is extremely hard to find, because it usually does not occur in formal conversation, such as broadcast news, and also because it requires skilled multilingual language practitioners for its annotation. The result is that manually-prepared datasets including code-switched speech in Africa are des-tined to remain rare and small.
In previous work, we have demonstrated that multilingual training using in-domain soap opera code-switched speech and poorly matched monolingual South African speech improves the performance of both bilingual and five-lingual automatic speech recognition (ASR) systems when the additional training data is from a closely-related language (Biswas et al., 2018a;Biswas et al., 2018b). Specifically, isiZulu, isiXhosa, Sesotho and Setswana belong to the same Bantu language family and were found to complement each other when combined into a multilingual training set for acoustic modelling. Hence, increasing the amount of in-domain code-switched speech data is a reliable way to achieve more robust ASR. However, the development of such in-domain data is a time-consuming and costly endeavour as it requires highly skilled human annotators and transcribers.
To address this lack of annotated data, automatically transcribed training material has been shown to be useful in under-resourced scenarios using semi-supervised training (Thomas et al., 2013;Yılmaz et al., 2018;Guo et al., 2018). This strategy was successfully implemented on South African code-switched speech to obtain bilingual and five-lingual ASR systems using 11.5 hours of manually segmented but untranscribed soap opera speech (Biswas et al., 2019). Recently a study has analyzed the performance of batch-wise semi-supervised training on South African code-switched ASR (Biswas et al., 2020). However, manual segmentation of the raw soap opera audio by skilled annotators was still required to identify the speech that is useful for ASR. Therefore, this approach is not fully automatic which remains an impediment in resource-scare settings.
In this study, we apply four automated approaches to the segmentation of soap opera speech and investigate the effect on ASR performance. A conventional energy-based voiced activity detector (VAD) (Povey et al., 2011), as well as CNN-HHM and CNN-GMM-HMM systems that we have developed are used to distinguish between speech, music and noise. In addition, an X-vector DNN embedding system is used for speaker diarization to obtain speaker specific metadata for three of the segmentation approaches (Snyder et al., 2018). For the experiments, 248 complete soap opera episodes, each approximately 22 minutes in length, were used. It is important to note that we also have the manual segmentation (approximately 24 hours of speech) of these 248 episodes and can therefore perform a comparative evaluation with the automated approaches. Semi-supervised systems trained using the manually-segmented speech were used as baselines and compared with systems trained on speech identified by the automatic approaches. Pseudo-labels or transcriptions of automatically segmented speech were generated using our best baseline systems trained on 21 hours manually transcribed speech and 11 hours of manually segmented but automatically transcribed speech. Given the multilingual nature of the data, the transcription systems must not only provide the orthography, but also the language(s) present at each location in each segment. To achieve this, each segment was presented to four individual code-switching systems as well as to a fivelingual system.

Data
For experimentation, we use a corpus of multilingual, code-switched speech compiled from South African soap opera episodes. This corpus contains both manually and automatically-annotated speech divided into four language pairs: English-isiZulu (EZ), English-isiXhosa (EX), English-Setswana (ET), and English-Sesotho (ES). Of the Bantu languages, isiZulu and isiXhosa belong to the Nguni language family while Setswana and Sesotho are Sotho-Tswana languages. The corpus contains 8 275, 11 352, 6 169, 1 902 and 2 792 unique English, isiZulu, isiXhosa, Setswana and Sesotho words, respectively. IsiZulu and isiXhosa have relatively large vocabularies due their agglutinative nature and conjunctive writing system. Although Setswana and Sesotho are also agglutinative, they use disjunctive writing systems which result in smaller vocabularies than isiZulu and isiXhosa. The speech in the soap opera episodes is also typically fast and often expresses emotion. These aspects of the data in combination with the high prevalence of codeswitching makes it a challenging corpus for conducting ASR experiments.

Manually segmented and transcribed data (ManT)
Our first code-switching ASR systems were developed and evaluated on 14.3 hours of speech divided into four language-balanced sets, as described in (van der Westhuizen and Niesler, 2018). In addition to the languagebalanced sets, approximately another nine hours of manually transcribed speech was available. This additional data is dominated by English and was initially excluded from our training set to avoid bias. However, pilot experiments indicated that, counter to expectations, its inclusion enhanced recognition performance in all languages. The additional data was therefore merged with the balanced sets for the experiments described here. Of this, 21.1 hours is used as a training set, 48 minutes as a development set, and 1.3 hours as a test set. The composition of the unbalanced training set is shown in Table 1.

Manually segmented automatically transcribed data: Non-expert segmentation (AutoT NonE )
A subsequent phase of corpus development, currently still underway, has produced manual segmentations for a further 248 soap opera episodes. These 248 episodes amount to 89 hours of audio data before segmentation, and 23 hours of speech data (AutoT NonE ) after segmentation. The segmentation was not performed by language experts and is therefore expected to be less accurate than that of the AutoT Exp data. Furthermore South African languages other than the five present in the transcribed data are known to occur in this batch, but to a limited extent. This set of 248 episodes was used in the automatic segmentation experiments described in the next section because the manually assigned segment labels were available as a reference in the form of AutoT NonE .

Automatic Segmentation
A number of automatic segmentation techniques were considered as alternatives to the labour-intensive process of manually segmenting the soap operas. Different voice activity detection (VAD) approaches were combined with the X-vector DNN embedding-based speaker diarization system introduced in (Snyder et al., 2018) to obtain speaker labels. In subsequent ASR experiments, the best performing VAD technique was also evaluated without speaker diarization.

VAD 1 : Energy-based
In our first experiment, the X-vector diarization recipe provided in the Kaldi toolkit was applied using an Xvector DNN model pre-trained on wide-band VoxCeleb data (Povey et al., 2011;Nagrani et al., 2017;Chung et al., 2018). This system uses 24-dimensional filterbank features based on 25ms frames. Speech frames are identified using a simple energy threshold and are subsequently passed to the pre-trained DNN which extracts the X-vectors. Finally, probabilistic linear discriminant analysis (PLDA) is applied to the X-vectors, and agglomerative hierarchical clustering is used to assign speaker labels. A difficulty observed when using this approach was that, while a simple energy VAD works reasonably well under low noise conditions where most frames are speech, it performs poorly when confronted with our soap opera data in which extensive non-speech segments containing music and other sounds are common. Post-diarization listening tests revealed that many non-speech segments were still present in the data classified as speech. Adjustment of the VAD threshold to more aggressively remove non-speech segments resulted in the loss of many speech segments.

VAD 2 : CNN-HMM
At the time of writing, the X-vector based system achieved state-of-the-art performance in diarization tasks. However, the energy based VAD it uses limits performance. For this reason, efforts to improve automatic segmentation focused  on developing improved VAD. Recently, CNNs have been successfully applied to the task of VAD (Thomas et al., 2014), and both large and small architectures have been found to perform well (Sehgal and Kehtarnavaz, 2018;Hershey et al., 2017). In our resource-constrained setting, computational efficiency is important since VAD will most likely occur on a mobile device.
We introduce a small CNN architecture (≈120 000 parameters) implemented in Python, using Tensorflow (v2.0.0) and , to create a fast, lightweight VAD system whose architecture is shown in Table 3. This system computes 32-dimensional log-mel filterbank energies using a frame length of 10ms and then stacks these over 320ms to form 32x32 spectrogram features as input to the CNN. The CNN was trained on the balanced subset (≈ 53 hours) of Audio Set  to classify frames as containing "speech" and/or "non-speech". While our CNN on its own performs well at the VAD task, it fails to capture temporal patterns in the data and was observed to often mislabel single frames within extended sections of speech or non-speech. In an initial attempt to address this, we introduce a HMM for smoothing. The AVA-Speech dataset (Chaudhuri et al., 2018) was used to train our HMMs, and for testing of the final VAD. The full dataset contains ≈ 46 hours of densely labeled, multilingual movie data, with the following class labels: "NoSpeech","CleanSpeech", "Speech+Music" and "Speech+Noise". AVA-Speech (train), a randomly selected ≈ 23 hour subset of the dataset, was used to train the HMM. Table 4 provides a description of the dataset used for training and testing of the automatic segmentation systems. For training the "CleanSpeech", "Speech+Music" and "Speech+Noise" classes were treated as a single "speech" class. A two state HMM was defined, with states representing ground truth "speech" and "no-speech" labels respectively. The HMM observations are the binary output of the "speech" neuron in the CNN, which indicates "speech" or "no-speech". Note, the "no-speech" label differs slightly from the "non-speech" label, since the "speech" and "nonspeech" sounds can co-occur, whereas "no-speech" implies "speech" does not occur in the signal. Transition and emission probabilities were trained in a supervised manner, by passing AVA-Speech (train) though the CNN, then using the labels predicted by the CNN and corresponding ground truth labels as observations and hidden state sequences respectively. Viterbi decoding was then used to find the most likely underlying label sequence,  given CNN predicted labels. Finally the VAD segments are used as input to the X-vector diarization system.

VAD 3 : CNN-GMM-HMM
While the CNN-HMM approach yields a large improvement over the energy-based VAD, it may be possible to improve it further by making use of the CNN soft label outputs, rather than the hard labels obtained by taking the argmax of the CNN outputs. In this case the HMM observation sequence is chosen to consist of the output probabilities computed by the CNN speech neuron, rather than the binary labels. Where the observations were previously modelled as repeated Bernoulli trials, they are now continuous and can therefore be modelled by a more complex distribution function. A 3-mixture GMM for each of the two HMM states was found to be an effective choice. Fewer mixtures led to deteriorated performance, while more mixtures did not result in further improvement. As before, the GMM-HMM is trained on AVA-Speech (train) in a supervised manner and the resulting segments used as input for the X-vector diarization system.

VAD 4 : VAD 3 without speaker diarization
While the X-vector diarization system is useful for obtaining speaker labels for each segment, it is computationally expensive and represents only a pre-processing step for downstream ASR. To determine its importance, our final experiment used the segments produced by our best performing VAD system directly, without diarization. Hence each segment was treated as being from a different speaker.

Automatic Transcription
Recent studies demonstrated that semi-supervised training can improve the performance of Frisian-Dutch codeswitched ASR (Yılmaz et al., 2018) as well as South African code-switched ASR (Biswas et al., 2019). The approach taken in this study is illustrated in Figure 1. The figure shows the two phases of semi-supervised training for the parallel bilingual as well as five-lingual configurations: automatic transcription followed by bilingual semisupervised acoustic model retraining. The five-lingual system was not retrained with the automatically transcribed data for this set of experiments as our primary motive was to study the effect of automatic segmented speech on bilingual semi-supervised ASR.

Parallel bilingual transcription
This system (System A) consists of four subsystems, each corresponding to a language pair for which code-switching occurs. Acoustic models were trained on the manually segmented and transcribed soap opera data (ManT, described in Section 2.1.) pooled with the manually segmented but automatically transcribed speech (AutoT Exp , introduced in Section 2.2.). Because the languages spoken in the untranscribed data were unknown, each segment was decoded in parallel by each of the bilingual decoders. The output with the highest confidence score provided both the transcription and a language pair label for each segment.

Five-lingual transcription
Some of our previous experiments indicated that the automatic transcriptions generated by the five-lingual baseline model enhanced the performance of the bilingual semisupervised systems (Biswas et al., 2019). Five-lingual transcriptions were therefore also included in this study. The five-lingual system (System H) is based on a single acoustic model trained on all five languages. It was trained on the same data as the bilingual systems, except for the fact that the AutoT Exp data was transcribed using a five-lingual baseline model. Since the five-lingual system is not restricted to bilingual output, its output allows Bantu-to-Bantu language switching. Examples of such switches were indeed observed in the transcriptions. Moreover, the automatically generated transcriptions sometimes contained more than two languages. Although the use of more than two languages within a single segment is not common, we have observed such cases during the compilation of the manually transcribed dataset. For our fast, continuous speech, the automatically generated segments have been observed to produce longer segments than manual segmentation of the data. This increases the likelihood of multiple language switches within the segment. Unfortunately, since the automatic segments are generated from untranscribed data, the degree to which multiple languages occur within a single automatic segment is difficult to quantify.

Acoustic modelling
All acoustic models were trained using the Kaldi ASR toolkit (Povey et al., 2011) and the data described in Section 2.. The models were trained on a multilingual dataset that included all the data in Table 1. In addition, three-fold data augmentation (Ko et al., 2015) was applied prior to feature extraction. The feature set included standard 40dimensional MFCCs (no derivatives), 3-dimensional pitch and 100 dimensional i-vectors. The models were trained with lattice free MMI (Povey et al., 2016) using the standard Kaldi CNN-TDNN-F  Librispeech recipe (6 CNN layers and 10 timedelay layers followed by a rank reduction layer) and the default hyperparameters. All acoustic models consist of a single shared softmax layer for all languages, as in general there is more than one target language in a segment. No phone merging was performed between languages and the acoustic models were all language dependent. For the bilingual experiments, the multilingual acoustic models were adapted to each of the four target language pairs.

Language modelling
The EZ, EX, ES, ET vocabularies contained 11 292, 8 805, 4 233, 4 957 word types respectively and were closed with respect to the training, development and test sets. The vocabularies were closed since the small datasets and the agglutinative character of the Bantu languages would otherwise lead to very high out-of-vocabulary rates. The SRILM toolkit (Stolcke, 2002) was used to train and evaluate all language models (LMs). Transcriptions of the balanced subset of the ManT dataset as well as monolingual English and Bantu out-of-domain text were used to develop trigram language models. Four bilingual and one five-lingual trigram language model were used for the transcription systems as well as for semisupervised training (Yılmaz et al., 2018;Biswas et al., 2019). Table 5 summarises the development and test set perplexities for the bilingual LMs. Details on the monolingual and code-switch perplexities are only provided for the test set (columns 3 to 6 in Table 5). The test set perplexities of the five-lingual LM are 1007.1, 1881.8, 345.3, and 277.5 for EZ, EX, ES and ET respectively. Further details regarding the five-lingual perplexities can be found in (Biswas et al., 2019). Much more monolingual English text was available for language model development than text in the Bantu languages (471M vs 8M words). Therefore, the monolingual perplexity (MPP) is much higher for the Bantu languages than for English for each language pair. Code-switch perplexities (CPP) for language switches indicate the uncertainty of the first word following a language switch. EB corresponds to switches from English to a Bantu language and BE indicates a switch in the other direction. Table 5 shows that the CPP for switching from English to isiZulu and isiXhosa is much higher than switching from these languages to English. This can be ascribed to the much larger isiZulu and isiXhosa vocabularies, which are, in turn, due to the high degree of agglutination and the use of conjunctive orthography in these languages. The CPP for switching from English to Sesotho and Setswana is found to be lower than switching from those languages to English. We believe that this difference is due to the much larger English training set. The CPP values are even higher for the five-lingual language model. This is because the five-lingual trigrams allow language switches not permitted by the bilingual models.

Semi-supervised Training
For semi-supervised ASR, lattice-based supervision was combined with the lattice-free MMI objective function (Manohar et al., 2018;Carmantini et al., 2019). Conventionally, semi-supervised training only considers the best path while lattice-based supervision uses the entire decoding lattice. Hence, the latter approach allows the model to learn from alternative hypotheses when the best path is not accurate. Table 6 gives an overview of the bilingual ASR systems that were trained using the manually segmented data (System B) as well as five different versions of the automatically segmented data (Systems C-G & I). In addition to manuallytranscribed speech, ManT, the AutoT Exp data was also included in all the training sets. Also defined in Table 6 are systems A and H, which are the bilingual and five-lingual baseline systems respectively, trained only on the ManT and AutoT Exp data. These baseline systems were used to obtain automatic transcriptions, AutoT A and AutoT H , for each version of the additional data shown in Table 6. These automatic transcriptions were subsequently used to train new acoustic models. VAD 2Sub was included to enable a fair comparison between automatic and manual segmentation. This is a 21-hour, randomly selected subset of the VAD 2 data which is comparable in size to the manually-segmented dataset (AutoT NonE ).

Results & Discussion
The next three subsections concern results of the systems described in Sections 3., 4. and 5. Finally, ASR results are presented for specific languages, as well as at codeswitching points.

Automatic segmentation
AVA-Speech (test), which is the ≈ 23 hour subset of AVA-Speech not used for HMM training, was used as a test set to evaluate VAD performance. This dataset provides similar conditions to our target domain of soap opera data, as well as dense voice activity labels. Furthermore, it is accompanied by baseline results for the WebRTC project VAD (We-bRTC.org, 2011) as well as two CNN-based systems based on the architecture proposed in (Hershey et al., 2017). The smaller of these two CNN-based systems, tiny320, is similar in size to our CNN, also containing three convolutional layers, while the other, resnet960, is based on the much larger ResNet-50 architecture (He et al., 2016). Frame-based true positive rates (TPR) for a fixed false positive rate (FPR), scored over 10ms frames are shown for all VAD systems in Table 7. To allow comparison, all VAD systems were tuned to achieve a FPR of 0.315, as described in (Chaudhuri et al., 2018). TPR is reported for each individual speech condition (clean speech, speech with noise and speech with music) as well as for all conditions combined.
As expected, VAD 1 performs poorly. However, it is interesting to note that it is the only system that performs better for the "Noise" and "Music" conditions than for the "Clean" condition. This is because noisy signals tend to have more energy than their clean counterparts, making noisy signals more likely to exceed an energy threshold. A large performance improvement is seen for VAD 2 which uses the CNN-HMM. In particular, this system already outperforms tiny320. A smaller performance increase is re-  ported for VAD 3 which uses the CNN-GMM-HMM. However, this increase brings its performance to a level comparable to the much larger resnet960 system. In the case of "Music", VAD 3 outperforms resnet960, whilst for "All" the TPR of VAD 3 is within 1% absolute. In terms of computational complexity, the energy VAD is about 30 times faster than the CNN based VADs. However, the speaker diarization system is two orders of magnitude slower than the slowest VAD, making the compute times of VAD 1 , VAD 2 and VAD 3 are roughly equivalent. VAD 4 , which removes the speaker diarization, is much faster.

Automatic transcription
The automatic transcription outputs of the bilingual (System A) and five-lingual (System H) baseline systems are summarised in Table 8. The first five rows of the table correspond to segments that were classified as monolingual while the last row shows the number of segments that contain code-switching. The values in this row reveal a high number of code-switched segments in the additional data.
In terms of the number of segments per category, the output of the automatic segmentation systems agree with the manual segmentation process. The only exception is the number of English segments identified by the five-lingual system, which is higher than for the other systems. We believe that this is because the five-lingual language model was trained on more in-domain English text (Biswas et al., 2019).
The table also shows that including speaker diarization in the segmentation process produces smaller chunks of words than using only the VAD. Due to the varying duration of each set, comparisons are difficult to make. For this reason, the set VAD 2Sub is included, which is of a similar duration to AutoT NonE , allowing comparison between the nonexpert manual segmentation and automatic segmentation. It  can be seen that for the same duration of data, the automatic segmentation produces fewer segments.

Automatic speech recognition
The performance of the ASR systems introduced in Table 6 were measured in terms of the word error rate (WER) achieved after semi-supervised training. Results for the different training configurations are reported in Table 9. The values in the table indicate that including the additional 23 hours of non-expert manually segmented data (AutoT NonE ) in the training set (System B) yields absolute improvements of 1.5% and 1.4% over the baseline (System A) for the development and test sets respectively. The results for System C show that using 83 hours of automatically-segmented speech results in an absolute improvement of 1.7% and 1.6% for the development and test sets relative to System A. Although System C's performance is on par with that of System B, its training set was much larger which means that the computational cost of developing the system is also much higher. According to Table 6, VAD 2 reduced the additional training data from 83.6 to 47 hours. Furthermore, including this reduced additional data in System D's training set resulted in lower WERs for both the development and test sets, when compared with Systems A, B and C. The semi-supervised system trained on the 21-hour subset of VAD 2 (System E) achieved results that are comparable to those of System B. The two additional dataset seem to have had almost the same impact on the accuracy of the resulting acoustic models. This result seems to indicate that, in terms of ASR performance, manually and automaticallyproduced segmentations are equally well suited for system development. However, it should be kept in mind that the segment labels used by System B were not assigned by experts. System F, trained on the segments generated by VAD 3 , yielded better performance than system D, despite the fact that System D's training set contained 10 more hours of data. The improvement in WER was found to be statistically significant at the 95% confidence level using bootstrap interval estimation (Bisani and Ney, 2004).
The results for System G show that automatic segments that do not take speaker identity into account (VAD 4 ) do not achieve the accuracy levels as those that do (System F). Therefore, the inclusion of speaker diarization does tend to improve ASR performance. The performance of System H (five-lingual baseline system) is included in Table 9 but should not be directly com- pared with the bilingual systems because the recognition task is inherently more complex. However, as has been observed before (Biswas et al., 2019), the bilingual System I, trained on automatic transcriptions generated by System H, shows the best overall performance of all the evaluated systems. The improvement on the test set over its closest competitor (System F) is 0.5% absolute and this was found to be statistically significant above the 90% confidence level using bootstrap interval estimation. This improvement may be due to the ability of the five-lingual system to transcribe more than two languages, as well as Bantu-to-Bantu switches. The untranscribed soap opera speech is known to contain at least some segments that do not conform to the four considered bilingual language groupings. The degree to which such language switches do occur is unfortunately difficult to quantify without manual transcriptions. However, since the key difference between the bilingual and five-lingual systems is the ability to handle a greater variety of language switches, we speculate that this is a likely cause for the superior performance of System I. The improvement in WER achieved by the semi-supervised ASR systems incorporating different versions of the additional data is summarised in Figure 2. The figure confirms that the largest gain in recognition accuracy was achieved by System I. It also affirms the observation that an equal amount of manually and automatically segmented   data yields an equal improvement in recognition accuracy in a semi-supervised set-up.

Language specific WER analysis
For code-switched ASR, the performance of the recogniser at code-switch points is of particular interest. Language specific WERs and code-switched bigram correct (Bi CS ) values for the different semi-supervised systems are presented in Table 10. Code-switch bigram correct is defined as the percentage of words correctly recognised immediately after code-switch points. All values are percentages. The table reveals that both the English and Bantu WERs for all the semi-supervised systems are substantially lower than the corresponding values for the baseline system. The accuracy at the code-switch points is also substantially higher for the semi-supervised systems. Hence, adding the additional training data enhances system performance at codeswitch points. Moreover, there are no substantial differences between the gains achieved by adding the manually (System B) or automatically (Systems F, I) segmented data.

Conclusions
In this study, we have evaluated the impact of using automatically-segmented instead of manually-segmented speech data for semi-supervised training of a code-switched automatic speech recognition system. Four different automatic segmentation approaches were evaluated, based respectively on simple energy thresholding with diarization, a CNN classification with two variants of HMM smoothing and diarization, and CNN classification with GMM-HMM smoothing and no diarization. It was found that applying our new CNN-GMM-HMM based VAD followed by Xvector speaker diarization resulted in the best ASR performance. The results also showed that the performance of systems that used automatically and manually-segmented data were comparable. We conclude that automaticsegmentation in combination with semi-supervised training is a viable approach to enhancing the recognition accuracy of a challenging five-language code-switched speech recognition task. This is a very positive outcome, since the difficulty in providing a manual segmentation of new broadcast material has remained an impediment to the development of speech technology in severely under resourced settings such as the one we describe. Future work will focus on improving the VAD and speaker diarization techniques as well as incorporating language identification into the automatic segmentation process.