{'fraction_non_alphanumeric': 0.03362505757715339, 'fraction_numerical': 0.009097190234914785, 'mean_word_length': 5.06036287508723, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
SpeechBlender: Speech Augmentation Framework for Mispronunciation Data Generation

One of the biggest challenges in designing mispronunciation detection models is the unavailability of labeled L2 speech data. To overcome such data scarcity, we introduce SpeechBlender -- a fine-grained data augmentation pipeline for generating mispronunciation errors. The SpeechBlender utilizes varieties of masks to target different regions of a phonetic unit, and use the mixing factors to linearly interpolate raw speech signals while generating erroneous pronunciation instances. The masks facilitate smooth blending of the signals, thus generating more effective samples than the `Cut/Paste' method. We show the effectiveness of our augmentation technique in a phoneme-level pronunciation quality assessment task, leveraging only a good pronunciation dataset. With SpeechBlender augmentation, we observed a 3% and 2% increase in Pearson correlation coefficient (PCC) compared to no-augmentation and goodness of pronunciation augmentation scenarios respectively for Speechocean762 testset. Moreover, a 2% rise in PCC is observed when comparing our single-task phoneme-level mispronunciation detection model with a multi-task learning model using multiple-granularity information.


INTRODUCTION
With the advent of globalization, more people are now inclined towards multilingualism. Such a shift in interest promotes the demand of the computer-aided pronunciation training (CAPT) system to language learning application [1,2]. The CAPT system potentially exploits the advancement of speech technology along with curriculum management and learner assessment among others. Mispronunciation detection (MD) and diagnosis system -a crucial component of the CAPT -detects the mispronunciation in a L2 learner's speech and highlights the error to give effective feedback.
Over years, researchers have exploited many methods to assess pronunciation quality. A majority of these methods use pre-trained automatic speech recognition (ASR) systems to either (i) pinpoint the mismatch between the aligned ASR output sequence with the reference; or (ii) utilize the log-posterior probability from the ASR to compute variants of goodness of pronunciation goodness of pronunciation (GOP) scores [3,4,5] to asses pronunciation. One successful line of research treated the mispronunciation detection as a regression or a classification task [6]. Using deep learning, the network is either trained in an end-to-end manner [7,8,9,10] or built upon the computed GOP features from the pre-trained ASR [11,12,13,14] to detect errors. These approaches are heavily dependent on the availability of a large amount of manually annotated datasets for the target demographic L2 speakers. However, such annotated datasets are quite scarce and often contain uneven positive class labels. 1 Data augmentation techniques are recently proved to be quite effective to overcome the data scarcity for MD. Techniques involved (i) modifying the canonical text with mismatched character pair keeping its original word-level speech intact [9]; (ii) using Text-To-Speech to synthesize 'incorrect stress' samples along with modified lexical stress [15]; (iii) using a mixup technique, in feature space -based on phone-level GOP pool to design word level training data [16]; and (iv) using error distance of the clustered self supervised learning model embedding to replace the phoneme sound with another close sound unit [17,18].
The aforementioned techniques simulate mispronunciation errors by finding a target phoneme unit, in a different -text, feature or audio space, and use it to replace the current unit. However, the L2 learners' pronunciation is highly affected by their native phoneme set [19] and 'Cut/Paste' approach reuses the available information without generating any new feature variation in phoneme level. Thus making such mixup -Cut/Paste augmentation methods less efficient.
In this study, we introduce SpeechBlender -a simple finegrained data augmentation pipeline for generating mispronunciation errors in low-resource settings. The SpeechBlender framework linearly interpolate raw input signals utilizing different masks and mixing factors, allowing to modify different regions of a target phoneme. Moreover, the proposed framework generates new training samples using only good pronunciation units and thus capable of generating MD training data from any native/non-native speech datasets.
We show the effectiveness of the proposed augmentation framework over text-and GOP-based augmentation techniques using a significantly faster non-autoregressive LSTM [20] model for phoneme-level MD task. We compare the performance of different masks -such as Smooth Overlay, Smooth Concatenation, CutMix, and Smooth Gaussian Overlay -in the SpeechBlender and show its efficacy over the well-known Cut/Paste technique. We present our results using a well-establish publicly available MD dataset, Speechocean762 [21]. While the proposed technique is studied for a specific task (MD) in this paper, the SpeechBlender can be broadly applicable as an augmentation technique for most sequence labeling such as accented ASR or classification task with audio/speech as input. Our contributions are: (a) Introduce the SpeechBlender, a fine-grained data augmentation pipeline capable of smoothly modifying regions of phonetic unit to generate erroneous pronunciation; (b) Show the effectiveness of the proposed method over textbased and GOP-based augmentation; (c) Analyse the performance of different masks in the Speech-Blender and compare it with cut/paste signal approach; (d) We make the SpeechBlender augmentation module publicly available. 2 Fig. 1. Overview of the SpeechBlender, a fine-grained data augmentation pipeline, for zero-shot mispronunciation class generation. UC : input speech utterance for candidate phoneme; UD: selected speech utterance for donor phoneme; xC (T ), xD(T ) candidate and donor phoneme frames of length T and L respectively;x is the resultant augmented phoneme. Figure 1 shows an overview of the SpeechBlender -a fine-grained data augmentation pipeline, for generating mispronunciation classes in a low-resource settings. 3 From the canonical phoneme sequence of an input utterance UC , we select a few good phonetic units as augmentation candidates. For each candidate unit, we pick a donor phoneme using a close phoneme-pair dictionary and choose another utterance, UD as a source signal of the donor phoneme. We obtain the alignments of the utterances UC and UD using a pre-trained ASR, and get the corresponding phoneme time boundaries. Using the start and end boundary of the candidate, xC , and donor, xD, phoneme, we select the speech frames and apply augmentation with the SpeechBlender. We then replace the candidate phoneme xC with the augmented phoneme framesx in the candidate utterance UC .

Close Phoneme Pair Dictionary
The 'close phoneme pair dictionary' is designed to randomly select a donor phoneme from a list of confused phonemes associated with the candidate phoneme. The dictionary is constructed using both the Speechocean762 [21] and L2-ARCTIC [19] training set. The entries are selected by analysing the confused phoneme pairs of speaker's pronunciation using ASR errors like substitution along with human annotation labels. Some examples of close phoneme pairs are presented in Table 1. Table 1. Examples of entries in Close Phoneme Pair Dictionary, with their % of confusion. 2 The code will be shared upon paper acceptance. 3 Presence of a few or zero erroneous pronunciation examples in the data.

Pre-trained ASR and Alignment Module
We align the acoustic frames with the corresponding reference phoneme to obtain the start and end timestamps. For the force alignment, we employ the Hidden Markov Model-Time Delay Neural Network (HMM-TDNN) 4 as the acoustic module. It is trained on Librispeech ASR task with Kaldi [22].
Algorithm 1: SpeechBlender function Input: Input phoneme frame sequences xC and xD where where (T , L) are total frames in (xC , xD) respectively. yC represent the original output score assigned to the good phoneme. Output: Augmented frame output sequencesx and the corresponding labelŷ of the output. Function Blender(xC , xD, yC ): Mask template id ρ ←− get property(mid, len(xC ), len(xD)) Mask property W, Λ ←− generate mask(ρ) W : list of regional mask width, Λ: list of regional mixing factor values.
Given the input frames of good-phoneme candidate xC and the donor-phoneme xD, of length T and L respectively, our approach outputs the augmented phonemex along with its corresponding output labelŷ. We first normalize the energy of xD to match xC . Subsequently, we apply the SpeechBlender function, shown in Algorithm 1, with a selected mask templates.
Mask Templates: We first randomly select a mask template id, mid ∈ [1 − 4]. We then generate the property ρ of the mask based on the mid, length of the phonemes T and L. The property ρ ∈ {µ, R, W, Λ} is parameterized by the features of the mask including: . The width of each region is determined by the minimum frame length of inputs when both the input pair is considered with λ > 0. A λ = 0 for a signal represent a mixing weight of zero, meaning the full signal is blocked out. Details of different mask templates along with its property, ρ, are presented in Figure 2.
Blending the Speech signals: The Given property ρ of the mask, we generated the mask with its W list and its corresponding Λ list containing region-based λ values. Then for each regions in the mask, we multiply the xC and xD with λ and 1 − λ respectively (shown in Algorithm 1) to create the regional augmented data θx(.). We create the new augmented phoneme,x by concatenating all the θx regional data. The labelŷ ∈ {0, 1} of the augmented phonemex is generated by averaging the regional label (θŷ = λ · y). The y is the value assigned to a good pronunciation of a phoneme. 5 The augmented outputŷ = 0 represent the mispronounced or missing phoneme; whereasŷ = 1 represent the accented or aberrant phoneme unit. In our experiment, the regional augmented output is assigned θŷ = 0 if the mixing factor λ < 0.25, otherwise it is assigned θŷ = 1.

Baseline Augmentation Methods
We compare the performance of the SpeechBlender with text-level augmentation and GOP feature-level augmentation.
Text-based Augmentation For text-based phoneme data augmentation, we increase the number of error phoneme by altering the phoneme embedding from the canonical phoneme sequence. The candidate phoneme was replaced with a selected donor phoneme, using the same 'close phoneme pair dictionary', when creating the canonical phoneme embedding space. An output labelŷ = 0 is assigned when the candidate phoneme is swapped with a distant donor phoneme, for e.g., SH ⇔ V ; whereasŷ = 1 is assigned for a close phoneme exchange such as SH ⇔ S.

GOP-based Augmentation
For GOP-based data augmentation, we first create a bag of GOP feature embedding for all good pronounced phonemes and grouped them by phoneme pairs using the same 'close phoneme pair dictionary'. We then replaced the candidate phoneme, in the utterance with a donor phoneme GOP embedding randomly selected from the bag. We follow the same scoring scheme as text-augmentation forŷ labels.

EXPERIMENTS
In this Section, we describe the dataset and the system architectures used to show the efficacy of the proposed SpeechBlender augmentation. 5 These values are determined based on the original data annotation scheme. For this study, y = 2.

Speech Corpus
For the study, we opt for the Speechocean762 [21], a free opensource dataset designed to assess the pronunciation quality of L2 learners. The dataset consists of 5000 English utterances from 250 non-native speakers with rich annotation labels. Along with different utterances and the word level quality aspects, the dataset include accuracy scores at phoneme-level that are divided based on three class labels, as shown in Table 2. Each phoneme is annotated by five experts and the final phoneme accuracy score is calculated as their average.

Mispronunciation Detection Model
An overview of the mispronunciation detection model architecture is presented in Figure 3. Given the input audio and the canonical transcription, we first extract the frame-level phonetic features using pre-trained ASR, described in Section 2.2. Using the acoustic model, we calculated the 84-dimensional GOP features for 42 pure phonemes. Each phone GOP representation contain two different features: (i) log phone posterior (LPP) [6] of a phone p of length T frames with per frame observation ot see Equation 1; and (ii) the log posterior ratio (LPR) [6] of phone pj with respect to phone pi, as given in Equation 2 LP P (p) = 1 T log p(p|ot) LP R(pj|pi) = log p(pj|ot) − log p(pi|ot) These 84-dimensional features are then passed through a feedforward layer to project the GOP embeddings in 24 dimensional feature space. Simultaneously, we inject the textual-content information using the one-hot vector representation of the canonical phoneme transcription. These vector representations are then projected to 24-dimensional phoneme embedding and are then added to the GOP representations. Consequently, we pass the final representation to the encoder network. As our encoder, we opt for a long short-term memory (LSTM) 6 network due to the model's strength in capturing preceding contextual information, and its reduced computational cost without losing much on the model accuracy. For further comparison, we implement the architecture with similar configuration as [11] with 3 stacked LSTM layers and an output embedding dimension of 24. We then pass the embedding per phoneme to the output layer for the regression task. Model Parameters: We train the mispronunciation detection model using an Adam optimizer for 20 epochs with a starting learning rate of 1e − 2 and a batch size of 32. The learning rate is then cut in to its half in every 3 epochs, after 10 initial epochs. As for the loss function, we use the mean squared error (MSE). We report the results based on the last epoch.

Evaluation Measures
We measure the proposed model performance using standard statistical metric: MSE and Pearson correlation coefficient (PCC). We base our discussion using PCC as it is more representative of the performance in such an imbalanced class scenario.

RESULTS AND DISCUSSION
In this Section, we report the results in Table 3 with different augmentation settings and compare it with no-augmentation and multitask learning setup proposed in [11]. We validate our proposed augmentation pipeline using the Speechocean762 test data. Furthermore, we show the strength of each mask proposed with the Speech-Blender and compare it with the Cut/Paste method in Table 4.
Comparison with Baselines: The reported PCC results in Table  3 indicates the importance of augmentation in general. Our results indicate both the GOP-augmentation and SpeechBlender augmentation outperforms the model with only Speechocean762 trainset by 1% and 3% respectively. However, the same is not seen for textaugmentation baseline. The GOP-augmentation, a feature space replacing technique, performs comparable with the multi-task learning setup with a PCC of 0.59. Our proposed augmentation technique out performs both the GOP-augmentation and the multi-task setup by 2%. using only phoneme-level information for training. 6 Additional models such as SVR [21], MLP are tested with Speech-Blender data augmentation. As the results follow the same trend , for brevity we are reporting the results using LSTM only.
Effectiveness of the Masks: We assess the effectiveness of each standalone mask incorporated in the SpeechBlender for creating different mispronunciation instances. From our results, reported in Table 4, we observe that the smooth blending masks (Smooth Overlay and Smooth-Gaussian Overlay) create more practical variation in mispronunciation and accented phoneme than their counter-part. When compared directly with the 'Cut/Paste' augmentation mask, our bending mask also shows its efficacy.

CONCLUSION
In this paper, we introduce a fine-grained phoneme-level augmentation technique, SpeechBlender, and show its efficacy in generating new error classes for a phoneme-level pronunciation quality detection task. The SpeechBlender linearly interpolate raw input signals using different blending masks like 'SmoothOverlay' and mixing factors λ. Varieties of masks included in the proposed method allows smooth modification in different regions of the phoneme. With SpeechBlender, we observe improved performance compared to (i) no-augmentation, (ii) multi-task learning scenario, (iii) textaugmentation, and (iv) GOP-augmentation setting. We noticed the 'Cut/Paste' method be it in text-, feature-or raw-audio domain, is not as efficient as the proposed interpolation technique. Moreover, our results suggest that with blending the signals, we can generate new positive error classes instead of reusing samples from the data itself in zero-shot setting. The different choices of masks allow variations in transition and style of errors. In future, we will further explore the capability of SpeechBlender for generating accented/erroneous phoneme candidate for other speech tasks like ASR, while exploring multiple granularities.