{'fraction_non_alphanumeric': 0.03825650674192537, 'fraction_numerical': 0.01191596111633741, 'mean_word_length': 4.857055402509948, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
SCALA: SUPERVISED CONTRASTIVE LEARNING FOR END-TO-END AUTOMATIC SPEECH RECOGNITION

End-to-end Automatic Speech Recognition (ASR) models are usually trained to reduce the losses of the whole token sequences, while neglecting explicit phonemic-granularity supervision. This could lead to recognition errors due to similar-phoneme confusion or phoneme reduction. To alleviate this problem, this paper proposes a novel framework of Supervised Contrastive Learning (SCaLa) to enhance phonemic information learning for end-to-end ASR systems. Specifically, we introduce the self-supervised Masked Contrastive Predictive Coding (MCPC) into the fully-supervised setting. To supervise phoneme learning explicitly, SCaLa first masks the variable-length encoder features corresponding to phonemes given phoneme forced-alignment extracted from a pre-trained acoustic model, and then predicts the masked phonemes via contrastive learning. The phoneme forced-alignment can mitigate the noise of positive-negative pairs in self-supervised MCPC. Experimental results conducted on reading and spontaneous speech datasets show that the proposed approach achieves 2.84% and 1.38% Character Error Rate (CER) reductions compared to the baseline, respectively.


INTRODUCTION
End-to-end Automatic Speech Recognition (ASR) systems have been significantly improved in recent years [1,2]. They are trained to map acoustic feature sequences to token sequences directly. The training normally aims to minimize the losses of the whole character or word sequences, while neglecting explicit phoneme level supervision. The models are powerful enough to learn latent representation partially corresponding to phonemes from each frame [3]. However, these models still suffer from various phonemic issues like similar-phoneme confusion [4] as well as constant or vowel reduction [5]. Inspired by recent development of contrastive learning in speech representation [6][7][8][9], we propose a novel framework named Supervised Contrastive Learning (SCaLa) for end-to-end ASR systems. Contrastive learning with guidance of phoneme labels is conducted to enhance ASR models to learn phoneme information.
Masked Contrastive Predictive Coding (MCPC) [10] is a representative method of contrastive learning used in self-supervised ASR systems. It masks a consecutive segment of the encoder features with a fixed/random length, and then selects anchor/positive and negative samples from masked and unmasked indices, respectively. The model is then trained to discriminate the anchor/positive features from a set of negative features via a contrastive task. Self-supervised MCPC has been proved to be able to learn the latent representation of speech units in the pre-training stage [10]. It achieves a remarkable performance via fine-tuning on small amounts of labeled data. Extending self-supervised MCPC to supervised ASR systems may improve the recognition performance. UniSpeech [11] is a first try to combine self-supervised MCPC and supervised ASR systems. However, it follows the pre-training and fine-tuning paradigm, and MCPC is still employed to learn speech representation without using labelling information. Applying MCPC to unlabeled data is challenging in both mask length selection and mitigation of the effect from noisy negative samples. Since phonemes usually have various lengths in speech, masking with a fixed/random length may not help the model learn phonemic information effectively [5,12]. In contrastive learning, the indices of negative features for contrasting are randomly selected [10]. Hence, there might be noisy negative pairs. For example, the anchor/positive-negative pairs might be from the same phoneme, or both of them may be silence or background noise, etc. As referred in [13], noisy negative samples will compromise the effectiveness of the feature representation.
Unlike previous studies [10,11], SCaLa applies MCPC in a fully-supervised manner. The above two challenges are addressed by using the phoneme label information. First, the mask length is customized for each particular phoneme duration, i.e., the masked unit can be a phoneme. This will help improve the prediction accuracy of reduced phonemes in speech [5]. Specifically, we perform forced-alignment between the speech and the labeled transcription to obtain the phoneme duration and labels. Then the encoder features are masked in phoneme level to enhance the model to learn phonemic latent representations explicitly. Second, the noisy anchor/positive-negative pairs are eliminated by selecting the negative features based on the phoneme forced-alignment labels. Although the alignments might not be perfect, one can still largely reduce noisy negative pairs, e.g., same-phoneme pairs or silence pairs. Hence the ASR model can learn better latent representation by phoneme discrimination.
To train the ASR model, SCaLa includes two sub-tasks: an ASR sub-task is to directly generate character or word sequences from acoustic features, and a contrastive sub-task is to predict masked phonemes for improving phoneme discrimination via contrastive learning. We combine the two sub-tasks to help the model learn the representations of character or word sequences and phoneme level information at the same time, to improve the performance on speech recognition tasks.
The main contributions are: 1) To the best of our knowledge, this is the first work extending the self-supervised MCPC approach to the fully-supervised ASR systems; 2) We propose a framework named SCaLa for end-to-end ASR to enhance phoneme-level information learning; 3) We show the effectiveness of our method with discussion on both reading and spontaneous speech data.

RELATED WORK
Contrastive learning based on Predictive coding (PC) has been widely used in self-supervised ASR training. Contrastive PC (CPC) [14] and its variants [15] predict future speech segments based on the past ones for pre-training unidirectional networks. The A CTC-based ASR sub-task is combined with a contrastive sub-task that leverages a forced-alignment model to perform phoneme masking and contrastive learning. The backbone CTC-base ASR network is composed of a successive stack of Convolutional (Conv) layers, Self-Attention Blocks (SABs), and Fully Connected (FC) layers. The contrastive sub-task selects masked items from context features as anchors, takes items in target features with same indices as anchors as positive samples, and selects legal items with unmasked indices from target features as negative samples.
CPC-based method was adopted for ASR tasks in Wav2vec [16] and Vq-wav2vec to learn discrete representation of speech units [17]. Masked PC (MPC) was proposed in [18], which can improve the performance of Transformer based ASR systems by predicting masked encoder features. In Wav2vec2.0 [10], the authors proposed MCPC, which involved contrastive learning compared to MPC. Significant performance improvements are achieved on downstream ASR tasks. However, as mentioned in [11], the self-supervised paradigm of Wav2vec2.0 needs to be carefully designed, and the representation is difficult to interpret. To help the model learn a more meaningful speech representation with MCPC, UniSpeech combined labeled and unlabeled speech for pre-training in a multitask manner [11]. However, MCPC was still performed without using labels in [11].
Research on contrastive learning for supervised ASR training is relatively new. The motivation of [9] is very similar to our work. Nevertheless, these two works are quite different. First, [9] adopted the SimCLR [19] method in the computer vision domain, while SCaLa utilizes the self-supervised MCPC for ASR. Also, new speech is generated using various data augmentation methods first in [9]. These new speech and the original speech are fed into an ASR framework to output representation corresponding to letterlevel tokens. The representations of the same letter are taken as the positive pairs for contrasting. There is a potential problem that the output of the ASR framework may represent a wrong letter. In this case, consequently, the contrastive pairs would be wrong, and such pairs would mislead the model training by itself. Differently, SCaLa uses the forced-alignment results to build contrastive pairs, which ensures the contrastive pairs are independent of the model training. Moreover, we generate the contrastive pairs by masking encoder features at the phonemic level.
Masking phonemes in the data augmentation domain has been introduced into supervised ASR training. The authors of [5] masked off certain phoneme features during model training to simulate the phoneme reduction phenomenon in Uyghur speech. With this method, the ASR model performed better on spontaneous speech. SCaLa employs the same masking strategy, and adds a new con-trastive learning loss. The loss is proposed to improve the robustness of representation to phoneme prediction and discrimination.

Model architecture
The model architecture of the proposed SCaLa is shown in Fig. 1, which consists of an ASR sub-task and a contrastive sub-task. The representative Connectionist Temporal Classification (CTC) framework used in ASR training [10,11,16] is adopted as our backbone model. Our experiments use the typical model network, which is composed of a successive stack of Convolutional (Conv) layers, Self-Attention Blocks (SABs), and Fully Connected (FC) layers [20]. Given a sequence of ds-dimensional acoustic spectrograms x ∈ R ds×T with length T , the ASR model tries to predict the labeled character sequence y ∈ L N with length N , where L is the size of the finite label character. The output of the last Conv layer is denoted as encoder features z ∈ R d f ×S , where d f is the latent feature dimension, and S is the sequence length. Note that S is less than T after subsampled by the Conv layers. The phoneme-level forced-alignment results are denoted by a ∈ R S [21]. To make the expression concise, we assume that each item in a is a phoneme label of items in the encoder features z instead of the input speech. The items in z are masked with a certain probability during training. The masked features are fed into SABs that yields context features c ∈ R d f ×S . Regarding the contrastive sub-task, a linear layer is adopted as done in [22] to obtain the contrastive target features q ∈ R d f ×S . Then we select masked items from context features as anchors [10], take items in target features with same indices as anchors as positive samples, and select legal items with unmasked indices from target features as negative samples. For the contrastive sub-task, the contrastive loss is designed to guide the model to decrease the similarities between the anchor-negative pairs, and to increase the similarities between the anchor-positive pairs. (a) Phoneme-level forced-alignment of a Mandarin speech (b) Phoneme masks on encoder features, and feature samples from context features and target features for contrasting with mitigating noisy negative pairs. If the first "q0" in context features is masked, the second "q0" of target features should not be selected as the negative features for contrasting.

Fig. 2.
Examples of phoneme-level forced-alignment, masking and feature sampling with mitigating noisy negative pairs. Note that masking is for selecting contrastive features and is actually performed on encoded features only; the masks on the context features here are merely for showing the features being selected.

Loss functions
To train the ASR model, we combine the ASR sub-task and the contrastive sub-task to help the model learn the representation of character sequences and phoneme-level information at the same time, and to improve the performance on speech recognition tasks. The total loss of SCaLa is a combination of a CTC loss based on phoneme masking LCTC and a contrastive loss LC, as follows, where hyperparameter λ is used to balance these terms.

CTC loss based on phoneme masking
Phoneme lengths are used for masking to help the model learn more phonemic information [5]. Specifically, a HMM-DNN acoustic model is trained offline using Kaldi [23] to get the phoneme-level label of each frame. An example is shown in Fig. 2. During the training, we randomly sample start indices of encoder features with a certain probability pe (We set pe = 7.5% same as [10]). A total of P phonemes adjacent to the start indices are integrally masked by leveraging the forced-alignment results. The choice of P is experimentally studied in Sec. 4.2 (see Fig. 3). For a data-label pair {x, y} and the forced-alignment result a, the CTC loss based on phoneme masking is obtained as where a valid CTC path π is a variant of the transcription y that allows occurrences of blank tokens and repetitions. The set φ(x, y) includes all valid CTC paths [24].

Contrastive loss
Supervised contrastive learning aims to improve the robustness of feature representation by discriminating an anchor/positive phoneme from a set of negative phonemes. In particular, to reduce noisy negative pairs, features having the same phoneme label with the masked phoneme are removed from the negative phoneme sets. As shown in Fig. 2, the second "q0" of target features will not be selected as negative phonemes to be contrasted with the first "q0" of context features. Then the contrastive loss is defined as where M is the set of all masked indices of encoder features, and |M | is the number of masked indices; cm and qm are the m th vectors in context features c and target features q, respectively; sim(α, β) = α T β/( α β ) is the cosine similarity; τ is a temperature SCaLa. The index set Nm consists of the masked index m and a negative index set K, which are uniformly sampled from all indices except those having the same alignment label with the masked phoneme am, i.e. a k = am, ∀k ∈ K. We set τ = 0.1 and |K| = 100 in our experiments same as [10]. Two datasets with different speaking styles are used in our experiments: 1) reading speech data: the open-source Aishell-1 which contains 170 hours of Mandarin speech with 16kHz sampling rate [25]; and 2) spontaneous speech data: a JD in-house Mandarin conversational telephony dataset (JD-tel) which contains 1500 hours of speech with 8kHz sampling rate. For Aishell-1, the original traintest splits are used. For the other one, 10% of the speech samples are randomly selected for testing. The 80-dimensional Mel-spectrograms are used as the input to the network. The frame size and step size are 20ms and 10ms, respectively. There are 3 Conv layers, 10 SABs, and 2 FCs in our model as shown in Fig. 1. For more model details refer to [20]. A 4-gram language model trained with the KenLM toolkit [26] is used for inference.
Same with [27], the alternate minimization training method is employed. The training losses LCTC and LC in Eq. (1) are used (a) CERs (%) on the reading speech data Aishell-1 (b) CERs (%) on the spontaneous speech data JD-tel Fig. 3. Ablation study and parameter searching. CERs for SCaLa compared with: 1) SCaLa without supervision in the contrastive sub-task (SCaLa-SC), and 2) SCaLa without the contrastive sub-task (SCaLa-C) for a variety of mask settings on reading and spontaneous speech data. The performance of the traditional CTC method (Baseline) is included for reference. alternately with ratio 1:1 to update the model parameters. Our model is trained with a mini-batch of 128 on 4 Nvidia V100 GPUs. The Learning Rate (LR) for LCTC is 2.5×10 −5 , and for LC is 5×10 −4 . The LR of LCTC is kept constant, while the LR of LC is decayed to 5 × 10 −5 at the end of training.

Experimental Results
As shown in Table 1, we compare SCaLa with state-of-the-art ASR systems including hybrid [28], end-to-end [5,29], and selfsupervised learning [10,30]. Numerically, SCaLa outperforms the traditional CTC models [20] with 2.84% and 1.38% CER reductions on reading and spontaneous speech data, respectively. The detailed analysis of our SCaLa is presented in Sec. 4.3. Experimental results also show that SCaLa significantly outperforms hybrid chain models [28], end-to-end CTC-Conformer systems [29], self-supervised systems learning [10,30], and methods with phoneme masking [5].
Parameter searching and ablation study are conducted. We compare two masking methods: masking F consecutive frames and masking P consecutive phonemes where F ∈ {1, 4, 7, 10} and P ∈ {1, 2, 3}. Different contrastive strategies including SCaLa, SCaLa without supervision in the contrastive sub-task (SCaLa-SC), and SCaLa without the contrastive sub-task (SCaLa-C) are also compared. The results are shown in Fig. 3. The baseline method in Fig. 3 is the traditional CTC method. One can see that both masking with phoneme information and supervised contrastive learning can improve system performance significantly. SCaLa achieved the best performance when P = 2.

Analysis of SCaLa
To further evaluate the effectiveness of SCaLa, we analyze the proposed method from the following three perspectives: Robustness to phonemic issues. The phonemic issues, such as similar-phoneme confusion and phoneme-reduction usually introduce substitution and deletion errors. The detailed CER, including substitution (SUB), deletion (DEL) and insert (INS) error rates, of SCaLa and the baseline CTC method are shown in Table 2. The results show that SCaLa achieves significant reductions on substitution and deletion errors. We also observe that the performance improvement of SCaLa on reading speech data is more than that on spontaneous speech data. This might be that recognition tasks on spontaneous speech are more challenging, and the given forced-alignment results might not be accurate enough. Nevertheless, our method can still improve the performance of speech recognition. Table 2. Detailed CER (%), including substitution (SUB), deletion (DEL) and insert (INS) error rates, for SCaLa and the baseline CTC method on reading (Aishell-1) and spontaneous (JD-tel) speech data. Noisy negative reduction. The forced-alignment results show that the noisy negative rates of self-supervised MCPC are 10.21% and 14.60% in the two data sets, respectively. The number is nonnegligible [13]. As shown in Fig. 3, SCaLa improves system performance compared to SCaLa-SC by reducing noisy negative features.
Regularization effect. Fig. 4 shows the CTC losses of SCaLa, SCaLa-SC, SCaLa-C and the baseline CTC method on the validation data of Aishell-1. SCaLa obtains the lowest and smoothest loss curve compared with the other three methods, which shows the regularization effect to the training.

CONCLUSION
In this paper, a novel framework named SCaLa has been proposed for ASR training. It extends the self-supervised MCPC approach to the fully-supervised setting. The labels are effectively leveraged to enhance ASR models to learn phoneme information. SCaLa significantly improved the performance on both reading and spontaneous Mandarin speech data compared to the baseline methods. In the future work, more other masking strategies will be investigated, and the performance on other languages will be evaluated.