{'fraction_non_alphanumeric': 0.04388926401080351, 'fraction_numerical': 0.016413026541318237, 'mean_word_length': 4.948100092678406, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Improving End-to-End Contextual Speech Recognition with Fine-Grained Contextual Knowledge Selection

Nowadays, most methods in end-to-end contextual speech recognition bias the recognition process towards contextual knowledge. Since all-neural contextual biasing methods rely on phrase-level contextual modeling and attention-based relevance modeling, they may encounter confusion between similar context-specific phrases, which hurts predictions at the token level. In this work, we focus on mitigating confusion problems with fine-grained contextual knowledge selection (FineCoS). In FineCoS, we introduce fine-grained knowledge to reduce the uncertainty of token predictions. Specifically, we first apply phrase selection to narrow the range of phrase candidates, and then conduct token attention on the tokens in the selected phrase candidates. Moreover, we re-normalize the attention weights of most relevant phrases in inference to obtain more focused phrase-level contextual representations, and inject position information to better discriminate phrases or tokens. On LibriSpeech and an in-house 160,000-hour dataset, we explore the proposed methods based on a controllable all-neural biasing method, collaborative decoding (ColDec). The proposed methods provide at most 6.1% relative word error rate reduction on LibriSpeech and 16.4% relative character error rate reduction on the in-house dataset over ColDec.


INTRODUCTION
In recent years, many end-to-end (E2E) automatic speech recognition (ASR) approaches, such as connectionist temporal classification (CTC) [1,2], recurrent neural network transducer (RNN-T) [3], attention-based encoder-decoder (AED) [4][5][6][7], have been widely employed in voice assistants, online meetings, etc. However, recognizing context-specific phrases in these scenarios remains to be improved because most contextual contents are rare in training data. For instance, the contextual contents for voice assistants are usually contacts, song playlists, etc., and the contextual contents in meetings are usually the names of those who attend meetings and some technical terms. Injecting contextual knowledge to bias decoding process of these E2E ASR models has become an important research field.
Currently, the most widely known contextual biasing methods for different E2E models are shallow fusion [8][9][10][11][12], attention-based deep context [13][14][15][16] and trie-based deep biasing [17,18]. Among these methods, shallow fusion fuses a finite state transducer (FST) compiled from a list of biasing phrases into the decoding process but does not add any neural networks. In contrast, the all-neural attention-based biasing method [13] and its extensions on other models [19] generally encode phrases with an encoder and integrate the relevant context with an attention module at each time step. Though the all-neural attention-based methods outperform shallow fusion [13], it still suffers from some problems, such as the high correlation between a large number of phrases [13] and the confusion between similar biasing phrases [14][15][16]. In [13], the performance degradation when injecting lots of distractors reveals that the high correlation between hundreds of biasing phrases tends to prevent the model from distinguishing target phrases from other similar ones. Meanwhile, in [14][15][16], the confusion between similar phrases is observed and reduced by injecting phoneme information or training with difficult negative examples. A typical instance is the confusion between "Joan" and "John". The all-neural attention-based methods represent these two names with phrase-level embeddings and thus cannot effectively describe subtle differences at the token level. Likewise, these obstacles exist in the extensions of the attention-based method on other models, such as collaborative decoding (ColDec) [20].
ColDec transfers deep context to the CIF-based ASR [21] in a more controllable way. The CIF module in the CIF-based model non-uniformly compresses acoustic feature sequence along the time axis according to the acoustic boundaries of tokens, and emits tokenlevel acoustic embeddings. These token-level acoustic embeddings provide a bridge to integrate textual context knowledge at the acoustic level. In ColDec, an extra context processing network (CPN) is trained to predict where to output which phrase according to the relevance between token-level acoustic embedding and contextual contents. Unlike deep context, ColDec combines the ASR outputs and the CPN biased outputs with a tunable weight to conduct collaborative decoding, thus leaving room for controlling in practice.
This paper improves the basic ColDec by dealing with the confusion with three techniques: fine-grained contextual knowledge selection (FineCoS), context purification, and position information. For FineCoS, we first restrict the range of phrase candidates according to their relevance to local acoustic embeddings, and then use fine-grained attention to extract token-level contextual representation from all tokens in these phrase candidates. In inference, we purify the phrase-level contextual representation by re-integrating the most relevant phrases to reduce context confusion. Besides, the influence of injecting position information on contextual modeling is also explored at different granularities. In previous works, [15,16] focus on improving phrase encoder with extra phoneme inputs to discriminate similar phrases better, while our work mainly improves attention modules and the decoder side, and meanwhile fully use fine-grained knowledge. Compared with training with difficult negative examples [14], our work focuses more on neural contextual modeling instead of training strategies.

Collaborative Decoding
Collaborative decoding (ColDec) [20] introduces phrase-level contextual modeling and attention-based relevance modeling to contextualize the CIF-based ASR model. In ColDec, besides the ASR model, a context processing network (CPN) is trained to extract target biasing phrases from transcription. For example, given reference "en@@ ni@@ s tri@@ ed to sleep" and a target phrase "en@@ ni@@ s", we generate the training target of CPN "en@@ ni@@ s # # # #" from the reference by keeping the tokens of target phrase and replacing other tokens with "#" (which represents no biasing output). As shown in Fig.1, the CIF-based model consists of an encoder, a CIF module and a decoder. And the CPN comprises an encoder, an attention module and a decoder. This method is named collaborative decoding because the token-level acoustic embedding sequence [c1, ..., ci, ..., cI ] emitted by the CIF module drives the decoding of the ASR model and the CPN simultaneously. Specifically, given raw biasing phrases [z1, ..., zn, ..., zN ], a nobias option z nb (represented as token "<NO-BIAS>") is introduced as an option of not using context, and then a token "<EXT>" is added to the start of each phrase for phrase embedding extraction. An example of processed biasing list Z = [z nb , z1, ..., zn, ..., zN ] is shown at the bottom right corner of Fig.1. At the top right corner of Fig.1, the CPN encoder encodes n-th phrase zn into a fixeddimensional phrase embedding denoted as en,e (e nb,e for no-bias option). With all phrase embeddings [e nb,e , e1,e, ..., en,e, ..., eN,e] as keys/values, the CPN attention consumes an token-level acoustic query ci to generate phrase-level contextual representation mi. Finally, ci and mi are concatenated and subsequently sent to CPN decoder. In inference, driven by the CIF output ci, ASR decoder and CPN decoder conduct decoding with interpolated log probability (logPi + λlogP c i , where λ controls the degree of biasing). Note that for CPN, the biasing phrases of each training batch is randomly sampled from n-grams in references, while the biasing phrases of test sets are usually extracted manually from context.

Fine-grained contextual knowledge selection
Fine-grained Contextual knowledge Selection (FineCoS) introduces token-level contextual knowledge to reduce the uncertainty  of token predictions. At first, we apply elementwise addition on token embedding en,j (where j denotes the index of token and n denotes the index of phrase) emitted by the CPN encoder and its corresponding phrase embedding en,e to generate the final token embeddingēn,j. This operation informs the model of which phrase a token embedding belongs to. Similar to phrase-level attention, a token-level no-bias optionē nb,1 is introduced to represent not using contextual knowledge. After the addition operation, tokenlevel attention, which can be seen as "soft selection", captures the relevance between acoustic embedding ci and final token embeddings [ē nb,1 ,ē1,1, ...,ēn,1,ēn,2, ...,ēN,1,ēN,2, ...], and outputs the token-level contextual representation gi. At last, the concatenation of gi and the output state of the CPN decoder is passed through a projection layer followed by a softmax layer to predict CPN targets. Unfortunately, the biasing phrases usually number in the thousands, which makes token-level attention intractable. Thus, we apply phrase-level hard selection to narrow the range of phrase candidates. Specifically, the top M phrases are selected from Z according to the ranking of CPN attention weight αi,n. After obtaining the top M relevant phrases [z1, ...,zm...,zM ] plus no-bias option z nb , we retain the token embeddings from these M phrases and conduct token-level attention with these retained token embeddings [ē1,1, ...,ēm,1, ...,ēM,1, ...] plus token-level no-bias optionē nb,1 as keys/values. Here, phrase selection with the averaged CPN attention of the whole sequence (αi,n = 1 I I q=1 αq,n) is marked as "global", while phrase selection with the averaged CPN attention of current step and past several steps (αi,n = 1 Q i q=i−Q+1 αq,n) is marked as "local". The process of FineCoS shown in Fig.2 is written as where tilde denotes "selected" and hat denotes "re-normalized".
Context purification is independently proposed in this work, but similar to the weak-attention suppression (WAS) [22] in some aspects. Both context purification and WAS mask some attention weights and re-normalize the rest, but context purification masks weights according to the ranking of weights and only functions in inference.

Position information
Compared with ColDec [20], the influence of position information on contextual modeling is explored at the phrase level and the token level. We inject position information to the inputs of the CPN encoder via position encoding [23]. Intuitively, the position encoding helps the CPN encoder model the differences in the token position distributions of phrases. Moreover, position information makes tokens more distinguishable (less confusing) because it tells the model which part of phrase a token is located in.

Datasets and Metrics
Our experiments are conducted on LibriSpeech [24] and an in-house code-switch ASR dataset. For LibriSpeech, we use 960 hours of labelled audio for training. As the test sets of LibriSpeech lack biasing lists, we construct biasing lists by collecting words that fall outside the 20,000 most common words in training data from references of test set and discarding short words that have less than 5 English letters. Finally, the simulated biasing lists for test-clean and test-other are composed of 1171 and 1129 phrases, respectively. Our in-house dataset consists of ∼160,000 hours of labelled English and Chinese Mandarin audios collected from videos and other common acoustic situations. Its test sets are named test-name and test-term, both of which are collected from real in-house meetings. The details are shown in Table 1. In the default biasing list, the total number of phrases of both biasing lists are 633 and 2415, and the number of distractors in them are 600 and 1775. Rare proportion, which denotes the proportion of rare non-distractor phrases (which appear less than 200 times in the training set) in all non-distractor phrases, is shown in Table 1. Note that all biasing lists are sessionlevel because we assume that audios are collected from meetings, so that all utterances in one test set share the same biasing list. We use word error rate (WER) for LibriSpeech and character error rate (CER) for the in-house dataset to measure ASR, and use precision (P), recall (R) and f1-score (F1) of biasing phrases to measure contextual biasing. More details about datasets are released 1 .

Configurations
The input features are 80-dimension log-Mel filter banks extracted with 25ms window length and 10ms frame-shift via Kaldi [25]. For the in-house dataset, the frequency masking and time masking in SpecAugment [26] are applied, and its ASR output vocabulary comprises 5740 Chinese characters and 4013 English word-pieces generated via BPE [27]. For LibriSpeech, speed perturbation [28] with fixed ± 10% and adaptive SpecAugment [29] are applied, and its ASR output vocabulary comprises 3726 word-pieces. For the CPN, its output vocabulary includes an extra token "#". The structure of the ASR model is almost the same as that in [21]. For LibriSpeech, we use 17 conformer blocks [30] as the ASR encoder, and 2-layer self-attention networks (SANs) as the ASR decoder. The hidden size d model , the projection dimension d f f and the number of attention heads h are 512, 2048 and 8, respectively. For the in-house dataset, we use 15-layer SANs as the ASR encoder, and 2-layer SANs as the ASR decoder (d model = 640, d f f = 2560, h = 8). As for CPN, we use the same structure for both datasets. The CPN comprises an encoder with 4-layer SANs, a decoder with 2layer SANs and an attention module with one SAN. The structure of proposed token attention is the same as that of CPN attention. The training of CPN is similar to that in [20]. With the trained ASR model being frozen, we train the CPN with the same audios. Here, the probability of discarding sampled n-grams in training is 0.3, and the number of contextual batches in gradient accumulation [20] is 8 for LibriSpeech and 2 for the in-house dataset. M in FineCoS is 5, and K in context purification is 2 for LibriSpeech and 10 for inhouse dataset. In inference, we conduct search with beam size 10. More details about configurations of LibriSpeech are also released. We validate our improvements on LibriSpeech with the simulated biasing lists. As shown in Table 2, the CIF-based ASR (S0) achieves 2.12% WER on test-clean and 5.26% on test-other, which is a strong ASR baseline without extra language models. First, the basic ColDec is applied on both test sets. ColDec (S1) brings limited F1 improvement and slight WER degradation. We hypothesize the reason for the poor performance in S1 is that S0 has achieved rather good results on recognizing biasing phrases, and thus the recall of biasing phrases does not overwhelm the errors brought Table 3. Results on test-name and test-term: the enhanced ColDec is compared with the CIF-based ASR model (E0) and the CIF-based ASR model equipped with the basic ColDec (E1). In addition, the relative CER reduction compared with E1 is provided. Our in-house dataset is used to explore our methods on largescale datasets and in real scenarios. As depicted in Table 3, compared with ASR baseline (E0), ColDec (E1) improves the recognition performance on both test sets. Then, we apply context purification and find that context purification (E2) brings both test sets with F1 improvements. Based on E2, E3 injects position information to enhance contextual modeling, and shows F1 and CER improvements. Injecting position information brings 8.3% relative CER reduction on test-name. To validate the effect of FineCoS, we build one branch (E4) from E1 and two branches (E6 and E7) from E3. Both E6 and E7 apply FineCoS, but E6 uses FineCoS with local phrase selection (Q = 5), and E7 uses global phrase selection. Compared with E3, both E6 and E7 improve F1 on test sets, but only E7 gets obvious CER reduction. To further validate the importance of position information on token-level contextual modeling, E5 disables position information and introduces FineCoS. Though E5 outperforms E2, it still falls behind E7 if without position information, which proves that position information helps model finegrained contextual knowledge. Our best model outperforms the basic ColDec with 6.65% absolute F1 improvement and 16.4% relative CER reduction on test-name, and with 1.64% absolute F1 improvement and 2.9% relative CER reduction on test-term. We conjecture that the improvements on test-name are due to the phrase-level confusion reduction brought by phrase selection and context purification, and the token-level uncertainty reduction brought by token attention that especially benefits rare word recognition.

Results on LibriSpeech
As mentioned in [13], the large biasing list introduces high correlations between phrases, and thus causes performance degradation. In Table 4, we investigate on test-name with E2 and E7 to verify the efficacy of position information and FineCoS on mitigating the degradation caused by the large biasing list. When not injecting distractors, the performance gain of E7 over E2 is limited. However, as the number of distractors grows, the gap between them widens. Finally, injecting 2,400 distractors causes 3.03% CER degradation in E2 but only causes 1.25% CER degradation in E7. These results prove that position information and FineCoS cooperatively mitigate the degradation caused by large biasing lists. Table 5. Examples from left to right columns are references, hypotheses of S1 and hypotheses of S4, respectively. To explore how FineCoS improves recognition, we extract examples from LibriSpeech test sets and analyze an example in terms of attention distribution. As shown in Table 5, compared with S1, enhanced ColDec with FineCoS (S4) shows advantages on correcting token predictions. Here, we analyze the second example. For "ki@@" in "kioscurists" generated by S1, we found its top 3 most relevant phrase candidates are "chiaroscurist", "chiaroscurists", and no-bias option. This proves that the phrase-level CPN attention module captures the relevant contextual knowledge, but the CPN decoder does not fully use phrase-level contextual knowledge to correctly bias predictions. As for "chi@@" in "chiaroscurists" generated by S4, the selected phrases in FineCoS includes "chiaroscurist" and "chiaroscurists". As expected, the "chi@@" in these two phrases dominates the token-level attention distribution (over 80%), which validates that FineCoS narrows the range of phrase candidates and token candidates, and finally boosts token-level predictions.

CONCLUSION
In this paper, we improve E2E contextual speech recognition with fine-grained contextual knowledge selection, context purification, and position information, hoping these methods alleviate the confusion at different granularities. The proposed methods improve the basic contextual biasing method when studied on LibriSpeech and our large-scale in-house dataset. Although our method is developed on ColDec to customize the CIF-based ASR, we believe that our thoughts could be extended to other contextual biasing methods and other E2E models.