{'fraction_non_alphanumeric': 0.03608786610878661, 'fraction_numerical': 0.014382845188284518, 'mean_word_length': 4.407211028632026, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Class LM and word mapping for contextual biasing in End-to-End ASR

In recent years, all-neural, end-to-end (E2E) ASR systems gained rapid interest in the speech recognition community. They convert speech input to text units in a single trainable Neural Network model. In ASR, many utterances contain rich named entities. Such named entities may be user or location specific and they are not seen during training. A single model makes it inflexible to utilize dynamic contextual information during inference. In this paper, we propose to train a context aware E2E model and allow the beam search to traverse into the context FST during inference. We also propose a simple method to adjust the cost discrepancy between the context FST and the base model. This algorithm is able to reduce the named entity utterance WER by 57% with little accuracy degradation on regular utterances. Although an E2E model does not need pronunciation dictionary, it's interesting to make use of existing pronunciation knowledge to improve accuracy. In this paper, we propose an algorithm to map the rare entity words to common words via pronunciation and treat the mapped words as an alternative form to the original word during recognition. This algorithm further reduces the WER on the named entity utterances by another 31%.


INTRODUCTION
Conventional speech recognition systems include several main components: acoustic model, language model, and pronunciation dictionary. Each of them is separately constructed and optimized. In recent years, an all-neural, end-to-end (E2E) model that directly converts speech into text through a sequence model became popular. Instead of separately optimized components, the E2E model is a single trainable neural network. It removes the HMM assumptions and enables end-to-end optimization. Architectures like Connectionist Temporal Classification (CTC) [1], attention based sequence models such as Listen, Attend and Spell (LAS) [2], Recurrent Neural Network Transducer (RNN-T) [3] have obtained impressive results. In particular, architectures like LAS can sometimes outperform the conventional system [4,5]. The base LAS model requires the whole input sequence before it can compute the attention. This makes it infeasible for online streaming application. In [6,7], a Monotonic Chunkwise Attention (MOCHA) was proposed for the attention based sequence-to-sequence model. The LAS-MOCHA structure becomes the base model for our study, although our proposed methods should be directly applicable to other types of E2E model.
Users' voice requests often involve personal content like contact names, app names, music titles, etc. There are a few issues here. First, such personal content frequently includes rare and foreign words. The general training set has very few occurrences of such words. Second, each user's personal content is different. The common entities may not be what the users want. For example, if a user has a contact Jain Smith, a phrase like call Jain Smith may be misrecognized as call Jane Smith. Another user may have a contact Jaine Smith. It's important to inject the userspecific content during inference.
In conventional hybrid HMM-DNN system, the contextual information is usually represented as a Weighted Finite-State Transducer (WFST, [8]), and injected into the main FST graph during the recognition [9,10,11]. A single E2E model lacks the flexibility to inject the contextual information during recognition. There have been various attempts to improve it, e.g. in [12], a separate attention component is added to model the contextual phrases (a.k.a. bias phrases); in [13,14,15], an on-the-fly rescoring (a.k.a. shallow fusion) with bias phrases was proposed. Our proposed method is similar to [13,15] with a few important differences. We identify the bias phrases in the training data and model the transitions between regular words and bias words, while during inference, the user-specific bias phrases are inserted as WFST graphs at the relevant place in the beam search. We propose to normalize the scores between the paths from the contextual bias FST and from the base search space. It's hard for the E2E system to output words it has never seen in the training data. Such words are especially prevalent in user contact lists and music lists. We propose a method to do word mapping, i.e, transform the rare words to common words through pronunciation. This method makes use of an existing lexicon and achieves nice accuracy gain. arXiv:2007.05609v1 [cs.CL] 10 Jul 2020

BASE LAS-MOCHA MODEL
Given the speech sequence x = {x 1 , ..., x L } with length L, and output word sequence y = {y 1 , ..., y U } with length U , the E2E model computes the probability An encoder converts x to intermediate outputs h = {h 1 , ..., h T } through a Recurrent Neural Network (RNN), typically an LSTM. One important aspect for ASR is there are many more speech frames than the number of output tokens, usually there is a reduction factor N , thus T = L N .
The decoder is also an RNN. It takes encoder outputs h (a.k.a. memory), and previous output token y i−1 , generates the current decoder state s i : which is then passed through a generation network, typically a feedforward network with softmax output, to produce the next output token y i : c i is the vector at decoder step i that summarizes information from the encoder: Where β i,j is attention weight at output step i on j-th encoder output. The difference between original LAS and the MOCHA attention is how β i,j is computed. MOCHA defines two levels of score functions for attention weight computation. Please refer to Section 2 in [7] for details.
With BPE tokens, we can in theory cover all the words in a language, so we don't have out-of-vocabulary (OOV) problems. This is important for the user-specific named entities. From the speech-text pair training data, we relabel the transcription to insert class LM tags. E.g, for utterance call Jain Smith mobile, it's relabelled into call @contact# Jain Smith #contact@ mobile; similarly for utterance open ClassDojo, it's relabelled into open @app# ClassDojo #app@. The tokens @contact#, @app#, #contact@, #app@ are class enter and exit tokens for classes contact and app respectively. The class tags are excluded from BPE processing. Human transcription is based on existing speech recognition system output that includes class LM tags. We apply an edit distance alignment between transcription and recognition output, then insert the class LM tags into the transcription. Alternatively, a process like named entity tagging can be used. Note that tagging does not need to be perfect as the content between the enter and exit tags will be replaced by user-specific FST during inference.

Contextual Bias FST Construction
The contextual bias FST is a transducer from BPE subword sequence to word sequence (phrase). The cost on the FST arcs is derived from the relevant frequency f i for phrase i: where N l is the number of phrases in this context FST. If there are M arcs on the path for this phrase, actual cost on each arc is The transducer T is determinized and minimized: We build one FST per class. Each use case (user-specific, location-specific, etc.) will have its own FSTs.

Inference
The contextual bias FST is constructed before inference. During inference, if an active token in the beam is one of the class enter tags like @contact#, the corresponding context FST will be activated and its path will be traversed. Such path will compete against other paths inside the context FST, and also at the same time, compete against the paths in the base search space. When a final state of a context FST is reached, it traverses the class exit tag #contact@, and gets back to the base search space. For t-th step of beam search, the score S t i for i-th beam is: where P b (·) is probability from the base E2E model and P c (·) is probability from the contextual bias FST, and λ c is the scale on the context FST. For the path outside of the context FST, the second term is 0, this makes the context FST path always worse than the base path (since log of probability is a negative number). Therefore, we define the following scoring function for paths outside of the context FST: Where λ b is a scale, and Γ t is the normalization score at step t: Where κ t is the number of active paths from the context FSTs at step t. If all the paths at step t are in the base search space, Γ t = 0; if any paths at step t are inside a context FST, Eq. 9 defines S t i for paths inside the context FST, and Eq. 10 defines S t i for paths in the base search space. Note that the actual ranking of the paths at step t are based on the accumulated score:

WORD MAPPING THROUGH PRONUNCIATION
The E2E system can output graphemes or words, therefore it does not need a pronunciation dictionary. On the other hand, we have created a large pronunciation dictionary in the many years of conventional system development. The graphemeto-phoneme (G2P) system is also mature. It is an interesting research question how to inject such knowledge into the E2E system. Rare or even foreign words are common in the named entity phrases. The E2E system has difficulty generating words it rarely sees. In this section, we describe a method to convert the rare named entity words into more common words through pronunciation so they are easier to recognize.
Given a word n-gram model G trained from the text data used in the E2E model training, and a pronunciation dictionary (lexicon) L, we construct For a word W , its pronunciation is represented as a phoneme FST P W , the pronunciation is either from a human generated lexicon or a G2P system and it may have multiple pronunciations for W . A new word W is obtained through Then the subword list corresponding to W is used as the subword list for W for contextual bias FST construction. We will also show experiments that adding the list to W instead of replacing the original one.

EXPERIMENTS
We use English data in all our experiments. The base E2E model is LAS-MOCHA. The encoder is 5-layer unidirectional LSTM, 1400 cells, with 700-dim projections. The 40-dim mel-filter bank is the input to the encoder. 4-head, 800-dim, chunk-size 2 MOCHA attention is used. The decoder uses 2-layer unidirectional LSTM, each has 800 cells, and 400-dim projection. The text is processed with 6.4K BPE tokens. We use two test sets, first is a named entity rich set (named entity set) with 13K utterances. Sample utterances include "call Jain Smith mobile" and "open ClassDojo"; the other has no named entities (regular set) and has 21K utterances. Note that the user-specific context FSTs are loaded even for the regular set so this is a good test set for measuring the "anti-biasing" phenomenon. Both test sets are typical voice assistant requests. The E2E model is trained with block-momentum SGD [17], 32 V100 GPUs, 20000 frames per minibatch, L2 normalization on gradient, and cross-entropy criterion, with label smoothing [18] weight 0.05 and schedule sampling [19] weight 0.1. The initial learning rate is 0.025 and decayed by factor 0.8 when no improvement observed on the validation set. The model was trained for 25 epochs. We use beam size 8 during beam search, and length penalty [20] 0.1.

Contextual bias FST in inference
The first set of experiments are to confirm the effectiveness of the proposed context FST algorithm in the inference. The amount of training data for the E2E model is a few thousands hours here. The class LM enter-exit tags can help the model learn the transition between regular words and named entity words. This is evident from the results in Table 1, where only the class LM tags were added to the training text and the accuracy is already improved for both named entity test set and regular test set.
Next, the contextual bias FST is added in the inference. When λ c = 0, it means no context FST is used; when λ b = 0, it means no score normalization is used. Table 2 shows the impact of using context FSTs and score normalization. From setup 1 to 2, using context FST and scale 0.1 reduces the WER from 18.1 to 14.4; from setup 2 to 3, turning on score normalization alone reduce the WER from 14.4 to 11, changing the context FST scale from 1.0 to 0.1 further reduces the WER to 9.0. The overall WER reduction of using context FST is 50% (from 18.1 to 9.0). The last column shows the WER on the regular set, the WER is increased slightly from 8.4 to 8.5.

Word mapping through pronunciation
The E2E model used in this experiment is trained with four times more data, SpecAugment [21], and Minimum Word Error Rate criterion (MWER, [22,23]), the weight on the crossentropy loss during MWER is 0.05. We ran one epoch of For word mapping, the unigram trained on the text portion of the training data is used and the base lexicon has less than a million unique words. A bi-LSTM based G2P was used to generate pronunciations for the OOV words in the user's context lists. The results are shown in Table 3. We use the configuration from setup 4 in Table 2 for the context FST. The WER reduction from context FST is still big (from 15.3 to 6.5, a 57.5% reduction). When using the subword sequence from the mapped words, the WER is reduced to 5.5. Interestingly, if using both the original sequence and mapped sequence in the FST, the WER is further reduced to 4.5. That's about 31% error reduction over original context FST. The overall WERR over the no-FST baseline is 70.6%, with total inference time increased by 10%. Fig. 1 shows the WER breakdown in terms of number of bias phrases for the utterances. The WER reduction is consistent across utterances with different number of bias phrases.
We examine some mapped words in Table 4, the first column is the original word surface form, the second column is the mapped-to words through pronunciation, the third column is the recognized words when the original context FST is used. After using the subword sequence of the mapped word in the FST, these cases are fixed. From Table 4, we observe that rare words are mapped to more common words, either one-to-one mapping, or the original word is decomposed into multiple more common words, or less frequently, a phrase is reduced to a single word.

CONCLUSIONS
User-specific content or location dependent content are challenging for E2E models since the model directly converts speech to text. In this study, we have empirically demon-  have proposed a simple score normalization algorithm. The accuracy improvement is 57%. We have also proposed to transform words through pronunciation. This algorithm converts the rare and unusual words into more common words so they are easier to recognize. This algorithm improves the accuracy by another 31%. The overall accuracy improvement over the no-FST baseline is 70.6%. Importantly, the proposed techniques cause little degradation on the non-named-entity type utterances. In the future, we plan to incorporate this word mapping process into training so the model is more accustomed to the sequences from both the original word and mapped-to word.

ACKNOWLEDGEMENT
We would like to thank Matt Mirsamadi, Kyuyeon Hwang, Tim Ng, Roger Hsiao, Henry Mason, Leo Liu, Arnab Ghoshal, Yuchen Zhang, Man-Hung Siu, and John Bridle for helpful discussions.