{'fraction_non_alphanumeric': 0.032247535739930805, 'fraction_numerical': 0.008616750440629284, 'mean_word_length': 5.031496062992126, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
CONVERSATIONAL SPEECH RECOGNITION LEVERAGING EFFECTIVE FUSION METHODS FOR CROSS-UTTERANCE LANGUAGE MODELING

Conversational speech normally is embodied with loose syntactic structures at the utterance level but simultaneously exhibits topical coherence relations across consecutive utterances. Prior work has shown that capturing longer context information with a recurrent neural network or long short-term memory language model (LM) may suffer from the recent bias while excluding the long-range context. In order to capture the long-term semantic interactions among words and across utterances, we put forward disparate conversation history fusion methods for language modeling in automatic speech recognition (ASR) of conversational speech. Furthermore, a novel audio-fusion mechanism is introduced, which manages to fuse and utilize the acoustic embeddings of a current utterance and the semantic content of its corresponding conversation history in a cooperative way. To flesh out our ideas, we frame the ASR N-best hypothesis rescoring task as a prediction problem, leveraging BERT, an iconic pre-trained LM, as the ingredient vehicle to facilitate selection of the oracle hypothesis from a given N-best hypothesis list. Empirical experiments conducted on the AMI benchmark dataset seem to demonstrate the feasibility and efficacy of our methods in relation to some current top-of-line methods.


INTRODUCTION
Recently, automatic speech recognition (ASR) has achieved remarkable success, thanks to the synergistic breakthroughs in neural model architectures and training algorithms [1] [2]. Despite the great progress, current ASR systems are vulnerable to performance degradation for conversation-related applications such as virtual assistants, smart speakers, interactive voice responses (IVR) and among others. The difficulties faced by conversational ASR is due mainly to its free-style speaking traits and long-term contextual dependencies across multiple utterances or speaker turns [3] [4]. Fortunately, topical coherences and frequently repeated words often reside in consecutive utterances. A simple and lightweight approach to dealing with conversational speech recognition is to endow a language model (LM) with the ability to capture long-span lexical/semantic dependencies among words or utterances and in turn use it to rerank the initial hypotheses generated by an ASR system. N-best hypothesis reranking is deemed to be a plug-and-play method for enhancing the performance of conversational ASR, which aims to find the oracle hypothesis that has the lowest word error rate from a given N-best hypothesis list. Most of the recent studies on cross-utterance language modeling have focused mostly on effective use of recurrent neural network (RNN), long short-term memory (LSTM) and Transformer-based language models, to name just a few, to compute an updated LM score for each hypothesis in an autoregressive manner for hypothesis reranking. In [5], an LSTM-based neural cache mechanism was proposed, which encodes consecutive words occurring across historical utterances into a fixed-length vector representation. The vector representation is in turn used to adapt a given neural LM. In addition, a so-called dynamic evaluation method endeavored to fit an LSTM LM in the recent historical sentences via gradient descent-based adaptation [6]. Yet more recently, an LSTM-based fusion module was introduced to modulate transformer-based LMs [7][8] for better modeling power. However, RNN and LSTM inherently suffer from the sharp nearby, fuzzy far away issues [9] and may fail to capture complex global structural dependencies among these utterances.
Building on these observations, we in this paper seek to integrate cross-utterances, long-term word interaction cues into ASR N-best hypothesis reranking with two disparate fusion strategies: early fusion and late fusion. The former performs input augmentation, which simultaneously considers within-sentence and cross-sentence interactions among words for hypothesis reranking. The latter, in contrast, renders the semantic relevance between the conversation history and the holistic embedding of each hypothesis. Furthermore, a novel audio-fusion mechanism is introduced, developed to fuse and utilize the acoustic embeddings of a current utterance in conjunction with the semantic content of its corresponding conversation history. To turn the idea into reality, the ASR N-best hypothesis rescoring task boils down to a prediction problem, leveraging BERT (short for bidirectional encoder representations from Transformers [10]), an iconic pre-trained LM, as the fundamental vehicle to facilitate selection of the oracle hypothesis from a given N-best hypothesis list.
The rest of the paper is organized as follows. Section 2 describes the proposed methodology for N-best list reranking. The experimental setup and results are reported in Section 3, followed by the conclusion and future research directions in Section 4.

BERT-based ASR N-based hypothesis reranking
Recently, recurrent neural LMs, such as those stemming from RNN, LSTM [11] [12] and others, have enjoyed much success on a wide variety of ASR tasks. These LMs in general are hardly be used at the first-pass decoding stage of ASR due to the exponential growth of hypothesis search space with the increase in the modeling context of an LM. As such, an alternative and lightweight therapy is to employ them to compute an LM score for each hypothesis at the second-pass (viz. N-best hypothesis reranking) stage, making most ASR modules remain unchanged. However, recurrent neural LMs and the like usually operate in an autoregressive manner, which would be computationally expensive and hinders them from being used in production systems.
Departing from these methods, an effective BERT-based modeling framework for ASR N-best hypothesis reranking has been prototyped in our recent work [13] [14]. This framework aims to predict a hypothesis that would have the lowest WER (i.e., the oracle hypothesis) from an N-best list (denoted by PBERT) in a one-shot, non-autoregressive (NAR) manner. In realization, PBERT consists of two model components, namely BERT stacked with an additional prediction layer which is a simple fully-connected feedforward network (FFN) for oracle hypothesis prediction, as graphically illustrated in Figure 1(a). For a given ASR N-best hypothesis list, each hypothesis of the list is respectively taken as an input to the BERT component, and meanwhile [CLS] and [SEP] tokens are inserted at the beginning and end of each hypothesis, respectively. In turn, the resulting embedding vector of [CLS] is used as a semantic-aggregated representation of the input hypothesis. The [CLS] embedding vectors of all the N-best hypotheses are further augmented with their respective initial ASR decoding scores and consequently spliced together to be fed into the FFN component to output a prediction score (with the Softmax normalization) for each hypothesis.
Given a set of training utterances, each of which is equipped with an N-best hypothesis list generated by ASR and the indication of the oracle hypothesis that has the lowest WER, we can train the FFN component and fine-tune the BERT component accordingly. We should also note here that there have been some very recent studies that applied BERT as a mask-predict LM to rescore an ASR hypothesis in terms of the so-called pseudo-log-likelihood (PLL) score for N-best hypothesis reranking (denoted by MBERT for short) [15][16] [17]. However, unlike PBERT, MBERT processes each hypothesis in an isolated and autoregressive manner, and its speed is normally orders of magnitude lower than PBERT.

Fusion of cross-utterance history information
For conversational ASR tasks, a sequence of consecutive utterances may jointly contain many important conversation-level phenomena such as topical coherence, lexical entrainment [18], and adjacency pairs [19] across utterances. It is therefore anticipated that if crossutterance conversation-level information clues could be infused into language modeling, the performance of ASR N-best hypothesis reranking, will be considerably improved. Following this line of thought, we in this paper seek to integrate cross-utterances, longterm word interaction cues into the ASR N-best hypothesis reranking process, taking PBERT as the backbone LM, with two disparate fusion strategies: early fusion and late fusion.

Early fusion
For the early-fusion strategy, the BERT module is modified to take input the concatenation of each hypothesis of the current utterance and the topmost ASR hypotheses (sequentially generated by N-best hypothesis reranking) of preceding utterances with a special symbol [SEP] as the delimiter, as shown in Figure 1(b). We thus can generate a cross-utterance, history-aware embedding for each hypothesis of the current utterance to achieve enhanced oracle hypothesis prediction (denoted by EHPBERT). Note here that EHPBERT by nature performs word-or token-level fusion between the conversation history and each hypothesis of the current utterance [14].

Late fusion
Unlike the early-fusion strategy that modifies the BERT module to conduct cross-utterance, multi-head attention between words (or (1) The resulting context embedding is in turn appended to the original semantic representation of each hypothesis of the current utterance to be fed in to the FNN module for oracle hypothesis prediction. We refer to this method as LHPBERT for short hereafter.

Audio fusion
It is anticipated to obtain enhanced ASR N-best hypothesis reranking by modeling cross-utterance, word-word interaction and semantic cues with either EHPBERT or LHPBERT. On a separate front, we believe that a reconsideration of the context relationship between the semantic representation of each hypothesis and the intermediate frame-wise acoustic embeddings E = ( 1 , … , ) generated by the encoder module of an end-to-end (E2E) ASR system would provide extra discriminative information to additionally benefit N-best hypothesis reranking. Figure 2 (a) shows a schematic description of such an audio-fusion strategy, while the attention mechanism can be succinctly expressed by where ⨀ represents the elementwise dot production and W , W , 2 and 3 are trainable parameters. More specifically, the functionality of Eqs.
(2) and (3) is to model the context correlation between the sequence of acoustic representations E and the semantic representation of each hypothesis , producing a gating vector with the Sigmoid function. After that, Eq. (4) seeks to obtain a modulated semantic vector ′ with the gating vector and the Relu function in succession.

Experimental setup
We evaluate our proposed methods on the AMI benchmark meeting transcription data base and task [21]. This speech corpus consisted of utterances collected with the individual headset microphones (IHM), while a pronunciation lexicon of 50K words was used. Table  1 shows some basic statistics of the AMI meeting corpus for our experiments. The baseline ASR system (which was employed to produce the N-best hypothesis list for a given utterance) adopted a pretrained Transformer-based model 1 with ESPnet toolkit [22]. The baseline LSTM LM was also built with ESPNet 2 , which consisted of 2 layers and 256 neurons for each. As to the various BERT-based LMs, we adopted the HuggingFace package [23] to develop them, using the bert-base-uncased 3 model as the initial model. It had 12 Transformer blocks, each of which contained 12 multi-head self-attention operators and 768 hidden units. The BERT-based LMs were finetuned with 2 epochs and optimized with AdamW method [24]. In 3 https://s3.amazonaws.com/models.huggingface.co/bert/bert-baseuncased.tar.gz  addition, the learning rate was set to 5e-5, the batch size was set to 16 and the gradient was accumulated 32 times for each update.

Experimental results
At the outset, we report on the word error rate (WER) result obtained by the ASR system and the those of the oracle hypothesis in the Nbest list. As the graphical depiction shown in Figure 3, the top-one hypothesis of the ASR system is 19.10%. In addition, the WER result of the oracle hypothesis is 12.87% when N is set equal to 10, whereas increasing the value of only leads to a slight WER reduction of the oracle hypotheses (when > 10). As such, we will set N=10 for the following ASR N-best hypothesis reranking experiments, unless stated otherwise. Furthermore, the baseline Nbest hypothesis results achieved by LSTM and PBERT are shown in Table 2. As can be seen from Table 2, both LSTM and PBERT can provide substantial WER reduction when compared to the top-one result of the ASR system (with WER of 19.10%). PBERT outperforms LSTM by a margin of 3.14% relative WER reduction, and the former is much faster than the later due to its one-shot execution process.
In the second set of experiments, we turn to the evaluation of the effectiveness of cross-utterance language modeling with the two fusion strategies proposed in this paper, viz. early function and late function (cf. Section 2.2). Their corresponding WER results are shown in Table 3. Inspection of Table 3 reveals several noteworthy points. First, early fusion (denoted by EHPBERT) and late fusion (denoted by LHPBERT) perform in line with each another. Second, early fusion has its best WER result when two immediately preceding utterances (ν = 2) are taken into consideration, while late fusion delivers its best WER result when 10 immediately preceding words in the conversation history (ω = 10) are used. Third, combination of early fusion and late fusion can yield an additional improvement, leading to an WER result of 17.64%.
In the last set of experiments, we assess the utility of integrating audio fusion (cf. Section 2.3) into the various BERT-based LM proposed in this paper. The corresponding results are shown in Table  4. A closer look at Table 4 reveals at least two things. First, the additional adoption of audio fusion can provide extra gains for all variants of our BERT-based LMs, although the respective WER reductions are less pronounced. Second, when combining these three fusion strategies (viz. late fusion, late fusion and audio fusion) all together, we can obtain the best WER result of 17.56%, which ultimately offers relative WER improvements of 4.82% and 1.73% in relation to LSTM and PBERT, respectively.

CONCLUSION AND FUTURE WORK
In this paper, we have explored different fusion strategies, viz. late fusion, late fusion and audio fusion, for ASR N-best hypothesis reranking. These fusion strategies work in tandem to infuse crossutterance word interaction and semantic cues, as well as the relatedness between the semantic embedding of each hypothesis and acoustic embeddings the test utterance, into language modeling. A series of empirical evaluations on the AMI benchmark meeting transcription task seem to confirm the practical utility of our methods in comparison to a de facto standard LSTM LM. In future work, we plan to investigate more effective attention mechanisms to infuse various lexical, semantic and acoustic cues into language modeling for ASR. It is also worthwhile to further validate our LM methods on more diverse ASR tasks, such as TED-LIUM 4 , LibriSpeech 5 and others.