{'fraction_non_alphanumeric': 0.03497735279315551, 'fraction_numerical': 0.00918470055359839, 'mean_word_length': 4.942803738317757, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Cross-Modal Transformer-Based Neural Correction Models for Automatic Speech Recognition

We propose a cross-modal transformer-based neural correction models that refines the output of an automatic speech recognition (ASR) system so as to exclude ASR errors. Generally, neural correction models are composed of encoder-decoder networks, which can directly model sequence-to-sequence mapping problems. The most successful method is to use both input speech and its ASR output text as the input contexts for the encoder-decoder networks. However, the conventional method cannot take into account the relationships between these two different modal inputs because the input contexts are separately encoded for each modal. To effectively leverage the correlated information between the two different modal inputs, our proposed models encode two different contexts jointly on the basis of cross-modal self-attention using a transformer. We expect that cross-modal self-attention can effectively capture the relationships between two different modals for refining ASR hypotheses. We also introduce a shallow fusion technique to efficiently integrate the first-pass ASR model and our proposed neural correction model. Experiments on Japanese natural language ASR tasks demonstrated that our proposed models achieve better ASR performance than conventional neural correction models.


Introduction
Neural networks have dramatically improved the performance of automatic speech recognition (ASR) systems. In particular, end-to-end ASR systems that directly convert an input speech utterance into an output text (e.g., characters, subwords, words) have attracted significant attention. Recent studies have investigated various end-to-end ASR methods including connectionist temporal classification [1][2][3], recurrent neural network transducers [4,5], and attention-based encoder-decoder models with recurrent neural networks [6][7][8] or transformers [9,10]. Though these end-to-end ASR methods have made great progress, their performance is insufficient when executing one-pass decoding using a single ASR model.
Various post-processing methods have been proposed to compensate for this insufficiency. One existing methods is to use neural language models with fusion techniques [11][12][13] or rescoring hypotheses (or lattices) from first-pass ASR [14][15][16]. Furthermore, methods that directly map ASR hypotheses to its reference text by using neural correction models have been investigated. One type of the neural correction models is text-totext conversion-based models that convert the ASR hypothesis into the correct sentence [17][18][19][20]. Besides, the neural correction model in [21] leverages both input speech and its ASR hypotheses as the input contexts for the encoder-decoder networks. The model separately encodes the speech and hypothesis embeddings and individually attends to each encoded information with attention mechanisms [22,23]. However, this method cannot explicitly take into account where to correct the input ASR hypothesis because an encoder cannot mutually detect the positions of input in the other encoder.
We propose cross-modal transformer-based neural correction models that jointly encode speech and the ASR hypothesis to take their relationships into consideration. Our proposed models use the self-attention in the transformer [10] to take into account the cross-modal relationships between the speech and ASR hypothesis. In our proposed models, two different modals are jointly encoded using cross-modal self-attention. We assume that the cross-modal self-attention enables the models to effectively learn the relationship between speech and hypothesis. In contrast, a conventional neural correction model [21] separately encodes the speech and ASR hypothesis. In short, the advantage of our proposed models is that different modal inputs are effectively encoded while considering each other. Our proposed models are assumed to explicitly take into account where to correct the input ASR hypothesis by capturing the relationship between it and the speech. Finally, the decoder network computes the generative probabilities of tokens using the context vectors and generates the refined result by beam-search decoding. Hu et al. used only the decoder of their neural correction model during beam-search decoding to generate refined results [21]. In contrast, we introduce shallow fusion [11,12] to integrate first-pass ASR and neural correction models. This enables us to complementarily use information from both decoders in ASR and neural correction models. To the best of our knowledge, this is the first study that introduces cross-modal processing and shallow fusion into neural correction models for ASR.
We verify the effectiveness of our models with Japanese ASR tasks. We prepare separate attention-based models to compare with our models.

Related Work
Neural correction models are closely related to neural language models. In particular, encoder-decoder-based neural language models conditioned by speech information [24][25][26] and hypotheses from ASR [27] are similar to neural correction models. These language models were often applied so as to rescore ASR hypotheses, i.e., n-bests. In contrast, neural correction models directly generate a sentence from ASR hypotheses and speech.
Our proposed models are related to studies that use selfattention to joinly encode cross-modal input features [28][29][30]. These studies proposed cross-modal attention of visual and linguistic features with transformer-based neural networks. In this paper, we focus on speech and text information for neural correction models for ASR and use self-attention to take into account the relationships between them.

End-to-End ASR
An end-to-end ASR system directly converts acoustic features of input speech into a token sequence. In this study, we use auto-regressive encoder-decoder models that predict the probability of a token given the previous predicted tokens. Given speech X = {x1, · · · , xI }, the encoder-decoder estimates the generative probability of a token sequence W = {w1, · · · , wT }, where I is the number of the acoustic features in the input speech and T is the number of the tokens in the token sequence. The generative probability of a token sequence is defined as where Λ represents the trainable parameters. The model parameters Λ in an end-to-end ASR system are updated to maximize the generative probability in the decoder when give an input speech. Thus, the model parameters are optimized by minimizing the cross entropy loss function: where D is the training set.

Transformer-Based Neural Correction Models for ASR
An ASR system generates a hypothesis from an input speech. Neural correction models then generate a refined result from the hypothesis and input speech. We use a transformer encoderdecoder for neural correction models. Given the acoustic feature sequence X = {x1, · · · , xI } and the ASR output generated by the ASR system F (·), neural correction models estimate the generative probability of W = {w1, · · · , wT } as where C = {c1, · · · , cJ } is the ASR hypothesis from the ASR system and Θ represents the model parameters in the neural correction model. Figure 1 illustrates the network structure of our proposed models that involves encoding speech and the ASR hypothesis with a cross-modal transformer. Our proposed models consist of embedding networks for speech and hypothesis. Speech Embedding: The acoustic features are embedded into continuous representations as  where SpeechEmbedding(·) is a neural network that converts acoustic features into continuous vectors, PosEncoding(·) is a function that adds a continuous vector in which position information is embedded and θx is the trainable parameter.

Proposed Method
Text Embedding: Each token cj in a hypothesis C = {c1, · · · , cJ } is encoded to one-hot representation and embedded into continuous representation as where TextEmbedding(·) is a function that converts a token into a continuous representation, and θ d is a trainable parameter.
Cross-Modal Encoder: The two embedded vectors are concatenated into a single sequence e as where ssep is a continuous representation of a separator token [sep]. We define the output of the m-th transformer encoder block as f m and the input of the first block is defined as f 0 = e. The computational process in the m-th block of the transformer encoder is defined as where CrossModalEncoder(·) is a transformer encoder including a scaled dot-product multi-head self-attention layer and a position-wise feed-forward network and θ f is a trainable parameter.
Text Decoder: The output of the final block f M in the crossmodal encoder is fed into the transformer decoder. When the output of the t-th time step for the n-th transformer block in the decoder is q n−1 t , the transformer decoder constructs a hidden representation with the context vector U n t from the encoder as q n t = TransformerDecoder(q n−1 1:t−1 , U n t ; θq),  where TransformerDecoder(·) is a transformer decoder including a scaled dot-product multi-head masked selfattention layer and a position-wise feed-forward network, SrcTgtAttention(·) is a scaled dot product multi-head source-target attention layer, M is the number of blocks in the transformer encoder, and θ h is the trainable parameter. The input of the first block is token embedding, which is calculated as where θw is the trainable parameter. The network estimates the probabilities of a distribution of the output tokens as P (wt|w1:t−1, X, C; Θ) = Softmax(q N 1:t−1 ; θo), where Softmax(·) represents the softmax function with linear transformation, N is the number of blocks in the transformer decoder, and θo is the trainable parameter. Finally, beam search decoding is conducted while calculating the probability distribution. The model parameters can be summarized as Θ = {θx, θ d , θ f , θU , θq, θw, θo}. Figure 2 illustrates the network structure of a conventional neural correction model that separately encodes speech and hypothesis by each encoder. The input speech and hypothesis are converted into continuous representations si and dj in the same manner as in Eqs. (5-6) and (7)(8). The embeddings s1:I and d1:J are input into the speech and text encoders. We define the output of the l-th speech encoder block and m-th text encoder block as f l and g m respectively. In this case, the inputs of the first blocks are defined as f 0 = s1:I and g 0 = d1:J . The computational process in the l-th block of the transformer speech encoder and m-th block of the transformer text encoder are defined as

Conventional Method
where SpeechEncoder(·) and TextEncoder(·) are transformer encoders including a scaled dot product multi-head selfattention layer and a position-wise feed-forward network and θ f and θg are the trainable parameters. The final outputs of the two encoders are attended to by source-target attention mechanisms and concatenated as where L is the number of blocks in the transformer of the speech encoder, and M is the number of blocks in the transformer of the text encoder. The decoder and the probability calculation are the same as those in the proposed model (Eq. (12-15)). The model parameters can be summarized as Θ = {θx, θ d , θ f , θg, θ U , θ U , θq, θw, θo}.

Shallow Fusion of ASR and Neural Correction Model
Shallow fusion [11,12] is a technique to incorporate an external model by log-linear interpolation at inference time. We use this technique to merge the first-pass ASR model and neural correction model. We use the following criterion during beam search decoding: where α is the weight of the neural correction model.

Training
In the training step, the parameters Θ for a neural correction model are updated to maximize the conditional generative probability in the decoder when given an input speech and the ASR hypothesis as a context from the encoder. Thus, these parameters are optimized by minimizing the cross entropy loss function: where D is the training set. Our proposed and conventional models can be trained with the same criterion. Note that these neural correction models can be trained from the same training data as ASR since the C is obtained from function F (·) in Eq. (4).

Setups
We used two corpora: the corpus of spontaneous Japanese (CSJ) [31] and our home-made corpus of natural two-person dialogue corpus (NTDC), which consists of about 24 hours of speech and its transcriptions. We split NTDC into 22 hours for the training set, 1 hour for development set, and 1 hour for the evaluation set. CSJ, which has about 545 hours of speech, was used for training data. We used three standard evaluation sets for CSJ (CSJ1, CSJ2, and CSJ3). We used a transformer-based encoder-decoder ASR model as the baseline (Baseline). The encoder and decoder each had six transformer blocks. The token embedding dimension, hidden state dimension, non-linear layer dimension, and the number of heads were 256, 256, 2048, and 4, respectively. The acoustic features were transformed by two layers of 2D convolutional neural network. In our proposed cross-modal  The acoustic features was a 40-dimensional log Melfilterbank with delta and acceleration coefficients. We applied SpecAugument [32] during training of all models. The vocabulary size was 3285 characters made from CSJ and NTDC. In the cross-modal transformer-based model, we added the special token [sep] to the vocabulary. We used Adam optimizer with Noam learning rate scheduler with 25000 warmup steps. When decoding by beam search, the beam size was set to 20. Table 1 shows the character error rate (CER) performance when using the baseline system and two correction-based systems with and without shallow fusion. Cross-modal transformerbased model and separate attention-based model showed similar performance when shallow fusion was not performed. When using shallow fusion, the cross-modal transformer-based model outperformed the separate attention-based model. These results indicated that cross-modal transformer effectively learned the relationship between the speech and hypothesis. We found that shallow fusion efficiently integrated ASR and cross-modal transformer-based model. Figure 3 shows the relationship between the weights of neu- ral correction models and CER in shallow fusion. Cross-modal transformer-based model showed better performance over a wider range than the separate attention-based model. Crossmodal transformer-based and separate attention-based model with shallow fusion showed better performance than baseline for all weights. This indicates that shallow fusion can efficiently integrate ASR and neural correction models. Figure 4 gives an example of the self-attention weights between key and query values in the first block for the cross-modal transformer-based model. Each image shows the weights of each head in the self-attention in the transformer encoder. The network detected the boundaries and extracted the features of both speech and ASR-hypothesis embeddings. It is assumed that head-1 extracted ASR-hypothesis information to determine the correspondence with the ASR hypothesis and output text, head-2 and head-4 extracted both speech and ASR-hypothesis information to determine the correspondence with the two embeddings, and head-3 extracted speech information to determine the correspondence with speech and output text. All the images in Figure 4 show that the cross-modal transformer functioned as expected.

Conclusions
We proposed cross-modal transformer-based neural correction models, which jointly encode speech and hypothesis. The key strength of our proposed models is effectively capturing the relationships between two different modals for refining ASR hypotheses. From experiments involving CERs, our model with shallow fusion showed the best performance of all the models in our experiments.