{'fraction_non_alphanumeric': 0.04394868934746236, 'fraction_numerical': 0.016564417177914112, 'mean_word_length': 4.672571970895286, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Minimum Latency Training of Sequence Transducers for Streaming End-to-End Speech Recognition

Sequence transducers, such as the RNN-T and the Conformer-T, are one of the most promising models of end-to-end speech recognition, especially in streaming scenarios where both latency and accuracy are important. Although various methods, such as alignment-restricted training and FastEmit, have been studied to reduce the latency, latency reduction is often accompanied with a significant degradation in accuracy. We argue that this suboptimal performance might be caused because none of the prior methods explicitly model and reduce the latency. In this paper, we propose a new training method to explicitly model and reduce the latency of sequence transducer models. First, we define the expected latency at each diagonal line on the lattice, and show that its gradient can be computed efficiently within the forward-backward algorithm. Then we augment the transducer loss with this expected latency, so that an optimal trade-off between latency and accuracy is achieved. Experimental results on the WSJ dataset show that the proposed minimum latency training reduces the latency of causal Conformer-T from 220 ms to 27 ms within a WER degradation of 0.7%, and outperforms conventional alignment-restricted training (110 ms) and FastEmit (67 ms) methods.


Introduction
End-to-end speech recognition is becoming more and more popular in both academia and industry due to the simplicity of the approach as well as the competitive performance relative to conventional hybrid models [1,2,3,4,5,6,7,8,9]. Sequence transducer models, such as the recurrent neural network transducer (RNN-T) [2,3], the Transformer transducer [10,11], and the Conformer transducer (Conformer-T) [12,13], are one of the most promising end-to-end models, especially in streaming scenarios because of its inherently streaming nature. In streaming speech recognition, latency is one of the primary performance metrics along with recognition accuracy, because a lower latency leads to a quick response of voice-enabled applications, and improves the user experience [14]. However, streaming transducer models tend to delay label emission so as to see more future context to predict labels more accurately, which leads to a large latency and a deteriorated user experience. Hence, much effort has been devoted to reduce the latency of sequence transducer models.
Alignment restricted training [15,16] is one of the most popular methods for reducing latency of sequence transducers. During the training of sequence transducers, it restricts the alignment between input and output sequences within a certain range from a reference alignment, i.e. those alignments that are too late (or too early) from the reference are pruned away. As a result, late emission of tokens is prohibited and latency reduc-tion is achieved. FastEmit [17] is another well-known method of latency reduction for transducer models. It boosts the gradient of the label emission probability by a specified factor during training, so that the model is encouraged to emit labels earlier.
It has been empirically shown that those methods considerably reduce the latency of sequence transducer models.
However, these conventional methods often have a significant degradation in recognition accuracy as a side effect of latency reduction [15,16,18] 1 . We argue that one of the reasons for the suboptimal latency-accuracy trade-off is that none of the prior methods directly model and reduce the latency of transducer models. For instance, alignment restricted training prohibits those alignments that are too late, but it does not compute the gradient of the latency to explicitly reduce the latency in coordination with the transducer loss.
In this paper, we propose a new training method to explicitly model and reduce the expected latency of sequence transducer models. First, we define the expected latency on each diagonal line on the lattice, and show that the expected latency and its gradient can be calculated efficiently within the forwardbackward algorithm. Then we propose to augment the transducer loss with this expected latency, so that the transducer loss and the expected latency are reduced in coordination with each other, and an optimal latency-accuracy trade-off is achieved. Experimental results on the Wall Street Journal (WSJ) dataset show that the proposed minimum latency training achieves a better latency-accuracy trade-off compared with alignment restricted training and FastEmit.

Related Work
Along with alignment-restricted training and FastEmit, self alignment [18] is another method of latency reduction for sequence transducers; self alignment is similar to our method in its spirit to adaptively modify gradients depending on locations, rather than uniformly boosting gradients like FastEmit, but different from ours in that it does not explicitly model latency and its gradient. It is worthwhile to note that blockprocessing encoders [20,13] and joint endpointing [21,22] have been also studied to achieve fast and accurate transducer models. There is another line of research on streaming end-toend speech recognition using attention-based encoder-decoder (AED) models. Although AED models [4,5,6] are inherently non-streaming because of the attention mechanism, it can be extended to streaming models [23,24,25,26]. Among the streaming AED models and its training methods, minimum latency training (MinLT) [26] is similar to our method in that it explicitly models the latency and uses its gradient. However, its formulation is completely different from ours because of the difference in architectures (AED vs. sequence transducer).

Sequence Transducers [2]
The sequence transducer is a neural network model to map an input sequence, X {x1, . . . , xT }, to an output sequence, Y {y1, . . . , yU }, where xt ∈ R d i is the di-dimensional input vector (e.g. filter-bank vector) at time t, yu ∈ V is the output token (e.g. character) at position u. T is the input sequence length, U is the output sequence length, and V represents the vocabulary (i.e. set of tokens). LetV be the extended vocabulary defined as V ∪ {φ}, where φ is a special token to represent a blank. Let alignment a ∈V * 2 represents an alignment between input and output sequences, for instance a = {φ, y1, φ, φ, y2, y3, φ}. The sequence transducer defines a conditional distribution of the alignment P (a|X), which is summed over all possible alignments to define where B :V * → V * is a function to remove the blank tokens from the alignment. The sequence transducer consists of an encoder to generate de-dimensional encoder representation ft ∈ R de from x1:t, a predictor network to generate dpdimensional predictor representation gu ∈ R dp from y0:u, and a joint network to define a conditional distribution P (k|t, u) of token k ∈V given ft and gu. The conditional distribution of an alignment P (a|X) is calculated by the product of probabilities P (k|t, u) accumulated over the path defined by the alignment a. The loss function of the sequence transducer model is defined as the negative log probability as Its gradients with respect to the label emission probability Py u+1 P (yu+1|t, u) and the blank emission probability P φ P (φ|t, u) for ∀t ∈ {1, . . . , T }, ∀u ∈ {0, . . . , U } can be calculated as, where the forward variable α(t, u) and the backward variable β(t, u) can be calculated efficiently by the forward and backward recursions as Note that the initial conditions are set as α(1, 0) = 1 and β(T + 1, U ) = 1. This procedure is also known as the forwardbackward algorithm.

Alignment restricted training [15]
Given a reference alignment, typically generated by forcedalignment using an HMM-hybrid model, alignment restricted training prohibits those alignments that are too late (or too early) compared with the reference, with the expectation that the model learns to emit labels not too late than the reference timing. Specifically, the forward recursion in (5) is modified as where m·(t, u) ∈ {0, 1} is a mask variable defined by the indicator function 1[·], reference emission timing tu, left buffer bl, and right buffer br as The backward recursion in (6) is also modified in the same manner.

FastEmit [17]
FastEmit boosts the gradient of the label emission probability by a specified factor, so that those alignments that emit labels earlier are encouraged. Specifically, the gradient is increased by a factor of (1 + λFE) as where λFE ∈ R>0 is the hyper-parameter to control the strength of the regularization. FastEmit does not require reference alignments, which makes the training pipeline much simpler. However, it cannot tell how much delay exists at each location (t, u), and uniformly boosts the gradients at all locations regardless of the delays. This blind approach may make it difficult to achieve an optimal latency-accuracy trade-off.

Expected latency
Let d(t, u) ∈ R ≥0 be the delay at position (t, u). Arbitrary values can be set as delays, but in this work we define them as, Here τ (n) ref ∈ {1, . . . , T } is the reference (ground-truth) time defined for ∀n ∈ {1, . . . , T +U +1}, and represents the reference alignment timing on the line (t, u) : t + u = n. It can be generated for instance by forced-alignment using an HMM-hybrid model. Figure 1 depicts an example of a reference alignment and delays on the lattice.
Letdn be an expected delay on the line (t, u) : t + u = n, ∀n ∈ {1, . . . , T + U + 1} defined as = (t,u):t+u=n Here the posterior probability of being at position (t, u) on line (t, u) : t+u = n, denoted by P (t, u|n, X) 3 , can be calculated using the forward and backward variables as described in (15).  (15). The gradients of the expected delay with respect to the label and blank emission probabilities are calculated by (16) and (17).
The gradients of the expected latencydt+u+1 in (15) with respect to the label and blank emission probabilities, Py u+1 P (yu+1|t, u) and P φ P (φ|t, u), can be calculated as 4 An intuitive interpretation of the gradients in (16) and (17) is as follows. When the expected delay, represented by the red dot in Figure 1, is located on the lower-right side of (t, u + 1), represented by the yellow dot,dt+u+1 gets larger than d(t, u+1) and the gradient in (16) becomes negative; hence, label emission probability Py u+1 should be increased to reduce the expected delay, i.e. to move the red dot to the upper-left direction. On the other hand, if the red dot is located on the upper-left side, the label emission probability should be decreased. The same argument can be made for the blank emission probability as well. So the gradients can be used to push the red dot towards the upper-left direction, i.e. to reduce the expected latency of the sequence transducer models.

Loss function
The sequence transducer model tends to delay the alignment so that the model can access more future context to improve the loss function in (2). In this work, we propose to augment the loss function with the expected delay introduced in Section 5.1 so that the model learns to predict labels accurately and fast. Specifically, the total loss is defined as the weighted sum of the transducer loss Ltrans in (2) and the expected latencydt+u+1 in (15) as LMLT Ltrans + λMLTdt+u+1, where λMLT ∈ R>0 is a hyper-parameter to control the strength of the regularization. By using (3), (4), (16), and (17), its gradients with respect to the label and blank emission probabilities 4 Derivation of the gradients is briefly explained in Appendix.
can be calculated as It can be seen from (19) and (20) that the gradient is discounted if the latency d is larger than the expected latencyd, whereas the gradient is boosted if the latency is smaller than the expected latency, so that the alignments with lower latency are encouraged. It can also be seen that the blank gradient is more strongly discounted than the label gradient, since delay d(t + 1, u) is larger than d(t, u+1) (typically by one), which means that label emission is encouraged in a similar manner as FastEmit. Note that FastEmit uniformly boosts the label gradient at all (t, u) by a constant factor, while our proposed method changes the boosting (or discounting) factor depending on (t, u) by considering the delays.

Experiments
Experiments were conducted on the Wall Street Journal dataset to compare minimum latency training with two conventional methods, namely alignment-restricted training and FastEmit, with respect to their accuracy and latency.

Experimental settings
The Wall Street Journal dataset [27] consisting of training set si284 (37,416 utterances), development set dev93 (503 utterances) and evaluation set eval92 (333 utterances), which amount to 81 hours in total, was used to train and test the models. Word Error Rate (WER) and Partial Recognition latency were used as the metrics of accuracy and latency, respectively. Specifically, Partial Recognition latency [17], which represents the difference between the time when the user stops speaking and the time when the last token is emitted, at 50-th (PR50) and 90-th (PR90) percentiles of all utterances in the test set were used as the metrics of the latency. Note that the times were measured in frame index t ∈ {1, . . . , T }, but reported in milliseconds by multiplying the frame period for better readability. Causal Conformer transducer (Conformed-T) models consisting of a 12-layer/512-dim encoder, a 1-layer/512-dim LSTM-based prediction network, and a 512-dim joint network were used in this experiment. The encoder used Conformer layers with causal convolutions of kernel size 7, and 4-head self-attention modules that only attends to the past and present frames 5 . Filter-bank features of 80-dim were extracted at a 10ms frame shift, and fed into a 4-layer VGG-style input layers prepended to the encoder, where the features were downsampled by a factor of six to reach a frame period of 60 ms. SpecAugment was used to regularize the training. The model was configured to predict 52 tokens including the characters and the blank. The Adam optimizer was used in combination with the noam learning-rate schedule, where 10,000 warmup steps was used. Inference was conducted using a standard beam search algorithm [2] with a beam size of 10. No language model was used in this experiment. It is noted that the VGG-style input layers have a look-ahead of 120 ms in total, but this delay is not included in the reported PR50/PR90 numbers. The causal Conformer-T models were trained with four different methods: the standard RNN-T training without any latency reduction technique (baseline) [2], alignment-restricted training [15], FastEmit [17], and the proposed minimum latency training. The reference alignments used for alignmentrestricted training and minimum-latency training were generated by an HMM-hybrid model trained with Kaldi [28]; it is noted that only word-level alignments were available, so the reference time of each character was set equal to the time of the word it belongs to. Regarding the alignment-restricted training, left buffer bl was set to 20, and right buffer br was swept to draw a latency-WER tradeoff curve. As for FastEmit and minimum latency training, regularization strengths λFE and λMLT were swept, respectively. Other hyper-parameters (e.g. learning-rate schedule and kernel size) were tuned on dev93 for the baseline, and used by the other three methods without further tuning. Three independent training runs were conducted with different random seeds, and the average of the three trials are reported for each condition. Experiments were conducted on ESPnet [29,30,31]. Figure 2 shows the results on WSJ. The baseline without using any latency reduction technique had a latency of 220 ms and a WER of 12.1%. Although the model is causal and does not see any future context, the WER was close to that of the Transformer model (11.89%) in [32], so the baseline should be reasonably strong. Alignment restricted training successfully reduced the latency as br was gradually reduced from 20 to 1, but the WER degraded significantly as a side effect. FastEmit had a better latency-WER trade-off than alignment-restricted training, but still had a noticeable loss in WER especially when λFE was large. Minimum-latency training had the best latency-WER trade-off among the three methods, reducing the latency by more than 190 ms while holding the degradation in WER within 0.7%. Table 1 presents the results on WSJ with representative hyper-parameter settings to achieve a similar WER on dev93, namely br = 9, λFE = 0.015, and λMLT = 0.03. As was already seen in Figure 2, minimum latency training had the best latency-WER trade-off among the three methods in terms of PR90. However, the gap in PR50 between FastEmit and minimum latency training was relatively small compared with PR90. It is conjectured that the proposed method works to reduce the expected delay until it reaches zero, but does not push it any further by its design (gradients in (16) and (17) approach zero), hence latency metrics do not take large negative numbers.

Conclusions
In this paper, we proposed minimum latency training of sequence transducer models for fast and accurate end-to-end speech recognition. We have defined the expected latency on each diagonal line on the lattice, and shown that its gradients can be computed efficiently within the forward-backward algorithm. In experiments on the WSJ dataset, the proposed minimum latency training consistently had a better latencyaccuracy trade-off over a wide range of hyper-parameters compared with alignment-restricted training and FastEmit. With a representative hyper-parameter, the proposed method reduced the latency of a causal Conformer-T model from 220 ms to 27 ms within a WER degradation of 0.7%, and outperformed alignment-restricted training (110 ms) and FastEmit (67 ms).