{'fraction_non_alphanumeric': 0.03400593471810089, 'fraction_numerical': 0.013353115727002967, 'mean_word_length': 4.89401888772298, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Weak-Attention Suppression for Transformer Based Speech Recognition

Transformers, originally proposed for natural language processing (NLP) tasks, have recently achieved great success in automatic speech recognition (ASR). However, adjacent acoustic units (i.e., frames) are highly correlated, and long-distance dependencies between them are weak, unlike text units. It suggests that ASR will likely benefit from sparse and localized attention. In this paper, we propose Weak-Attention Suppression (WAS), a method that dynamically induces sparsity in attention probabilities. We demonstrate that WAS leads to consistent Word Error Rate (WER) improvement over strong transformer baselines. On the widely used LibriSpeech benchmark, our proposed method reduced WER by 10%$ on test-clean and 5% on test-other for streamable transformers, resulting in a new state-of-the-art among streaming models. Further analysis shows that WAS learns to suppress attention of non-critical and redundant continuous acoustic frames, and is more likely to suppress past frames rather than future ones. It indicates the importance of lookahead in attention-based ASR models.


Introduction
In recent years, models based on transformers [1] and their variants have achieved state-of-the-art results in many natural language processing (NLP) tasks, such as language modeling, machine translation, and natural language understanding [2,3,4]. The transformer relies on the multi-head self-attention method, eschewing the recurrent connection in recurrent neural networks. The attention method connects arbitrary positions in the whole sequence directly, allowing the model to capture long-range dependencies regardless of distance. Each transformer layer performs computations for the whole sequence in parallel, thus improving efficiency.
While similar in some aspects, ASR tasks are very different from many NLP tasks. ASR systems extract the input features from continuous signals rather than discrete text units. Many acoustic units (i.e., frames) are typically needed to convey a single text unit's semantic meaning. Many of these acoustic units are not critical in long-range dependencies, for example, silence. Based on these observations, we propose the Weak-Attention Suppression (WAS) method to improve transformerbased models for ASR. The method aims to induce sparsity in the attention probability distribution by dynamically determining a threshold for each time frame. All attention probabilities smaller than this threshold are set to zero, and the remaining probabilities are re-normalized to sum to one.
To verify the performance of the proposed method, we carry out experiments on the widely used LibriSpeech corpus [14]. WAS consistently improves Word Error Rate (WER) of transformer-based hybrid acoustic models across different model sizes (12-layer 40M parameters vs. 24-layer 81M parameters) and variants (non-streamable full context [12] vs. streamable limited context [15]). The relative WER improvement ranges from 6% to 10% on test-clean and 2% to 6% on test-other, with larger impact on streamable transformers. Our results with WAS and streamable transformers established a new state-of-the-art on Librispeech among streaming models to the best of our knowledge.
A more in-depth analysis of individual utterances and the whole dev-clean split in LibriSpeech data shows that WAS suppresses attention from non-critical acoustic units, such as silence. For adjacent acoustic units, attention from redundant units is suppressed, especially for the lower layers. The analysis also demonstrates that past acoustic units are more likely to be suppressed than future ones. In other words, lookahead is especially crucial for transformer-based acoustic models.

Related Work
The proposed WAS method aims to improve transformer-based acoustic models in hybrid speech recognition for both streaming [11,15], and non-streaming [12] applications.
For the non-streaming case, we use the same model architecture proposed in [12], where the whole model architecture consists of three parts, convolution layers, transformer layers, and auxiliary intermediate layer losses. The convolution layers play a similar role as another positional encoding method [16]. A stack of transformers with both multi-head self-attention and feed-forward network (FFN) is used on top of the convolution layers. To effectively train the deep transformers, the final loss function is interpolated with auxiliary intermediate layer losses [17] that pass the gradients to the intermediate layers in the deep structure.
To extend transformer-based acoustic models for streaming applications, the work [15] applies block processing to segment the whole utterance into multiple chunks with lookahead context. To carry over information across chunks, they modify the self-attention with the augmented memory bank. Each slot in the augmented memory bank stores the embedding vectors for previous chunks.
Sparse attention in the transformer has been explored in [18] to reduce model inference complexity. Unlike this work, our method aims to improve ASR accuracy by inducing sparsity in both training and inference.
Different from the works in [19,20] that achieve sparse attention by replacing the softmax function with α-entmax or sparsemax, our method keeps the softmax function and sup-presses the attention based on the probability distribution. Comparing these different methods for suppressing attention will be an interesting topic for future research.

Transformer Based Acoustic Model With Weak-Attention Suppression
Before explaining the details of the WAS, the following subsection gives a short description of multi-head self-attention.

Multi-Head Self Attention
Given the input embedding sequence X ∈ R L×d i with sequence length L, the projection matrices Wq, W k and Wv transform the X into query, key and value space, respectively.
The attention probabilities are computed in dot-product way as follows: where A is a L × L matrix. Each element αi,j represents the attention probability of the query at i position with the key at j position. Given A, the output embedding sequence of self-attention is obtained via: Rather than performing a single self-attention function with queries, keys and values, it was found beneficial to run selfattention h times in parallel with different projection matrices Wq, W k and Wv. The outputs from each self-attention are concatenated and linearly projected, resulting to final output, i.e., where Wo ∈ R d i ×hdv , and Z i is the output from the i-th selfattention.

Weak-Attention Suppression
In speech recognition tasks, many acoustic units, e.g., silence, may not play a critical role in long-distance dependencies. For continuous acoustic units that share similarities, some of them may be redundant for long-distance dependencies. Comparing with many NLP tasks, the number of acoustic units in audio utterances are much more significant than the number of text units in sentences. Due to these factors, the sparse attention is more desirable for speech recognition. However, in self-attention, the softmax function is used to get the attention probabilities. One limitation of softmax function is that it always generates dense attention, i.e., softmax(X)i = 0; all the elements of the softmax function is over zero.
The WAS sets the attention probabilities smaller than a threshold to zero and normalizes the rest attention probabilities. The threshold is determined based on the following schema.
where mi and δi are the mean and standard deviation of the attention probability for i-th position in query, respectively. So the threshold θi can be represented as where L is the length of key in self-attention. The attention probability αi,j less than θi, is set to zero. The rest non-zero attention probabilities are re-normalized.
In practical implementation, we realize the renormalization in two steps. Firstly, using softmax function on attention weights, we get the attention probabilities. Based on formula (6), we replace the attention weight to negative infinity when its attention probability is lower than the threshold. Applying the softmax function on the processed attention weight generates the re-normalized attention probabilities.

Data
To verify the performance of the proposed method, we conduct experiments on the LibriSpeech corpus [14]. LibriSpeech is an open-source speech corpus that contains 1000 hours of speech derived from audiobooks in the LibriVox project. The audio data 1 , language model data, and pre-trained language models 2 are available for downloading. The development data and evaluation data in LibriSpeech are split into simple "clean" subsets and more difficult "other" subsets. We use the official 4-gram language model for decoding in all experiments.

Setup
In all experiments, we use context and positional dependent graphemes (i.e., chenones) as output units [21]. We bootstrap the HMM-GMM system following the standard Kaldi [22] Lib-riSpeech recipe. For feature extraction, we use 80-dimensional logMel filter bank features at a 10ms frame rate. To increase the training robustness, both speed perturbation [23] and SpecAugment [24] without time warping are used.
We apply the same model architecture as [12,15], which has two VGG blocks [25] followed by a stack of transformer layers. Each VGG block has two consecutive 3-by-3 convolution layers followed by a Relu activation function and a maxpooling layer. While the first VGG block uses 32 channels, the second VGG block uses 64 channels. Both VGG blocks use a 2-by-2 Max-pooling. The first VGG block uses stride 2; the second one uses stride 1. Overall, two VGG blocks have stride 2. From an input sequence of 80-dim feature vector at a 10ms rate, the VGG blocks generate a 2560-dim feature vector sequence at a 20ms rate.
Each transformer layer applies 8 heads of self-attention. The dimensionality of input and output for each transformer layer is 512, and the inner-layer of FFN has dimensionality 2048. We experiment with different model sizes (12 transformer layers, and 24 transformer layers architectures) to verify the improvement from the WAS. To overcome the training divergence issue for deep transformer models, e.g., 24 transformer layers, we apply an auxiliary incremental loss [17]. A linear transformation, together with a Relu function, projects the outputs from the 6/12/18-th transformer layers to the final output space. The auxiliary incremental loss takes the projected outputs to compute the auxiliary CE losses interpolated with the original CE loss with a 0.3 weight.
In all experiments, we use the adam optimizer [26] with a tri-stages learning-rate strategy, i.e., warming-up stage, holding stage, and decaying stage. 8K updates in the warming-up stage increase the learning rate from 1e-5 to 1e-3 for non-streaming transformer-based models and 3e-4 for streaming transformerbased models, respectively. The holding stage uses 100K updates. The decaying stage exponentially decreases the learning rate to 1e-5.
To efficiently use GPU resources, the batch size is dynamically determined. Each batch contains around 10,000 to 20,000 frames, including padding frames. We train all the models using 32 Nvidia V100 GPUs. To train the transformer-based model effectively, we segment the training utterances into less than 10 seconds using forced alignment results from an existing latencycontrolled BLSTM acoustic model. We select the best model based on WER on the dev set and report its result on test data.  Table 2: Word error rate comparison of different γ selection in weak-attention suppression on LibriSpeech data. Table 2 gives the WER results from different γ selection on LibriSpeech data using the small size of the transformer model. γ = 0.5 gives us the best results. We constrain the γ selection from 0 to 1. If the attention probability is close to a normal distribution, we suppress roughly 16% to 50% the attention.

Model
According to our analysis, when γ = 0.5, 36% of attention got suppressed in the first transformer layer.

Attention Suppression Analysis
In this section, we analyze the effect of WAS in one individual utterance and the whole dev-clean split in LibriSpeech using the model 'vggTrf-12-WAS' in Table 1.
In individual utterance analysis, we average the portion of the suppressed attention over the multiple heads and the whole input sequence for each transformer layer. The following formula represents the processing.
where α k i,j represents the attention probability of the i-th position in query with j-th position in key from the k-th head. θ k i is the suppression threshold for i-th position in query in the k-th head. The threshold is determined according to formula (6).
where L is the length of key and H the number heads in multihead self-attention. The function value f (j) indicates the weakness of the attention at the j-th position in the key. Figure 1(b),1(c) and 1(d) draws the function y = s(j) for 1st, 6-th and 12-th transformer layer. The horizontal axis stands for the position in the key. The vertical axis stands for the averaged portion of suppressed attention. Note 'vggTrf-12-WAS' uses VGG blocks with stride 2. For one-second audio using frameshift 10ms, the length of the input sequence to each transformer layer is 500.
By comparing the audio wave 1(a) with 1(b), it is obvious to see there are three peaks (at the beginning, in the middle and at the end) in 1(b) that are the exact position of a long silence in the audio wave. The rest of the small peaks in 1(b) seem to be in the rough location of short silence in the audio wave in 1(a). This phenomenon indicates that using weak-attention suppression in the first transformer layer can suppress the attention to silence.
Zooming into Figure 1(b) and Figure 1(c), there are dramatic ups-and-downs even for the non-silence consecutive positions in the input sequence that are supposed to share acoustic similarities. This phenomenon suggests that for consecutive frames that share acoustic similarities, the weak attention suppress method suppresses the redundant information for attention.
Comparing the Figures 1(b), 1(c) and 1(d) vertically, there are less fluctuations with higher layer. This observation may suggest that the lower transformer's self-attention can capture the small difference for consecutive frames while the higher layer captures the salient invariant features for consecutive frames. Figure 2 gives the weak-attention suppression analysis at different positions from different transformer layers. For a specific position i, we average the portion of the suppressed attention over multiple heads and different utterances in the devclean dataset. We represent the process as follows: where s k,n i,j means whether to suppress the attention between ith position in query with j-th position in key from k-th head in   Table 1. Each sub-figure gives the average portion of attention suppressed at a specific position in a specific layer within the window size 200. We take the average over different heads in multi-head self-attention and different utterances over the dev-clean dataset.
100 left and 100 right contexts. Figure 2 reveals the following three phenomenons. Firstly, the left column in the figure shows that more attention gets suppressed in the lower layer when the attention distance gets further. The deep valley of attention suppression happens in the context window of -5 to 5, i.e., 100 ms left context and 100 ms right context. The right column shows that in the 12-th layer, more attention gets suppressed from the left context than from the right context. One common phenomenon in both columns is that there are roughly 10 times more attention gets suppressed in the 1-st layer than the 12th layer.

Conclusions
In this paper, we proposed a weak-attention suppression (WAS) method to address the sparse attention for speech recognition. The attention probability smaller than a dynamic estimated threshold was zeroed out by WAS. The analysis showed several exciting phenomena of the proposed method. In the lower transformer layer, the attention suppression happens for the silence and redundant acoustic units. Even using the same attention suppression schema, less attention got suppressed in the upper layer than the lower layer. The further the dependency distance was, the more attention got suppressed. In the top transformer layer, the WAS suppressed more attention from the left context than the right context. Experiments on LibriSpeech showed that the WAS improves the transformer-based model with different model sizes in both streaming and non-streaming scenarios. Especially for a streamable transformer-based acoustic model, the proposed WAS got relative word error rate reduction by 10.0% on test-clean and 5.2% on test-other.