{'fraction_non_alphanumeric': 0.03742860614898529, 'fraction_numerical': 0.013306598614655487, 'mean_word_length': 4.871922939707456, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Multi-encoder multi-resolution framework for end-to-end speech recognition

Attention-based methods and Connectionist Temporal Classification (CTC) network have been promising research directions for end-to-end Automatic Speech Recognition (ASR). The joint CTC/Attention model has achieved great success by utilizing both architectures during multi-task training and joint decoding. In this work, we present a novel Multi-Encoder Multi-Resolution (MEMR) framework based on the joint CTC/Attention model. Two heterogeneous encoders with different architectures, temporal resolutions and separate CTC networks work in parallel to extract complimentary acoustic information. A hierarchical attention mechanism is then used to combine the encoder-level information. To demonstrate the effectiveness of the proposed model, experiments are conducted on Wall Street Journal (WSJ) and CHiME-4, resulting in relative Word Error Rate (WER) reduction of 18.0-32.1%. Moreover, the proposed MEMR model achieves 3.6% WER in the WSJ eval92 test set, which is the best WER reported for an end-to-end system on this benchmark.


INTRODUCTION
Recent advancements in deep neural networks enabled several practical applications of automatic speech recognition (ASR) technology. The main paradigm for an ASR system is the so-called hybrid approach, which involves training a DNN to predict context dependent phoneme states (or senones) from the acoustic features. During inference the predicted senone distributions are provided as inputs to decoder, which combines with lexicon and language model to estimate the word sequence. Despite the impressive accuracy of the hybrid system, it requires hand-crafted pronunciation dictionary based on linguistic assumptions, extra training steps to derive context-dependent phonetic models, and text preprocessing such as tokenization for languages without explicit word boundaries. Consequently, it is quite difficult for non-experts to develop ASR systems for new applications, especially for new languages.
End-to-End speech recognition approaches are designed to directly output word or character sequences from the input audio signal. This model subsumes several disjoint components in the hybrid ASR model (acoustic model, pronunciation model, language model) into a single neural network. As a result, all the components of an end-to-end model can be trained jointly to optimize a single objective. Two dominant end-to-end architectures for ASR are Connectionist Temporal Classification (CTC) [1,2,3] and attention-based encoder decoder [4,5] models. While CTC efficiently addresses sequential problem (speech vectors to word sequence mapping) by avoiding the alignment pre-construction step using dynamic programming, it assumes conditional independence of label sequence given the input. Attention model does not assume conditional independence of label sequence resulting in a more flexible model. However, attention-based methods encounter difficulty in satisfying the speech-label monotonic property. To alleviate this issues, a joint CTC/Attention framework was proposed in [6,7,8]. The joint model was shown to provide the state-of-the-art end-to-end results in several benchmark datasets [8].
In end-to-end ASR approaches, the encoder acts as an acoustic model providing higher-level features for decoding. Bi-directional Long Short-Term Memory (BLSTM) has been widely used due to its ability to model temporal sequences and their long-term dependencies as the encoder architecture; Deep convolutional Neural Network (CNN) was introduced to model spectral local correlations and reduce spectral variations in end-to-end framework [7,9]. The encoder architecture combining CNN with recurrent layers, was suggested to address the limitation of LSTM. While temporal subsampling in RNN and max-pooling in CNN aim to reduce the computational complexity and enhance the robustness, it is likely that subsampling technique results in loss of temporal resolution.
In this work, we propose a Multi-Encoder Multi-Resolution (MEMR) model within the joint CTC/Attention framework. This is strongly motivated by the success of multi-stream paradigm in Hybrid ASR [10,11,12] mimicking human speech processing cognitive system. Two parallel encoders with heterogeneous structures, RNN-based and CNN-RNN-based, are mutually complementary in characterizing the speech signal.
Several studies have shown that attention-based model benefits from having multiple attention mechanisms [13,14,15,16,17,18]. Inspired by the advances in Hierarchical Attention Network (HAN) in document classification [16], multi-modal video description [17] and machine translation [18], we adapt HAN into our MEMR model. The encoder that carries the most discriminate information for the prediction can dynamically receive a stronger weight. Each encoder is associated with a CTC network to guide the frame-wise alignment process for individual encoder. This paper is organized as follows: section 2 explains the joint CTC/Attention model. The description of the proposed MEMR framework is in section 3. Experiments with results and several analyses are presented in section 4. Finally, in section 5 the conclusion is derived.

JOINT CTC/ATTENTION MECHANISM
In this section, we review the joint CTC/attention architecture, which takes advantage of both CTC and attention-based end-to-end ASR approaches during training and decoding.

Connectionist Temporal Classification (CTC)
Following Bayes decision theory, CTC enforces a monotonic mapping from a T -length speech feature sequence, X = {xt ∈ R D |t = 1, 2, ..., T }, to an L-length letter sequence, C = {c l ∈ U|l = 1, 2, ..., L}. Here xt is a D-dimensional acoustic vector at frame t, and c l is at position l a letter from U, a set of distinct letters.
The CTC network introduces a many-to-one function from frame-wise latent variable sequences, Z = {zt ∈ U blank|t = 1, 2, ..., T }, to letter predictions of shorter lengths. Note that the additional "blank" symbol is used to handle the merging of repeating letters. With several conditional independence assumptions, the posterior distribution, p(C|X), is represented as follows: where p(zt|X) is a frame-wise posterior distribution, and we also define the CTC objective function pctc(C|X). CTC preserves the benefits that it avoids the HMM/GMM construction step and preparation of pronunciation dictionary.

Attention-based Encoder-Decoder
As one of the most commonly used sequence modeling techniques, the attention-based framework selectively encodes an audio sequence of variable length into a fixed dimension vector representation, which is then consumed by the decoder to produce a distribution over the outputs. We can directly estimate the posterior distribution p(C|X) using the chain rule: where patt(C|X) is defined as the attention-based objective function. Typically, a BLSTM-based encoder transforms the speech vectors X into frame-wise hidden vector ht If the encoder subsamples the input by a factor s, there will be T /s time steps in H = {h1, ..., h T /s }. The letter-wise context vector r l is formed as a weighted summation of frame-wise hidden vectors H using contentbased attention mechanism. In comparison to CTC, not requiring conditional independence assumptions is one of the advantages of using the attention-based model. However, the attention is too flexible to satisfy monotonic alignment constraint in speech recognition tasks.

Joint CTC/Attention
The joint CTC/Attention architecture benefits from both CTC and attention-based models since the attention-based encoder-decoder is trained together with CTC within the Multi-Task Learning (MTL) framework. The encoder is shared across CTC and attention-based encoders. And the objective function to be maximized is a logarithmic linear combination of the CTC and attention objectives, i.e., pctc(C|X) and p † att (C|X): where λ is a tunable scalar satisfying 0 ≤ λ ≤ 1. p † att (C|X) is an approximated letter-wise objective where the probability of a prediction is conditioned on previous true labels.
During inference, the joint CTC/Attention model performs a label-synchronous beam search. The most probable letter sequencê C given the speech input X is computed according tô where external RNN-LM probability log p lm (C) is added with a scaling factor γ.

PROPOSED MEMR FRAMWORK
The overall architecture is shown in Fig. 1. Two types of encoders with different temporal resolutions are presented in parallel to capture acoustic information in various ways, followed by an attention fusion mechanism together with per-encoder CTC. An external RNN-LM is also involved during the inference step. We will describe the details of each component in the following sections.

Multi-Encoder with Multi-Resolution
We propose a Multi-Encoder Multi-Resolution (MEMR) architecture that has two encoders, RNN-based and CNN-RNN-based. Both encoders take the same input features in parallel operating on different temporal resolutions, aiming to capture complimentary information in the speech.
The RNN-based encoder is designed to model temporal sequences with their long-range dependencies. In MEMR, the BLSTM encoder has only BLSTM layers that extract the frame-wise hidden vector h 1 t without subsampling in any layer: where the BLSTM decoder is labeled as index 1.
The combination of CNN and RNN allows the convolutional feature extractor applied on the input to reveal local correlations in both time and frequency dimensions. The RNN block on top of CNN makes it easier to learn temporal structure from the CNN output, to avoid modeling direct speech features with more underlying variations. The pooling layer is essential in CNN to reduce the spatial size of the representation to control over-fitting. In MEMR, we use the initial layers of the VGG net architecture [19] followed by BLSTM layers as VGGBLSTM decoder labeled as index 2: The configuration of convolutional layers in VGGBLSTM encoder is the same as in [7].

Hierarchical Attention
Since the encoders in MEMR describe the speech signal differently by catching acoustic knowledge in their own ways, encoder-level fusion is suitable to boost the network's ability to retrieve the relevant information. We adapt Hierarchical Attention Network (HAN) in [16] for information fusion. The decoder with HAN is trained to selectively attend to appropriate encoder, based on the context of each prediction in the sentence as well as the higher-level acoustic features from both encoders, to achieve a better prediction. The letter-wise context vectors, r 1 l and r 2 l , from individual encoders are computed as follows: where the attention weights are obtained using a scontent-based attention mechanism. Note that since Encoder2 performs downsampling by 4, the summation is till T /4 in Eq. (7). The fusion context vector r l is obtained as a convex combination of r 1 l and r 2 l as illustrated in the following: The stream-level attention weights β l1 and β l2 are estimated according to the previous decoder state, q l−1 , and context vectors, r 1 l and r 2 l , from individual encoders as described in Eq. (9). The fusion context vector is then fed into the decoder to predict the next letter.

Per-encoder CTC
In the CTC/Attention model with a single encoder, the CTC objective serves as an auxiliary task to speed up the procedure of realizing monotonic alignment and providing a sequence-level objective. In the MEMR framework, we introduce per-encoder CTC where a separate CTC mechanism is active for each encoder stream during training and decoding. Sharing one set of CTC among encoders is a soft constraint that limits the potential of diverse encoders to reveal complimentary information. In the case that both encoders are with different temporal resolutions and network architectures, perencoder CTC can further align speech with labels in a monotonic order and customize the sequence modeling of individual streams.

Experimental Setup
We demonstrate our proposed MEMR model using two datasets: WSJ1 [20] (81 hours) and CHiME-4 [21] (18 hours). In WSJ1, we used the standard configuration: "si284" for training, "dev93" for validation, and "eval92" for test. The CHiME-4 dataset is a noisy speech corpus recorded or simulated using a tablet equipped with 6 microphones in four noisy environments: a cafe, a street junction, public transport, and a pedestrian area. For training, we used both "tr05 real" and "tr05 simu" with additional WSJ1 corpora to support end-to-end training. "dt05 multi isolated 1ch track" is used for validation. We evaluated the real recordings with 1, 2, 6-channel in the evaluation set. The BEAMFORMIT method was applied to multichannel evaluation. In all experiments, 80-dimensional mel-scale filterbank coefficients with additional 3-dimensional pitch features served as the input features. The Encoder1 contains four BLSTM layers, in which each layer has 320 cells in both directions followed by a 320-unit linear projection layer. The Encoder2 combines the convolution layers with RNN-based network that has the same architecture as Encoder1. A content-based attention mechanism with 320 attention units is used in encoder-level and frame-level attention mechanisms. The decoder is a one-layer unidirectional LSTM with 300 cells. We use 50 distinct labels including 26 English letters and other special tokens, i.e., punctuations and sos/eos.
We incorporated the look-ahead word-level RNN-LM [25] of 1-layer LSTM with 1000 cells and 65K vocabulary, that is, 65Kdimensional output in Softmax layer. In addition to the original speech transcription, the WSJ text data with 37M words from 1.6M sentences was supplied as training data. RNN-LM was trained separately using Stochastic Gradient Descent (SGD) with learning rate = 0.5 for 60 epochs.
The MEMR model is implemented using Pytorch backend on ESPnet. Training procedure is operated using the AdaDelta algorithm with gradient clipping on single GPUs, "GTX 1080ti". The mini-batch size is set to be 15. We also apply a unigram label smoothing technique to avoid over-confidence predictions. The beam width is set to 30 for WSJ1 and 20 for CHiME-4 in decoding. For model jointly trained with CTC and attention objectives, λ = 0.2 is used for training, and λ = 0.3 for decoding. RNN-LM scaling factor γ is 1.0 for all experiments with the exception of using γ = 0.1 in decoding attention-only models.

Results
The overall experimental results on WSJ1 and CHiME-4 are shown in Table 1. Compared to joint CTC/Attetion single-encoder models, the proposed MEMR model with per-encoder CTC and HAN achieves relative improvements of 9.6% (28.4% → 26.4%) in For fair comparison, we increase the number of BLSTM layers from 4 to 8 in Encoder2 to train a single-encoder model. In Table  2, the MEMR system outperforms the single-encoder model by a significant margin with similar amount of parameters, 21.9M v.s. 21.3M. In CHiME-4, we evaluate the model using real test data from 1, 2, 6-channel resulting in an average of 19% relative improvement from all three setups. In WSJ1, we reach 3.6% WER in eval92 in our MEMR framework with relatively 32.1% improvement.
The results in Table 3 shows the contribution of multiple resolution. The WER goes up when increasing subsampling factor s1 closer to s2 = 4 in both datasets. In other words, the fusion works better when two encoders are more heterogeneous which supports our hypothesis. As shown in Table 4, We analyze the average streamlevel attention weight for Encoder2 when we gradually decrease the number of LSTM layers while keeping Encoder1 with the original  configuration. It aims to show that HAN is able to attend to the appropriate encoder seeking for the right knowledge. As suggested in the table, more attention goes to Encoder1 from Encoder2 as we intentionally make Encoder2 weaker.

CONCLUSION
In this work, we present our MEMR framework to build an endto-end ASR system. Higher-level frame-wise acoustic features are carried out from RNN-based and CNN-RNN-based encoders with subsampling only in convolutional layers. Stream fusion selectively attends to each encoder via a content-based attention. We also investigated that assigning a CTC network to individual encoder further enhance the heterogeneous configuration of encoders. The MEMR model outperforms various single-encoder models, reaching the state-of-the-art performance on WSJ among end-to-end systems.