{'fraction_non_alphanumeric': 0.03884618563517646, 'fraction_numerical': 0.010414083808579222, 'mean_word_length': 4.758924321751547, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Distilling Knowledge from Ensembles of Acoustic Models for Joint CTC-Attention End-to-End Speech Recognition

Knowledge distillation has been widely used to compress existing deep learning models while preserving the performance on a wide range of applications. In the specific context of Automatic Speech Recognition (ASR), distillation from ensembles of acoustic models has recently shown promising results in increasing recognition performance. In this paper, we propose an extension of multi-teacher distillation methods to joint CTC-attention end-to-end ASR systems. We also introduce three novel distillation strategies. The core intuition behind them is to integrate the error rate metric to the teacher selection rather than solely focusing on the observed losses. In this way, we directly distill and optimize the student toward the relevant metric for speech recognition. We evaluate these strategies under a selection of training procedures on different datasets (TIMIT, Librispeech, Common Voice) and various languages (English, French, Italian). In particular, state-of-the-art error rates are reported on the Common Voice French, Italian and TIMIT datasets.


INTRODUCTION
Knowledge Distillation (KD) [1], also known as teacherstudent training, is commonly used to narrow the gap of performance between a smaller model and a larger one [2,3,4,5,6]. A typical KD training procedure consists of two stages. First, a deep neural network referred as the teacher is trained in line with standard supervised training rules based on numerous samples and their corresponding ground truth labels. Second, a compressed network, the student model, is trained on a selection of original ground truths and soft targets labelled by the teacher. These soft targets are the posterior probabilities obtained from the pre-trained teacher. Knowledge distillation has been shown to be particularly efficient to reduce the student size while matching its performance to that of the teacher. Common applications include Computer Vision (CV) [4,7,8], Natural Language Processing [9,10,11] (NLP) and Automatic Speech Recognition (ASR) [12,13,14,15].
An alternative approach to KD focuses solely on increasing the performances of the student model without considering its complexity. Distillation from ensembles of teachers has been commonly conducted under this approach. This method is referred to as the multi-teacher distillation [16,17].
Modern deep learning based ASR systems have been shown to strongly benefit from multi-teacher distillation strategies [12,17]. Empirically, ensembles of teacher models capture complementary information by making different errors that can be further distill to a student model. A critical aspect of multi-teacher distillation in the context of ASR is to find suitable strategies to maximize the distillation with respect to a specific set of teachers. For instance, [12] proposed to pre-assign weights to teachers to control their impact on the distilled information. Another strategy is to sample the considered teachers randomly [17]. However, both strategies may give higher weighting, and thus higher importance, to teachers that are performing worse than others in the teacher set when applied to specific sentences.
End-to-End (E2E) ASR models are particularly well suited for KD as the whole pipeline is composed of neural networks only [13,15,18]. One set of E2E ASR systems commonly rely either on the Connectionist Temporal Classification (CTC) loss [19], Sequence to Sequence models (Seq2Seq) [20], or a combination of the two [21]. While single teacher distillation to achieve acoustic model compression have been widely investigated on the CTC and Seq2Seq families of models [13,3], works on ensembles of teachers to enhance the performances remain scarce.
The multi-teacher setup holds untapped potential as different E2E ASR systems often lead to different transcriptions given a fixed audio sample, which strongly increases the diversity of the teachable distributions that could be distilled to the student. Therefore, it is of crucial interest to explore the use of diverse sets of E2E teachers to increase both the robustness and the performance of the student acoustic model. Potential use-cases include: Federated Learning (FL) [22,23] with hundreds of acoustic models being trained concurrently, thus needing a proper aggregation or distillation strategy to further reduce the error rate and training time; Productionoriented training pipelines of ASR systems relying on strong hyper-parameters tuning phases with multiple models that could be further used rather than discarded to improve the quality of the final model. In this paper, we first propose and investigate an extension of multi-teacher KD strategies to joint CTC-attention based ASR models. Motivated by error-weighted ensemble methods [24], we introduce three novel Error Rate (ER)-based multi-teacher distillation strategies. Indeed, common distillation strategies only consider the loss as an indicator to assess the teacher quality, while a more relevant scheme for ASR is to optimise our student toward the transcription quality.
First, the "Weighted'' strategy enables the student to directly assign weights to all the teachers in the course of training based on the average observed ER on the processed minibatch. The impact of the teachers is therefore dynamically changed between mini-batches. Then, the "Top-1'' strategy offers the student an option to choose a single teacher with respect to the best ER observed at the sentence level on the processed mini-batch. Finally, The "Top-k'' strategy allows the student to learn from the a set of best teachers that perform equally in terms of error rate on the processed mini-batch as equal ER do not necessarily imply identical transcripts.
In short, our contributions are: a. Introduce multi-teacher distillation for ER reduction on joint CTC-attention based E2E systems (Sec. 2 & Sec. 3); b. Propose three novel distillation strategies focusing on the reduction of the ER (Sec. 3); c. Compare all the models on four different datasets: TIMIT, Librispeech, Common Voice (CV) French and Italian (Sec. 4). The code to replicate our results and to facilitate further research is distributed within the SpeechBrain 1 toolkit [25]. State-of-the-art error rates of 13.11%, 15.57% and 17.04% are reported on TIMIT and CV Italian, French respectively. 1 https://speechbrain.github.io

DISTILLATION FOR JOINT CTC-ATTENTION SPEECH RECOGNITION
Joint CTC-Attention E2E systems [21] combine a sequenceto-sequence (Seq2Seq) attention-based model [20] with the CTC loss [19]. The CTC is applied to facilitate the training of the attention decoder by directing the attention toward the correct alignment. A typical Seq2Seq model includes three modules: an encoder, a decoder and an attention module. The encoder processes an input sequence x = [x 1 , ..., x Tx ] ∈ X with a length T x , where X is the whole set of training data, and creates a hidden latent representation h e = [h e 1 , ..., h e Tx ]. Then the decoder attends h e combined with an attention context vector c t obtained with the attention module to produce the different decoder hidden states h d = [h d 1 , ..., h d Ty ], with T y the length of the target sequence y = [y 1 , ..., y Ty ] ∈ Y , and Y the target utterances dataset composed with a vocabulary V = [v 1 , ...v z ] of z tokens (e.g. characters, tokens, words ...). Note that in a speech recognition scenario, the length of the original signal T x is much longer than the utterance length T y (i.e. many to one).
The standard supervised training procedure of the joint CTC-Attention ASR pipeline is based on two different losses. First, the CTC loss is derived with respect to the prediction obtained from the encoder module of the Seq2Seq model: with S = {X, Y } denoting the training dataset. Second, the attention-based decoder is optimized following the Cross Entropy (CE) loss: Both losses are finally combined and controlled with a fixed hyperparameter α (0 ≤ α ≤ 1) as: Single teacher distillation. Different techniques have been proposed to adapt both L CE and L CT C to single teacher distillation [13,15,18]. Following these findings, we decided to follow frame-level and sequence-level distillations for L CE−KD and L CT C−KD respectively. More precisely, the former loss (i.e. attentional decoder) will be computed by considering the soft label produced for every frame as the true output distribution (i.e. not a one hot encoding) for the teacher, while the latter one receives the entire sequence from the teacher as one hot true labels. In this context, [15] proposed to increase the diversity by generating a list of N-best true sentences with a beam-search. Therefore, L CE−KD is computed as: with, h e tea and h e st the hidden vector representations of the teacher and student encoders respectively, and p tea (v|h d tea ) and p st (v|h d st ) the posterior probabilities estimated by the teacher and student models with respect to the label v. Then, L CT C−KD is defined at the sequence-level instead as: withŷ n the n-th hypothesis from the set of N-best hypothesis obtained via beam-search (i.e. beam width=N) for the teacher. Then, Eq. 3 is extended to knowledge distillation: Finally, an extra-layer of customization is added with a global loss combining KD and supervised training: with β ∈ [0, 1] an hyperparameter controlling the impact of KD during the training (e.g. β = 1 in all our experiments) .

MULTI-TEACHER ERROR RATE DISTILLATION
Different E2E ASR models make different mistakes while transcribing the same audio recording. Therefore, distillation from multiple pre-trained teachers has potential to help the student model to improve considerably. Finding a good teacher weight assignment strategy, however, is not trivial. An existing approach [12,17] is to simply compute an average over the set of teachers: with w m ∈ [0, 1] the pre-assigned weight corresponding to the m-th teacher model and equal to 1/M . M is the total number of teachers composing the ensemble. However, this method gives to a poor teacher the same importance as a good one while a natural solution would be to associate wellperforming teachers with higher weights. We propose to consider the error rate metric as a proxy to determine which teacher loss to consider during distillation. Indeed, cross-entropy and CTC losses are not directly linked to error rates, and there is no evidence that a teacher with the lowest global loss also provides the lowest error rate. The multi-teacher distillation can, therefore, benefit from the introduction of the ER metric to the training procedure. Hence, we introduce three different weighting strategies: "Weighted", "Top-1" and "Top-k".
"Weighted" Strategy: A weight is assigned to all the teachers with respect to their average error rates on the current minibatch, such that the weights dynamically change during training. More precisely, w m from Eq.8 is computed as the softmax distribution from the ER of the current mini-batch: In particular, we propose to extend L CT C−KD to multi teacher distillation by replacing the N -best hypothesis with the number of teachers M . For instance, in [15] a beam of size N is necessary to generate enough diverse sentences from a single teacher. In our approach, we replace the beam-search with the M 1-best sentences generated by M teachers as: with p tea (ŷ m |h e tea ) = w m . This method is extended to L CE−KD in the same manner but at the frame-level. Nevertheless, with this strategy, the worst teacher would still impact negatively the training even though it has a lowest weight, and the variation of ERs in one mini-batch could be large enough to not reflect properly the quality of a teacher.
"Top-1" Strategy: The best teacher (i.e. lowest ER) for each sentence is used during distillation. Similarly to the previous strategy L CE−KD can dynamically be computed following Eq. 4 during training. This approach slightly reduces the computational complexity, but also suffers from a lack of diversity. Indeed, the same teacher will always be picked for a specific sentence from one epoch to an other one. To mitigate this issue, a third strategy is introduced.
"Top-K" Strategy: This method proposes to consider all the teachers that obtain identical error rates at the sentence level as equal candidates for distillation. In particular, identical ER do not necessarily mean that posterior probabilities are also equivalent as different word-level mistakes could be observed. Thus, Eq. 4 integrates the K-best teachers as: Finally, the global losses L KD and L total are computed with the new L CE−KD and L CT C−KD based on Eq. 6 and Eq. 7 respectively.
During all our experiments, the L CT C−KD only benefits from the "Weighted" strategy while the others are investigated for L CE−KD . As a matter of fact, and following [25], the CTC decoder is only used to help the attentional decoder learing a correct alignment at training time, and is completely discarded at testing time, lowering the impact and the interest of various KD strategies for this part.

EXPERIMENTS
The multi-teacher knowledge distillation approach for joint CTC-attention E2E ASR systems (Sec. 4.2) and the proposed distillation strategies are investigated and discussed (Sec

Speech recognition datasets
Four datasets of different sizes, languages and complexities are proposed to validate our multiteacher KD approach.
TIMIT [26]. Consists of the standard 462-speaker training set, a 50-speakers development set and a core test set of 192 sentences for a total of 5 hours of clean speech. During the experiments, the SA records of the training set are removed and the development set is used for tuning. Librispeech [28]. We considered the official "train-clean-100" subset of the dataset composed with 100 hours of speech in English. Models are evaluated on the standard "dev-clean" and "test-clean" sets.

Ensemble of teachers and distillation
A total of 40 different teacher architectures have been generated across the four datasets (i.e. 10 for each). For readability reasons, Tab. 1 only reports the different topologies for the TIMIT experiments. Nevertheless, the entire list of teachers along with their SpeechBrain training scripts are distributed on GitHub 2 . We also integrated the TIMIT example as an official SpeechBrain recipe to facilitate further investigations.
Teacher models. All teachers share the CRDNN encoder first introduced in [25]. Hence, N repeated convolutional (RC) blocks composed with two 2D convolutional layers, a layer normalisation, and a 1D frequency pooling operation are followed with a bidirectional RNN (e.g. LiGRU, LSTM or GRU). The recurrent network is finally connected to two dense layers with a batch normalisation operation in between. The CTC decoder is a simple linear layer, while the attentional one is a location-aware LSTM / GRU. Different teachers are obtained by varying all the possible hyperparameters including numbers of neurons per layer, type of RNN, dropout rate, learning rate, attention dimension, etc. Examples for TIMIT are reported in Tab. 1 and exact details are given in the configuration files on GitHub and SpeechBrain. Models are fed with 80 dimensional Mel filter banks with or without data augmentation (SpecAugment). Teachers are trained during 50, 30, 50, and 25 epochs for TIMIT, Librispeech, CV French and CV Italian respectively, following a validation-based learning rate annealing that ensures an optimal convergence for all models (i.e. no more performance can be obtained from longer training).
Student selection and distillation. The student model architecture is based on the best performing teacher from the ensemble. Therefore, we propose to pick the best teacher with respect to the best error rate on the validation set of the different datasets. Then, the selected model is trained with KD following the four training strategies detailed in Sec. 3 and compared to some other baselines (Sec. 4.3). For the initialization scheme, we propose to start from the pre-trained teacher neural parameters except for the last layer that is reinitialized randomly. Then, we fine-tune the whole architecture. More precisely, KD is applied for 50, 20, 40 and 20 epochs for TIMIT, Librispeech, CV French and CV Italian Table 1. Example of the different teacher models used to compose the ensemble on the TIMIT dataset. "RC" is the number of repeated convolutional blocks, "RNN" is the type of decoder RNN, "neurons" is the size of the hidden dimension in the encoder, "layers" is the number of RNN encoder layers, "dropout" is the dropout rate, and "data aug." represents whether data augmentation (Y) is applied or not (N). The selected teacher to be used as the student is in bold. "PER" is Phoneme Error Rate. RC  SOTA" are state-of-the-art ER obtained from the literature w.r.t the closest experimental setup (i.e. E2E ASR without language models). "Original" results give the ER obtained with teacher models selected to become students prior to distillation. "Average" is the baseline multi-teacher KD strategy detailed in Eq. 8. "Top-1", "Top-k" and "Weighted" are the error rate based KD strategies introduced in Sec. 3. respectively with SGD and with a validation-based learning rate annealing. For fair comparison, we also fine-tuned the best teacher models and did not observe any noticeable gain.

Speech recognition results
Tab. 2 shows the Phoneme and Word Error Rates (PER / WER) of the tested student-teacher strategies on the TIMIT, Librispeech and Common Voice test sets. It is important to note that results are obtained with respect to the best validation performance (i.e. not tuned on the test sets). We compare our proposed strategies with two different baselines: the literature, and an average weighted distillation strategy. Error rates are reported without further rescoring with a language model or additional self-supervised or unsupervised methods to enable a fair comparison of the acoustic models performance. Literature baselines report previous SOTA results with an hybrid DNN-HMM ASR system for TIMIT [29], and two different E2E encoder-decoder with attention systems for Common Voice [25] and Librispeech [30]. Best teachers have also been fine-tuned without any KD for more epochs for fair comparison, but failed at improving the performance observed prior to fine-tuning. First, it is worth noticing that the proposed error-based distillation strategies obtain SOTA results in the four considered tasks with a PER of 13.11% on TIMIT and WER of 17.04%, 15.57% and 10.09% on Common Voice French, Italian and Librispeech respectively, demonstrating the effectiveness of the approach. Interestingly however, the best ERbased stategy is not the same across all tasks. This may be explained by the very nature of the considered dataset. For instance, the "Weighted" strategy works better than the others on TIMIT while it is the "Top-k" on CV Italian. TIMIT is an extremely clean dataset, while CV Italian is the exact opposite. Hence, assigning weights to poor teachers with the "Weighted" strategy on TIMIT is not as negatively impactful as for CV Italian where WER can easily be higher than 100%.
Then, the "Average" baseline consistently fails at outperforming the original teacher model, empirically demonstrating and validating the importance of teacher weight assignment strategy. This method gives to a poor teacher the same emphasis as a good one. Indeed, students obtained from KD with "Average" degrade the performance on the four datasets compared to the original best teachers.
It is clear from Tab. 2 that the three proposed ER-based strategy are viable as they provide competitive performances across most datasets. For example, on Librispeech, "Top-k" and "Weighted" are only separated by 0.04% of WER. This demonstrates that error rate may be used as a robust proxy to be optimised instead of the loss as the variation in performance observed from different ER strategies remains low. On average, and over the four tasks, error rate based strategies improved the performance in WER by 0.74 over the best teach- Fig. 2. Output tokens (i.e. BPE) obtained on a French recording with the best (top) and worst (bottom) teachers. The given sentence contains 5 occurences of 3 different homophones (i.e. "vers", "verre", "vert"). While the worst teacher is grammatically wrong for 3 out of 5, produced tokens remain acoustically correct thus acting as an informed label smoothing during distillation. The original sentence is: "ce verre en verre de couleur verte est vers l'autre verre". ers. While such a gain might appear as not being enormous, it is worth underlining that this approach is recycling already trained models instead of simply discarding them and throwing away compute time and energy. During hyper parameter search these additional models could be used collectively to arrive at the final model.

Distillation Analysis
It is necessary to further investigate the effect of KD on an ensemble of teachers to obtain insights on the observed performance gain. To approach this, we propose two steps: a. empirically illustrate the impact of KD outside of the selected distillation strategy; b. analyse the behavior of the teacher selection within the different ER-based strategies on TIMIT. Fig. 2 illustrates the output tokens obtained with the attentional decoder of the best (top) and worst (bottom) teachers prior to distillation and originally trained on Common Voice French. The given utterance has been recorded in a clean environment and with a consumer-range microphone. We decided to voluntarily select a sentence containing five occurrences of three different French homophones (i.e. "vers", "verre", "vert"). From Fig. 2, it becomes clear that KD acts as an informed label smoothing, as bad teachers might provide grammatically wrong outputs that remain acoustically plausible. For instance, the worst teacher (bottom) gives the homophone " VERS" twice instead of " VER RE". Despite being wrong, this teacher still distill the acoustic information giving higher probabilities to words that "sound the same" Table 3. Cumulative number of selection for each teacher (Tab. 1) under "Top-1" and "Top-k" distillation strategies. instead of random ones.
Then, Fig. 3 and Tab. 3 provide the needed information to analyse the impact of the ER-based weighting strategy on the teacher selection process during the TIMIT knowledge distillation stage. For instance, Tab. 3 reports the cumulative number of selection for the ten teachers under the "Top-1" and "Top-k" strategies (same order as in Tab. 1). As expected, good teachers are selected more frequently than bad ones (e.g. T1 is the best performing teacher and is the most selected one with all strategies). Then, it is also confirmed that "Top-k" enables a more uniform selection among the teachers. For instance, teacher 8 goes from 1 selection during "Top-1" to 61 with "Top-k", thus increasing the diversity of labels. Finally, Fig. 3 depicts the variation of the weights associated to the three first teachers during the first 100 mini-batches with the "Weighted" strategy. Similarly to the others methods, T1 has a far higher average weight than the others. Nevertheless, by crossing with Tab. 3 it becomes clear that we will still always associate a weight to bad teachers that are most of the time ignored by the others strategies, thus explaining worse results with noisy datasets.

CONCLUSION
This paper introduces error rate based knowledge distillation from ensembles of teachers for joint ctc-attention end-to-end ASR systems with the definition of three different distillation strategies taking into account sequence-level error rates. The conducted experiments on four different datasets have highlighted promising performance improvements achieved under these strategies with state-of-the-art error rates achieved across all tasks. These findings enable the community to benefit from multiple trained models to further improve the error rate performance instead of discarding them, thus minimizing the loss of compute time.