{'fraction_non_alphanumeric': 0.023260331180498226, 'fraction_numerical': 0.010780362259020055, 'mean_word_length': 5.231164901664145, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Probing Statistical Representations For End-To-End ASR

End-to-End automatic speech recognition (ASR) models aim to learn a generalised speech representation to perform recognition. In this domain there is little research to analyse internal representation dependencies and their relationship to modelling approaches. This paper investigates cross-domain language model dependencies within transformer architectures using SVCCA and uses these insights to exploit modelling approaches. It was found that specific neural representations within the transformer layers exhibit correlated behaviour which impacts recognition performance. Altogether, this work provides analysis of the modelling approaches affecting contextual dependencies and ASR performance, and can be used to create or adapt better performing End-to-End ASR models and also for downstream tasks.


INTRODUCTION
The typical approach to develop a framework for ASR has been to use deep neural networks to recognise and align acoustic features to graphemes or phonemes; replacing the requirement for distinctlyoptimised modules, such as acoustic, pronunciation or language models (LMs). Using End-to-End modelling approaches reduces the need for expert domain knowledge as it aims to jointly optimise the training regime while adapting to diverse speech environments. These factors have led to End-to-End ASR models becoming a popular choice for on-device deployment.
Current research in End-to-End ASR modelling is dominated by three approaches: recurrent-transducers [1], Connectionist Temporal Classification models [2] or attention-based encoder-decoder architectures [3,4]. End-to-End ASR frameworks are typically dependant on the amount of data resources and are commonly fine-tuned to the corpora in order to improve the recognition performance, which has directed techniques to improve the representation capacity of the modelling approaches [5,6]. The ability to use larger amounts of training data significantly improves the recognition performance of End-to-End models [7]. However, previous work [8] suggested that the relationship between memorisation and generalisation within these networks remains elusive, being referred to as a "black-box". It can be hypothesised that richer neural representations are not analogous to increasing neural depth or model size [9]. By learning representations of speech that are robust, the general recognition performance of the model should improve proportionately without the requirement for increasing model size or training data.
As the development of End-to-End ASR models continues, there is also a corresponding demand to be able to interpret the internal latent representations and the explainability of modelling approaches, This work was partly supported by Voicebase Inc. at the Voicebase Research Center such as the potential relationship between training data and the integration of LMs. Despite numerous variations of modelling approaches, there has also been little exploration of the internal model representations and their relationship to model recognition performance across different domains. Layer-wise analysis of models has been used to interpret modelling approaches and relationships between representations in multiple domains [10,11,12]. SVCCA analysis techniques have been used to highlight neural representations with respect to their ability to generalise, by observing the relationship between the correlation coefficients of neural layers during training [13].
As transformer modelling approaches achieve state-of-the-art results for End-to-End ASR, this work aims to understand how representations in transformer models adapt to out-of-domain LMs by analysing how the representations evolve across layers. The relationship between ASR performance and the neural representations is explored. Using unmatched sub-word LMs, it is possible to observe the dependencies of the representations within model layers and observe the impact of the variations. Section 5 shows that observing the representation dependencies is important to develop intuitive modelling approaches and improve recognition performance.
The analysis methodology is first defined in Section 2, then the developed framework for analysis experiments is described in Section 3.3. The implementation of cross-domain LMs in the End-to-End ASR models is defined in Section 3.2 and the analysis experiments are shown in Section 4. Experiments analysing the adaptation of transformer model parameters are conducted in Section 5 with the results discussed in further detail in Section 6. It was found that deeper transformer model layers contain learned representation dependencies for cross-domain LMs and recognition performance can be improved by tuning the parameters to the hierarchical dependencies.

CORRELATION ANALYSIS METHODOLOGY FOR END-TO-END ASR MODELS
To map the posterior distribution of p(Y |X) jointly, End-to-End ASR modelling approaches aim to approximate function of acoustic input X = {x1, ..., xT } of length T over output labels Y = {y1, ..., yN }. The dependencies of the End-to-End model parameters, are not as clearly distinguished as traditional modular approaches to acoustic and language modelling, as they are not separately defined. These "black-box" models are complex and it can be difficult to understand the relationship between the internal parameter dependencies and the resulting performance across datasets. By using SVCCA analysis from [14], it is possible to observe a relationship between two sets of observations by the maximum correlations between the linear projections. This correlation analysis method is used to measure the linear relationship between vectors representing neural layers. For ASR models, the activation output of a layer z l i = {z l i (x1), ..., (z l i (xN )} are extracted for dataset X and neuron i in layer l. To observe the similarity between or within neural layers and to compare the relationship of the internal representation dependencies upon different variables, the use of different LMs and model parameters are varied and the similarities between layer representations are compared to derive insights regarding their relationship.
To measure correlation for N datapoints, pairs of vectors are sampled from views of layers l1 and l2, which are projected, where The projected views of l1 and l2 are pruned by the application of singular vector decomposition (SVD) to retain 99% of the representative dimensions and to reduce the impact of potential noise. The application of SVD forms subspaces l 1 ⊂ l1 and l 2 ⊂ l2 and then CCA can then be applied to find vectors v and s that maximise correlation ρ between the projections of l 1 and l 2 : where v, s are transforms that aim to maximise the correlation of the vectors. ρ increases where neural representations have encoded more similar information. Further details regarding SVCCA analysis can be found in [14].

Transformers for ASR
Transformers, initially published in [3], are a widely chosen encoderdecoder architecture for speech recognition frameworks due their ability to parallelise the training regime. This enables use of larger amounts of training data. The encoder structure is comprised of stacked self-attention and point-wise, fully connected layers in blocks, with each block managed by a multi-head self-attention layer and feed forward layer. The attention module is a major component of the architecture as it enables pairwise position measurement for windows of the input speech. After the attention vector has passed through the feed forward layer, the output is passed onto another self-attention layer within the decoder to compute the embedding vectors. A final attention mechanism attends over the embedding vectors, in order to compute the relationship between the input and output sequences, before being passed to the final linear unit. A softmax function is then calculated over the target output.
A CNN front-end is incorporated with the transformer layers for feature extraction. The final convolutional layer was then projected to 12 stacked transformer encoder blocks with embedding dimensions of 512 × 2048 and 6 decoder layers with positional embeddings.

Language Modelling in End-to-End ASR
The integration of LMs within End-to-End architectures is relatively common-place as they can be used to supervise the training optimisation and also for decoding to aid recognition performance [15,16]. However, it is unclear how the internal dependencies of the End-to-End models handle latent LM representations and whether there are similar learned representation spaces that are robust across different domains. By training models with cross-domain LMs, it is possible to observe these dependencies by comparing models with SVCCA analysis.
For the analysis experiments, a sub-word LM is integrated by shallow-fusion decoding [17] and label smoothing [18] techniques. The sub-word LM is a 3 layer LSTM model. Shallow-fusion decoding computes the weighted sum of a pair of posterior distributions over sub-words; using one from the ASR model and one from the sub-word LM. The sub-word LM is an LSTM-based LM trained with restricted computational complexity, by only keeping the most frequent sub-words and splitting the rest into characters, to enable conversion with low information loss. Label smoothing computes the cross entropy loss during the model's training regime with a weighted mix of distributions from a unigram LM and one-hot targets from the dataset.

Correlation Analysis Framework
The framework developed in [12] was utilised to investigate the relationships between internal dependencies and save the models during training. For all the experiments, the transformer models and LMs were trained using the ESPRESSO framework [16]. The analysis was conducted for all models by extracting the activation outputs of each neural layer of the encoder for each training epoch. Each model was saved throughout all epochs and then a controlled input of 100 frames of unseen speech data was fed through the layers, whilst simultaneously extracting the activation outputs for each layer. 80dimensional log Mel acoustic features with additional pitch features were extracted, from 25ms windows with a stride of 10ms, for all models.

Data
For the experiments, three common US-English datasets from differing domains for ASR were chosen: Switchboard [19], Librispeech [20] and WSJ [21]. The Switchboard dataset contains conversational telephone speech, Librispeech is a compilation of read audiobooks, and WSJ contains read news. The test sets for the Switchboard dataset, referred to as Swbd and Callhome, are derived from the LDC2002S09 set and contain 20 unreleased telephone conversations from Switchboard and 20 telephone unscripted conversations from Callhome. To ensure the transformer models were converged by training on comparative data to the LM, the model trained with Switchboard used up-sampled data (to 16kHz). The LMs trained with Librispeech were trained using the full 960 hour training set and the ASR models were tested on the test-clean and test-other sets. The training set for the WSJ LM was the si284 set, with the Dev93 set for validation and Eval92 for testing the ASR model performance.

CROSS-DOMAIN LANGUAGE MODEL EXPERIMENTATION
Correlation analysis of the neural representations across the transformer model layers is used to measure and analyse the changes in correlation when cross-domain LMs are integrated. Figure 1 shows the SVCCA coefficients, as training converges, between the encoder layers of two transformer models. The models were trained with Switchboard data but one model uses an in-domain Fisher sub-word LM, and the other model uses an out-of-domain WSJ sub-word LM. These models are both trained with sub-word units using Senten-cePiece [22] and integrated during the training process using the scheduled sampling method and decoded with shallow-fusion, as described in Section 3.2. The correlation analysis shows very little difference in coefficiency between layers 1 to 6 (top graph of Figure 1), aside from in the initial epochs which could be attributed to the random initialisation of parameters. This suggests that the neural layers of both of these models are converging to similar representation spaces. However, between layers 7 to 12 (bottom graph of Figure 1, the differences in coefficiency are much larger throughout training. This suggests that the representations learned in these deeper layers are more dependent upon the LM domain. The bottom graph in Figure 2 displays the standard deviations of the coefficiency between the models trained with cross-domain LMs. This aims to show the variation in coefficiency by layer more clearly, where the standard deviations in layers 10, 11 and 12 are highest. The top graph of Figure 2 shows the variance in coefficiency within the neural layers of a model trained without scheduled sampling or shallow-fusion decoding compared to the model trained with the Fisher sub-word LM. This suggests that a similar observation can be made for LM specific representations, whereby the variance is higher overall and the coefficiency of layers 8 to 12 deviates the most. The results in Figure 2 also imply that layers 1 to 4 have very little dependency on LM representations. These insights suggest that encoder layers 1 to 4 of the transformer model can be frozen when fine-tuning with LMs and the optimisation regime of End-to-End ASR models can be adapted to improve downstream tasks. Regarding performance, the model that was trained with the Fisher LM reached 9.5% word error rate (WER) on the Switchboard test set and 19.1% on the Callhome test set, while the model that was trained with the WSJ LM was 10.7% and 21.1% respectively. The differences in recognition performance are attributed to the domains of the LMs and the test sets used for evaluation.

MODELLING STRUCTURE ANALYSIS
To optimise the parameters for state-of-the-art End-to-End ASR models, many iterations are trained with slight parameter modifications. Optimisation of model parameters to specific datasets to achieve the best recognition performance possible [16,15] is referred to here as tuned. For example, the dimensionality, number of layers and also the hyperparameters have been observed to impact the recognition performance. As shown in Table 1, using a transformer model with the same parameters and composition for several datasets does not achieve the lowest WER across all of the datasets. These tuned models are typically reached by extensive hyperparameter optimisation techniques, which are computationally expensive and considerably time consuming without providing observational evidence regarding the dependencies of certain parameters upon the recognition performance.
Using cross-corpora correlation analysis while varying the parameters, it is possible to interpret these dependencies in a more meaningful way and provide some observational evidence to reduce the future need for extensive hyperparameter optimisation when developing new models or fine-tuning trained models. By understanding the representation dependencies, this can potentially reduce the computational resources required to improve speech recognition model architectures. Table 1 shows the results of 3 transformer models with variations in model parameters that are used in stateof-the-art End-to-End ASR frameworks. All models are the same transformer-based encoder-decoder architecture with the following variations: • Model 1 has an embedding dimension of 512, a feed forward embedding dimension of 2048, 4 attention heads, and an attention dropout of 0.25.
• Model 2 has an embedding dimension of 256, a feed forward embedding dimension of 1024, 4 attention heads, and attention dropout of 0.25.
• Model 3 has an embedding dimension of 512, a feed forward embedding dimension of 2048, 8 attention heads, and an attention dropout of 0.1.
To observe the relationship between the learned representations of the adapted models and attribute these adaptations to improved recognition performance with specific data, the model performance was assessed across all test sets, as shown in Table 1. For the Switchboard and Callhome test sets, the recognition performance of model 1 is the best, while model 2 reaches slightly worse performance on the Callhome set and model 3 has the highest WER for both test sets.  Figure 3 displays the SVCCA coefficients for each model trained with the Switchboard dataset. Model 2's mean coefficiency, across layerwise representations, are substantially less correlated than the other models. The standard deviations of the correlations within these layers also vary significantly higher than Model 1 or 3. Model 3's mean coefficiency across layerwise representations is fairly similar for all layers with very small standard deviation. It is observed that correlations within the layers of model 2 have lower coefficiency, and the recognition performance of this model is lower for the Switchboard test sets. Also, there are little hierarchical coefficiency patterns throughout the layers of model 3, and this model also has a slightly worse performance, which corroborates with results from [12]. Model 1 has lower coefficiency within layers 8-12 and has the best recognition performance.

DISCUSSION
The findings in Section 3.2 correlate with findings from [23] where semantic and syntax level features of speech are predominantly dependent upon deeper layers of transformer-based models, while acoustic and fluency features are predominantly dependent on the shallower layers. In the case of End-to-End transformers for ASR tasks, the LM-dependent representations are shown to be primarily dependent within layers 7-12. The cross-domain LM-dependent representations are observed within layers 10-12. Further experiments training with the WSJ dataset with Fisher and WSJ sub-word LMs showed very similar observed behaviours across layer coefficiency. These observations can be used to check for possible biases in the modelling process that affect recognition performance, without the need for extensive training requirements, to improve joint optimisation. The analysis also aids in the interpretability of the impact of representation dependencies within End-to-End ASR models.
The experiments in Section 5, attempt to show these internal dependencies with regard to the model parameters within the same modelling approaches. As shown in Figure 3, Model 2 used shallower embedding dimensions than model 1, which has caused the coefficiency of many of the layers to become highly uncorrelated. Model 3 is observed to have very highly correlated layers, however there are little distinct hierarchies in the neural representations when the attention heads are increased to 8 and the attention dropout is reduced. By adapting the parameters of transformer models, the layers with the most dependency for representing domain-specific information are altered. These changes in hierarchical representations have been observed to impact recognition performance, and further suggests a relationship between correlated hierarchical representations and the ability for the model to generalise, particularly for crossdomain speech recognition. Increasing the attention dropout is theorised to improve model robustness [6], where typical features of conversational speech are boundary uncertainties and hesitations. In the case of End-to-End conversational speech recognition, the results show that using substantial attention dropout in transformer models is important to produce correlated hierarchies in dependent layers but also utilise a model with sufficient embedding dimensionality that the representations within context-critical layers don't become too uncorrelated.

CONCLUSION
Using SVCCA as a correlation index has highlighted several aspects of the relationships between the neural representations, transformerbased modelling parameters and the impact these have upon recognition performance. Interpretative analysis is important to develop future modelling approaches for meaningful improvement strategies. Expanding the scope of the investigation into the attributes and potential learned features that could be classified within the layers would provide a deeper understanding of the properties of these dependencies and how these could be further exploited. The insights into the dependencies of the neural layers can be used for the development of models for few-shot learning and downstream tasks for End-to-End ASR.