{'fraction_non_alphanumeric': 0.025080821774950465, 'fraction_numerical': 0.0076128897695275834, 'mean_word_length': 4.956211180124224, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Towards Disentangled Speech Representations

The careful construction of audio representations has become a dominant feature in the design of approaches to many speech tasks. Increasingly, such approaches have emphasized"disentanglement", where a representation contains only parts of the speech signal relevant to transcription while discarding irrelevant information. In this paper, we construct a representation learning task based on joint modeling of ASR and TTS, and seek to learn a representation of audio that disentangles that part of the speech signal that is relevant to transcription from that part which is not. We present empirical evidence that successfully finding such a representation is tied to the randomness inherent in training. We then make the observation that these desired, disentangled solutions to the optimization problem possess unique statistical properties. Finally, we show that enforcing these properties during training improves WER by 24.5% relative on average for our joint modeling task. These observations motivate a novel approach to learning effective audio representations.


Introduction
The recent success of unsupervised pretraining in language processing can be credited to the advent of sophisticated techniques for learning representations of text. Methods like ELMo [1], GPT [2], and BERT [3] work by using an unsupervised task that develops a representation of text that is useful for downstream tasks in a way that is agnostic to what that task is.
In applying these lessons to unsupervised pretraining in speech great progress has been made with the discovery that in a data-intensive domain like audio, it is best to learn a representation that discards unimportant parts of the signal. Contrastive estimation [4], in which a full reconstruction is not learned, has yielded representations that achieve strong results in speaker identification and speech recognition [5]. State-of-the-art methods combine contrastive learning with masked language modeling as in Wav2Vec 2.0 [6] and Adaptive SpecAugment [7].
Such successes can be seen as signaling a movement away from task-agnostic representations and towards "lossy" representations, in which a model learns not only to summarize relevant portions of a signal but also to discard portions that are irrelevant to the downstream task. This distinction is particularly clear in the world of multi-modal representation learning, where we seek a representation specifically of the intersection between two domains (e.g. audio and images [8,9] or audio and text [10]). However, while there are several natural methods for learning a representation that models components of a signal that are required for a task, it is difficult to craft a method that compels a model to specifically exclude irrelevant components. Approaches in this space have commonly relied on techniques like adversarial learning to exclude particular parts of a signal thought to be irrelevant, as in [11].
In this study, we present a novel architecture specifically designed to learn a measurably disentangled representation of audio using supervised data. Our model is based on the paradigm of dual learning [12,13,14,15], which seeks to exploit the "duality" between ASR and TTS. Traditionally, this is done by training a model that performs both ASR and TTS with a shared encoder that is tasked with representing inputs from both the speech and text domains [16,17]. Our model adds a secondary encoder, which is intended to capture specifically those parts of the audio signal that are irrelevant to the transcript. While the primary encoder is utilized for both ASR and TTS, this secondary encoder is used only for audio reconstruction, which is a task that requires both that part of the audio signal that predicts the transcript and the "residual" signal that does not. We argue that disentanglement is facilitated by the explicit modeling of the residual signal by the secondary encoder, and demonstrate this disentanglement by training a speaker-ID classifier on the outputs of both the primary and secondary encoders.
Other studies have shown that in scenarios where more than one solution to an optimization problem is possible (such as generalized vs. overfit solutions [18] and selection of significant units in a DNN [19]), the stochasticity of parameter initialization and minibatch selection can be decisive. We present empirical evidence that speech signal disentanglement is such a problem. We find that both entangled and disentangled solutions to our dual learning problem are possible, and that the superior, disentangled solution is arrived at randomly. We then observe that the disentangled solution has the unique statistical property of using a large amount of its variational capacity in both encoders. Finally, we show that enforcing this property during training with an additional loss term substantially improves ASR quality.
Possible applications of our joint modeling task include refinement of back-transcription based semi-supervised learning systems such as speech chains [20] and Sequential MixMatch [21]. We believe that our discoveries motivate the usage of a secondary encoder in such systems to achieve disentanglement in semi-supervised audio representations.
The rest of this paper is structured as follows. Section 2 describes our architecture for a joint audio and text model that can simultaneously perform ASR, TTS, and audio and text reconstruction. Section 3 presents the design for our experiment investigating the nature of disentangled representations. Section 4 details the results of that experiment and observations of the statistical differences between entangled and disentangled representations. We summarize our findings and discuss future work in Section 5.

Architecture
In this section, we describe a joint ASR, TTS, and reconstruction model built on the dual-learning paradigm. Our architecture is depicted in Figure 1.

Architecture Summary
Our model is trained to consume either text or audio input, and to emit both text and audio. In that way, for a given input the model either performs ASR and speech reconstruction (audio input) or TTS and text reconstruction (text input).
These tasks are performed by way of a pair of encoders, each of which yield a data representation. The "joint" (or primary) encoder can consume either either audio or text, while the "audio-only" (or secondary) encoder consumes only audio input. There are also two decoders, one corresponding to each of the domains. The text decoder consumes the output of the joint encoder only, while the audio decoder consumes the outputs of both the joint and audio-only encoders, combined by way of a "embedding combination module", which consists simply of three transformer layers.
For tasks with audio input (ASR and audio reconstruction), the joint encoder consumes zeros instead of text. For tasks with text input (TTS and text reconstruction), the joint encoder consumes zeros instead of audio, and the embedding combination module consumes zeros instead of the outputs of the audio-only encoder.

Encoder Architecture
The joint encoder is split into two sub-encoders, one to encode audio and one to encode text, each implementing a state-ofthe-art encoding scheme. The audio sub-encoder is based on Conformer [22], and consists of 17 conformer blocks with subsampling so that the length of the audio input sequence is reduced by a factor of four. The text sub-encoder is based on Tacotron 2 [23], and consists of a embedding projection and positional encoding followed by a pre-net and transformer module. As in [23], the pre-net consists of three blocks of a 1D convolution with a 5x1 filter and a dropout layer that zeros out 10% of its input. The transformer block borrows from the original transformer architecture in [24] and consists of three blocks of multi-headed self-attention followed by a feed-forward layer.
In order to produce a representation that is agnostic to the input domain, we would like to ensure the joint encoder emits a representation of approximately equal length for both domains. Otherwise, for example, the audio decoder might learn to model audio reconstruction and TTS separately based on encoding length. To this end, we adapt the length-transformation component from [25] by which a representation is compressed into a shorter sequence where each element is a weighted average of elements from the original sequence. In particular, a sequence z1, ..., zM is converted to toz1, ...,zN (with N < M ) as: where s is a learnable spread parameter and σ represents softmax normalization across all weights α j k for fixed j. The audio-only encoder consists simply of four conformer blocks. These blocks do not include sub-sampling, so the output of the audio-only encoder is the same length as the audio input.

Decoder Architecture
The audio and text decoders are adapted from Tacotron 2 [23] and Transformer [24] respectively.
The audio decoder consists of a pre-net, autoregressive RNN, and post-net. The autoregressive component consumes its own previous output, and passes it though a simple audio pre-net which consists simply of a projection and dropout layer. We then attend to the outputs of the embedding combination module and concatenate the obtained context vector to the processed audio. This input is passed to a small recurrent network (two LSTM layers) which emits the autoregressive prediction. As in [26], we find that tuning the dropout in the audio pre-net is critical to convergence, since without dropout in the autoregressive input the model simply learns to copy the previous frame. We find the best results with 10% dropout.
As in [23], we find significant improvement in TTS when the autoregressive decoder output is further processed by a nonautoregressive convolutional post-net. We use a stack of five convolutions to refine the autoregressive prediction. During training, we jointly optimize the cross-entropy of both predictions.
The text decoder is a conventional Transformer [24] decoder, consisting of two blocks each containing a projection, self-attention, and cross-attention.

Experiments
We've described how an audio input passed to our model is represented separately by the joint encoder and audio-only encoder. When optimized to perform the four tasks of ASR, TTS, and audio and text reconstruction, we may naturally imagine two classes of solutions that the model might arrive at: • A "disentangled" representation, in which the joint encoder output (which will be consumed by the text decoder) represents that part of the audio signal relevant to the transcript, while the audio-only encoder output (which is only consumed by the audio decoder) represents that part of the audio signal that is not relevant to the transcript. For example, the joint encoder might represent phonetics, while the audio-only encoder might represent prosody, background noise, and channel effects.
• An "entangled" representation, in which that part of the audio signal relevant to transcription is not particularly favored by either representation.
We seek to observe which of these two representations is learned by our model. To this end, we train our model fifteen times on the given joint task, arriving at fifteen different solutions to the optimization problem. We then freeze the parameters of the model, and for each of the fifteen instances we train: • A classifier to determine the speaker ID for a speech example given the model's joint encoder output.
• A classifier to determine the speaker ID for a speech example given the model's audio-only encoder output.
For a model that has learned a disentangled representation, we expect to be able to predict speaker ID best from the audioonly encoder output, since speaker information is ostensibly required for audio reconstruction but irrelevant to transcription.

Entanglement Classifiers
Speaker IDs are learned using a custom classifier that applies a positional embedding to the selected encoder output followed by three transformer blocks with multi-headed self-attention, five convolutions with a 3x3 filter and stride of 2 and finally a projection and softmax layer.

Model Settings
As in [23] and [26], we process audio inputs into mel spectrograms with a short-term Fourier transform (STFT) using a frame size of 50 ms and frame hop of 12.5 ms. We then apply a Han windowing function before applying a mel filterbank, yielding 80-dimensional vectors for our model's audio input.
For text, we choose to use grapheme-level inputs such that the outputs of the embedding layers are 72-dimensional vectors. While a wordpiece representation might have yielded stronger ASR results, we found that graphemes most reliably ensured convergence of all tasks.
All components use a model dimension of 256, with the model containing about 68 million parameters in total. Each model is trained with a batch size of 256 split across 16 TPUs. After 150k steps, we freeze the joint model and train each disentanglement classifier for 100k steps.

Training
To jointly optimize our four tasks, we split each batch into two halves. The first half consists of text inputs and represents the TTS and text reconstruction tasks, while the second half consists of audio inputs and represents the ASR and audio reconstruction tasks. For all elements in the batch, we optimize the loss where Ltext is the cross-entropy loss for the text output, Laudio ar is the cross-entropy loss for the audio output before the convolutional post-net, and Laudio final is the cross-entropy loss for the audio output after the covolutional post-net. We find this setup to train more quickly and to converge better than regimens in which tasks alternate across batches.

Data
For training data, we choose the Librispeech corpus [27]. We see Librispeech as ideal for this experiment since it contains a diverse set of speakers such that there is a significant part of the audio signal to represent outside of the transcript. We train our models in particular on the "clean" subset of the training data, which contains about 460 hours of speech. For WER measurements we evaluate on the "clean" test set.

Results
In this section, we report the results of our experiments and analyze the learned representations.
We point out that by the nature of this experiment, our model had to be trained from scratch many times, leading to considerable resource constraints. These constraints forced us to simplify the training procedure by using only the clean Librispeech data, a small batch size, and a small number of training steps. This combined with the additional TTS and reconstruction tasks leads to WER values considerably worse than the state of the art for ASR only. With that in mind, we draw conclusions based on the changes in WER and representation properties across different solutions. Figure 3 plots the WER of the joint model against the two classification losses described above after training with frozen encoder parameters. We quickly make the observation that of our fifteen runs, one has an unusually strong result with a WER of about 9%.

Classification
The speaker ID classification task shows a clear pattern. The strongest model achieves a training loss that is more than ten times better than the next strongest model on the task from its audio embedding, and more than two times better than on its own joint embedding, suggesting that speaker information has been mostly disentangled from the transcript and localized to the audio embedding.

Representation Properties
Having seen that our model can sometimes, subject to the randomness of training, achieve a much better WER than average, we seek to understand the nature of that stronger, disentangled representation. In particular, we suspect that in a model without the desired disentanglement, the audio-only embedding is underused.
To this end, we sample the audio-only and joint representations of our best model and of one of our other models. Since each input contains a large number of frames, we are able to collect several thousand 256-dimensional vectors from just a few examples. For each representation, we perform an SVD on those vectors and normalize the squared singular values. In this manner, we obtain a measurement of the proportion of variance in the representation attributable to each of its 256 dimensions. We consider a representation with significant variance in a large number of dimensions to be more used by a model than one in which only a few dimensions vary.
The results of these measurements are plotted in Figure 2. We see a stark difference between the distribution of variance in a weak, non-disentangled representation and our strong, disentangled representation. In particular, the disentangled solution has very few significant dimensions in its audio-only embedding, with the first three dimensions capturing more than 95% of the variance. By contrast, the disentangled solution has a much larger number of significant dimensions, with almost 50 dimensions containing more than 0.1% of the total variance each.

Correlation Loss
Having observed that strong performance occurs together with a relatively uncorrelated audio-only embedding, we naturally wonder if optimizing for that property at training time will yield better WER. To test this, we interpolate an additional "correlation loss" into the the training of our joint model:  where b ∈ B are the elements in the batch, X is the matrix formed by stacking the audio representation of the batch element along the time axis, A is a learnable linear projection, I is the identity matrix, and the inner summation adds up each (unsigned) element of the given matrix. This loss is intended to act as regularization that pushes the off-diagonal elements of the correlation matrix to zero, yielding a representation with uncorrelated elements. We achieve the strongest results setting the hyperparameter α = 10 −5 .

Model
Average WER Non-Disentangled 15.5% Disentangled 9.8% Correlation Loss 11.7% Table 1: WER with and without the Correlation Loss The distribution of singular values in Figure 4 shows clearly that the added loss has the intended effect of decorrelating the audio embedding. It also suggests that this is done by moving information over from the joint embedding, which has become lower-dimensional. WER results are given in Table 1. We see that the correlation loss yields on average a 24.5% reduction in WER.

Conclusions
In this paper, we presented a novel dual-learning architecture capable of learning a disentangled representation of audio. We associated disentanglement directly with strong performance on the ASR task and with a high-dimensional audio embedding.
We envision future work in semi-supervised ASR that will train our dual-learning model on supervised data to learn a disentangled audio representation which can then be fine-tuned with both unpaired audio and text data via back-transcription. We also believe that our work gives rise to more fundamental optimization questions. We plan to investigate if correlation loss truly promotes disentanglement, or if it reduces WER by some other means.