{'fraction_non_alphanumeric': 0.031125511233866254, 'fraction_numerical': 0.012163382376374356, 'mean_word_length': 4.9170333123821495, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Semi-supervised Time Domain Target Speaker Extraction with Attention

In this work, we propose Exformer, a time-domain architecture for target speaker extraction. It consists of a pre-trained speaker embedder network and a separator network based on transformer encoder blocks. We study multiple methods to combine speaker information with the input mixture, and the resulting Exformer architecture obtains superior extraction performance compared to prior time-domain networks. Furthermore, we investigate a two-stage procedure to train the model using mixtures without reference signals upon a pre-trained supervised model. Experimental results show that the proposed semi-supervised learning procedure improves the performance of the supervised baselines.


Introduction
Communications in the real world often take place in complex auditory scenes, where the speech of a target speaker is likely to be contaminated by an interfering speaker or background noise.
To improve the quality of speech, deep learning methods have been developed for target speaker extraction (TSE) with promising performance [1,2]. As opposed to blind source separation (BSS) systems where all individual sources are separated by the model, TSE systems isolate the speech signal of the speaker in interest from all other speakers and ambiance noise. By focusing only on the target speaker and treating the rest of the signal as interference, TSE systems do not require the information of the total number of sources beforehand and do not suffer from the permutation ambiguity, which are the two primary challenges in BSS [3,4].
The VoiceFilter [1] is one of the first neural-based architectures to extract the speech of a target speaker. It contains a bidirectional LSTM (BLSTM)-based separation module to extract the magnitude mask of the target signal from the mixture spectrogram. A pre-trained speaker embedding module extracts cues of the target speaker, which requires an enrollment utterance of the target speaker as an additional input. The speaker embedding guides the separator during the mask estimation. Several studies follow this embedder-separator design with improvements in the architecture to enhance the extraction quality or running-time efficiency using spectral [5,2,6] or perceptual [7] features as input.
The recent success of end-to-end networks such as Con-vTasNet [8] in BSS has drawn attention to time-domain architectures in TSE. The separator includes paired 1D convolutional encoder and decoder, along with stacks of convolutional blocks in the middle. The embedder network can be pre-trained [9] or * Part of the work performed while at Amazon Web Services. learned jointly with the separator [10,11,12,13,14,15]. Inspired by the Sepformer [16] in BSS, we propose the Exformer and study the effectiveness of the multi-head attention (MHA) mechanisms with the transformer encoder [17] as the building block of the separator module for time-domain TSE. Specifically, we compare multiple configurations of the fusion of the speaker embedding into the separator module within the transformer blocks.
Despite the performance gain from the architectural improvement, these networks usually require a vast amount of labeled data to train. While it is unrealistic to obtain both the mixture and its source components in real-world acoustic conditions, noisy speech recordings exist on a large scale and are easy to collect. In the mixture invariant training (MixIT) [18], separation models are trained with mixture of mixtures in both unsupervised and semi-supervised setups. An alternative approach to leverage the noisy mixtures is to train the network with pseudo-labels assigned by a pre-trained teacher model, with promising results in singing-voice separation [19,20] and speech enhancement [21,22]. Meanwhile, training with noisy data in TSE is yet to be explored. We propose a semi-supervised framework to train the Exformer with both labeled and unlabeled data. Empirical studies on two-speaker mixtures show that the proposed Exformer outperforms the prior Conv-TasNetbased architectures under supervised learning, and the incorporation of unlabeled data with the semi-supervised setup further improves the quality of the extracted signal.

Target Speaker Extraction Model
The proposed Exformer model ( Figure 1) is based upon the time domain speech separation network Sepformer [16] with a speech encoder, masking network, and a speech decoder. A pre-trained BLSTM speaker embedder is used to extract speaker information to guide the extraction. The speaker extractor (encoder, masking network, and decoder) is trained with the negative scale-invariant signal-to-distortion ratio (SI-SDR) [23], and the parameters of the embedder network are fixed.

Speaker Embedder
The embedder network obtains a fix-sized embedding vector ze = femb(xe) ∈ R D of the target speaker, where femb is the embedder network and xe is the enrollment utterance. Following the embedder design in [1], the mel-spectrogram of the enrollment speech is processed by a stack of BLSTM layers to model sequential information. The average of the hidden-state representation from the first and the last frames is projected to a lower dimension with a linear layer. Finally, the projection output is normalized with unit l2 norm to obtain the speaker embedding ze. The embedder network is pre-trained with the generalized end-to-end loss (GE2E) [24].

Encoder
The encoder network consists of a 1D convolutional layer followed by ReLU activation. It transforms the time domain input mixture x ∈ R L with L samples to a hidden representation H ∈ R F ×T with F features and T temporal frames.

Masking Network
The masking network takes the mixture representation H and the speaker embedding ze as input and estimates a mask Mc ∈ R F ×T , c ∈ {1, 2, . . . C}, where C is the total number of sources. We set C = 2 as the model estimates one mask for the target speech and another mask for the residual interfering signal. Similar to DPRNN [25] and Sepformer [16], the masking network consists of three modules: segmentation, embeddingguided dual-path processing, and overlap-add.

Segmentation
The segmentation module divides the T frames from the mixture representation H into S segments, each with length K, with 50% overlap between segments. All segments are concatenated to form a tensor

Embedding Fusion
The embedding fusion module combines the segmented mixture representation and the target speaker embedding ze. We apply embedding fusion at the beginning of each processing block as in [13,14,11,12]. At the beginning of block j, we perform where fus is the fusion function that applies independently to each segment and frame, and the superscript j denotes the block index.
We experiment with multiple approaches to incorporate speaker information into the separation pipeline. The first method concatenates the embedding and the latent mixture representation similar to [1,26,13,14,15,11]. The embedding ze is first duplicated on the frame and the chunk dimensions to obtain Ze ∈ R D×K×S . Then, Ze is concatenated along the feature dimension to V (j−1) and linearly projected back to the F -dimensional space:  The dual-path processing module is made up of a stack of dual-path transformer blocks, where each block contains a fusion layer for processing speaker information, an intra-chunk layer for learning local dependency, and an interchunk layer for modeling global dependency.
In the second approach, we first project the embedding Ze to the same dimension as V (j−1) and then element-wise multiply with the mixture representation following [12]: where stands for element-wise multiplication. We also consider replacing the element-wise multiplication with addition similar to the application of positional embeddings in transformer [17], leading to our third approach: Figure 2a shows the pipeline of a dual-path transformer block. After the embedding fusion, the dual-path module alternates the processing between short-term and long-term information. The intra-chunk layer computes the local dependency within each segment, while the inter-chunk layer communicates between the segments to model global information. As shown in Figure 2b, each of the intra-chunk and inter-chunk layers contains multiple transformer encoder blocks [17], including layer normalization [27], multi-head attention (MHA), and a feed-forward layer with residual connections. A sinusoidal positional encoding is applied before each MHA layer. Each transformer block preserves the number of features F . The intra-chunk layer operates on each of the S segments independently and computes the local dependency within each segment. For the j-th dual-path block, the intra-chunk layer is applied to the second dimension of the speaker-guided input tensor W (j) by

Dual-Path Processing
where f (j) intra represents the j-th intra-chunk layer, and the intrachunk output U (j) is obtained by concatenating {U along the third dimension. The inter-chunk layer processes the long-term, global dependency between the segments and is applied to the third dimension of the intra-chunk layer's output and we concatenate {V along the second dimension to obtain the block output V (j) , which is further processed by block j + 1. With the interleaving intra-chunk and inter-chunk layers, the model effectively models both local and global dependency in turn.
With a total of B dual-path blocks, the output V (B) is processed by a 2D convolutional layer to increase its dimension by C folds, representing each of the C output sources with

Overlap-add
Finally, an overlap-add operation transforms the S segments in each M c back to T frames followed by a ReLU activation to obtain the estimated mask Mc ∈ R F ×T for source c.

Decoder
The decoder is a 1D transposed convolutional layer that transforms the masked representation Hc = Mc H ∈ R F ×T of each source c to the estimation of the time domain signal sc = TransposedConv1D(Hc) ∈ R L .

Semi-supervised Setup
Prior works on TSE focus on supervised training with the reference speech available. To incorporate a large unlabeled corpus, where the ground-truth target is missing, we propose a semi-supervised approach by applying a triplet speaker representation loss [28]. Given the estimated target signalŝt and residualŝr, we compute the embeddings zt = femb(ŝt) and zr = femb(ŝr) with the pre-trained embedder, and compute their distances to ze which is obtained from the enrollment speech. If the estimated target is separated from the residual, ze should be much more similar to zt than to zr. The triplet embedder loss is defined as Lemb(ze, zt, zr) = max{ ze − zt 2− ze − zr 2 + γ, 0}, where γ = 1 represents the margin. The embedder loss is computed along with the reconstruction loss in a supervised context in [28], while we extend its application to unlabeled data. Specifically, when the ground-truth signal is not available, we only compute the embedder loss; otherwise, we compute both the reconstruction loss LSI-SDR and Lemb. The overall objective for the semi-supervised setup is where λs, λu are hyperparameters controlling the two losses, 1 is the indicator function, and Ds represents the dataset with ground-truth target speech. One concern with applying semi-supervised training from scratch is that it might be difficult for the model to obtain optimal separation performance quickly since the parameters are drifted away by the embedder loss (7). We experiment with a two-stage setup where the model is first trained with SI-SDR using only labeled data and then trained with both labeled and unlabeled data with (8). In the supervised stage, the model learns to adapt the speaker embedding and to extract the target speech for high-quality reconstruction. When the performance reaches the plateau, we enter the second stage by introducing the unlabeled data and the embedder loss. The goal for the second stage is to improve the model's ability to generalize with the additional unlabeled data.

Dataset
We train and evaluate the system with the simulated 2-speaker mixture from the WSJ0-2mix-extr 1 dataset. The recordings are at a sampling rate of 8 kHz. The training, development, and test set contains 20,000, 5,000, and 3,000 utterances, respectively. The mixtures are generated from speech recordings at a signalto-noise ratio (SNR) between 0 and 5 dB. The first speaker is chosen to be the target speaker and the second speaker is treated as interference. For each target speaker, another speech recording from this speaker is randomly selected as the enrollment utterance. For the semi-supervised setup, we incorporate over 100,000 recordings from the VoxCeleb 1 corpus [29]. Since the recordings may contain background and reverberation, they are not suitable for the standard supervised setup.

Model and Training Details
First, we pre-train the speaker embedder following [1]. The embedder takes the 40-dimensional log mel-spectrogram as input. It consists of 3 BLSTM layers, each with 768 hidden units, and a linear projection head is attached to obtain an embedding with size 256. The network is trained on the VoxCeleb 1 & 2 [29] and the clean training partition of the Librispeech [30] corpus combined, which may contain ambiance noise or reverberation. To adapt to the memory constraint, each batch consists of 46 speakers, each speaker with 4 utterances of 4 seconds long. The network parameters are updated with the Adam optimizer [31] with an initial learning rate of 5e-4. During preliminary experiments on supervised TSE, we obtain the best performing teacher network with the embedder checkpoint at 550,000 steps with an equal error rate (EER) of 6.5% on the VoxCeleb1 verification test set.
We train the extractor network with the optimal configuration from [16]. The encoder convolution has a kernel size of 16 and a stride of 8 with 256 output channels. The masking network contains two dual-path blocks. Each intra-chunk or inter-chunk module contains 8 transformer encoder layers, each layer with 8-head attention and a 1024-dimensional feed-forward projection. The decoder mirrors the encoder with the same kernel and stride sizes. We experiment with concatenation, element-wise multiplication and addition as the embeddingmixture fusion discussed in 2.3.2.
During training, we apply dynamic mixing [16] to generate mixtures on the fly from a random 3-second segment from each source recording. If either of the sources is less than three seconds, zero padding is applied on both ends. We also apply time-domain speed perturbing by 5% as data augmentation. We set the batch size as 1 for the experiments, and we use an Adam optimizer with an initial learning rate of 1.5e-4. The learning rate is reduced by half if the validation loss is not improved for two consecutive epochs after 85 epochs until the learning rate reaches 1e-6.
For the semi-supervised training setup, we weight the two loss functions with λs = 1 and λu = 0.05. We experiment with different probabilities of data sampled from the labeled versus unlabeled corpus. For the two-stage setup, we proceed with the best-performing model from the supervised setup. We adjust the initial learning rate for the second stage to be 7.5e-5, and the aforementioned learning rate scheduler applies after 65 epochs. We encourage the readers to refer to our implementation 2 .

Results and Discussions
We compare the different configurations of the proposed Exformer with previous works using the test partition of the WSJ0-2mix-extr dataset. We evaluate each system with the SI-SDR and the perceptual evaluation of speech quality (PESQ) [32] score of the model's output with respect to the reference speech.  Table 1 summarizes the performance of previous methods on time-domain target speech extraction as well as the proposed architecture with multiple configurations under the supervised setup. We report the mean SI-SDR and PESQ for each setup. The statistics for the baselines are adapted from [14].

Supervised Training and Fusion Configuration
All three variants of the Exformer outperform the baseline methods [26,14]. The performance gain can be attributed to the use of the dual-path transformer blocks in the masking network which effectively model local and global dependency. Our observations are consistent with the findings of BSS systems in [16], where the performance of the transformer blocks dominates over that of the convolutional blocks.
We also study the impact of different fusion strategies for speaker embedding and mixture representation. Among the three fusion types, additive fusion has the best performance with a gain of 0.3 dB SI-SDR over element-wise multiplication and a 0.8dB gain over concatenation. Addition and multiplication share the same PESQ score slightly higher than concatenation. We show in our experiments the effectiveness of adding a timeinvariant speaker embedding to the latent representation, allowing the extraction network to effectively capture the conditioning speaker information. We next study the impact of the embedder loss (7) and the use of unlabeled data. We report in Table 2 the performance of the Exformer trained on different combinations of the loss functions, number of training stages, and the probability of unlabeled data being sampled into a training batch. We initialize the networks in the second stage with the weights of the best performing supervised system in Table 1.

Semi-supervised Training and Embedder Loss
Training the network from scratch using the embedder loss yields worse performance. This observation verifies our hypothesis that the embedder loss does not contribute to optimal separation performance with random initialization compared to training with reconstruction loss alone. With two-stage training, however, the extraction performance improves consistently among various percentages of applying unlabeled data. As the percentage increases, the performance reaches the peak at a probability of 10% of sampling unlabeled data in both evaluation metrics with a 0.6 dB gain of SI-SDR over the supervised baseline. The results show that the system benefits from a twostage procedure with the embedder loss using a small ratio of unlabeled to labeled data.

Conclusion
We propose the Exformer, a time-domain transformer-based architecture for target speaker extraction. Under the supervised training setup, the Exformer significantly outperforms prior time-domain networks. We further show that the extraction performance can be enhanced with a two-stage semi-supervised pipeline incorporating mixtures without reference signals. In future work, we would explore target speaker extraction under noisy and reverberant conditions and investigate approaches to include massive unlabeled data such as curriculum learning.