{'fraction_non_alphanumeric': 0.042054532250610684, 'fraction_numerical': 0.014524328249818447, 'mean_word_length': 4.662803738317757, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
L-SpEx: Localized Target Speaker Extraction

Speaker extraction aims to extract the target speaker's voice from a multi-talker speech mixture given an auxiliary reference utterance. Recent studies show that speaker extraction benefits from the location or direction of the target speaker. However, these studies assume that the target speaker's location is known in advance or detected by an extra visual cue, e.g., face image or video. In this paper, we propose an end-to-end localized target speaker extraction on pure speech cues, that is called L-SpEx. Specifically, we design a speaker localizer driven by the target speaker's embedding to extract the spatial features, including direction-of-arrival (DOA) of the target speaker and beamforming output. Then, the spatial cues and target speaker's embedding are both used to form a top-down auditory attention to the target speaker. Experiments on the multi-channel reverberant dataset called MC-Libri2Mix show that our L-SpEx approach significantly outperforms the baseline system.


INTRODUCTION
Human has the ability to selectively listen to a particular speaker through various stimuli in a multi-talker scenario, that is called selective auditory attention in cocktail party problem [1]. Ever since the theory is proposed, researchers never stop seeking the engineering solution to confer human's selective attention capability on machines, as there is high demand for various real-world applications, such as speech recognition [2,3] and speaker verification [4,5].
With the advent of deep learning in recent years, blind speech separation methods have been widely studied to solve the cocktail party problem by applying neural networks [6][7][8][9] and beamforming [10][11][12][13]. The neural network seeks the regular patterns (i.e., masks) between the time-frequency representation of the target speech and mixture speech, while beamforming incorporates the spatial statistics (i.e., spatial covariance matrix) obtained from the estimated masks to compute beamformer's weights and filter the desired voice. However, speech separation always requires that the number of speakers is known as a prior, and assumes the label permutation is unchanged during training, which greatly limits its scope of real-world applications.
Unlike blind speech separation, speaker extraction only extracts the target speech from a mixture speech driven by spectral [14,15] or spatial cues [16][17][18] of the target speaker. The spectral cue is always represented by a speaker embedding from an enrolled reference utterance, while the spatial cue is usually transformed into spectrum-like features derived from the target speaker's location. For example, Chen et al. [16] introduced a location-based angle feature to guide separation network, which was the cosine distance between the steering vector and inter-channel phase difference (IPD) for each speaker in the mixture. Gu et al. [17] suggested that the beamforming output could be regarded as an alternative way of spatial cues, as beamforming aimed to summarize the signals from the target speaker's direction and suppress non-target signals. However, these studies often require the speaker location is known in advance or detected using an extra visual cue.
To address this issue, we propose an end-to-end localized target speaker extraction on pure speech cues, that is called L-SpEx. We design a target speaker localizer driven by an enrolled utterance of the target speaker to extract the target speaker's DOA and beamforming output, simultaneously. By doing so, the extracted spatial cues and the enrolled utterance can be further used to guide the network to learn which direction and speaker to be extracted.

L-SPEX ARCHITECTURE
As illustrated in Fig. 1, the difference between the proposed L-SpEx system and other speaker extraction systems lies in the target speaker localizer (Fig. 2).

Target speaker localizer
The target speaker localizer learns to encode the spatial cues related to the target speaker's direction from the multichannel mixture signal y(c, n), with reference to a reference utterance x(n) by the target speaker. We have, where c denotes the channel index and n denotes discrete time index. s(c, n) represents the target signal and b i (c, n) represents the interference signal corresponding to speaker i. Formally, let Y t,f,c ∈ C be the STFT coefficient of the c-th channel mixture signal y(c, n) at time-frequency bin (t, f ), and let X t,f ∈ C be the STFT coefficient of an enrolled single-channel utterance x(n) of the corresponding target speaker. As shown in Fig. 2, we first employ a network to estimate a complex-valued mask, as opposed to a real-valued mask, for speaker location estimation. This is motivated by Sharath's work [19], which shows that speaker location estimation relies strongly on both phase-differences and magnitude-differences between the microphones. Thus, the complex-valued masks (cm) tgt t,f is estimated as follow: where y t,f = {Y t,f,c } C c=1 ∈ C C is the spatial vector of the signals obtained from all C-microphones for each timefrequency bin (t, f ). CMaskEst(·) and Enc speaker (·) represent a complex mask estimator and speaker encoder, and both structures are built with several BLSTM layers as shown in   and Im[·] denotes the real and imaginary part of a complex tensor, respectively. After the masks are estimated, we use (cm) tgt t,f y t,f as the input of DOA estimator to predict target speaker's DOA. We employ two CNN layers with kernel size of 1 × 7 and 1 × 4 on the masked input, followed by residual network blocks with a number of 5. Then, a 1×1 convolutional layer with 181 output channels (corresponding to 181 azimuth directions) projects the features to the DOA space. Finally, three 1 × 1 convolutional layers and a mean pooling operation summaries the time and frequency to obtain the 181-dimension DOA vector: where DOAEst(·) denotes the DOA estimator. Motivated by the studies [17,20], beamforming output is also regarded as a direction-related spatial cue, as beamforming has the ability to summarize the signals from target speaker's direction. Therefore, the estimated masks are also used to compute the cross-channel spatial covariance matrices (SCMs) Φ j f and then obtain the beamforming outputB t,f , Here, j ∈ {tgt, inf} and m inf t,f = 1 − m tgt t,f . u ∈ R C is a vector denoting the reference microphone, and tr(·) denotes the trace operation. w f = {W t,f,c } C c=1 ∈ C C is corresponding to time-invariant beamformer coefficients. C denotes the number of channels and H denotes the conjugate transpose.

Localized target speaker extraction
Given the estimated DOA likelihood codingd and beamforming outputB t,f of the target speaker in Sec. 2.1, we can obtain two direction-related spatial features, i.e., DF angle and DF beam . The former [16] is derived from the estimated target speaker's angleθ = argmax(d), the latter is derived from beamforming outputB t,f , which are defined as follows where Ω contains P microphone pairs, and o l,r = ∠Y t,f,l − ∠Y t,f,r represents the observed inter-channel phase difference (IPD) between left channel l and right channel r. N FFT is the number of FFT bins, v is the sound velocity and f s is the sampling rate. Note that f ranges from 0 to (N FFT − 1). ∆ l,r denotes the distance between the microphone pair (l, r).
The above two direction-related spatial features have an ability to inform the extraction network of target speaker's direction, while the speaker embedding obtained from a speaker encoder can guide the network to attend to the target speaker. Thus, we use both spatial features as the inputs of the mask estimator to predict better masks for the target speaker extraction. The better masks (cm) tgt t,f are calculated as where Concat[·] is the concatenation operation. CMaskEst and Enc speaker denote the complex mask estimator and speaker encoder in extraction network for speaker extraction. Finally, the masks are used to obtain the extracted STFT spectrumŜ t,f ∈ C by using MVDR formula, i.e., Eq. (4) and (5), and further estimate the target signalŝ via iSTFT, s = iSTFT(Ŝ t,f ) = iSTFT(MVDR ((cm) tgt t,f , y t,f )) (10)

End-to-end training
We first pretrain the target speaker localizer by a multi-task learning strategy, and then optimize the whole network. The loss function of target speaker localizer is defined as follow: where α and β are scaling factors. L SI-SDR aims to minimize the signal reconstruction error.ŝ b = iSTFT(B t,f ) and s are the estimated signal and the target clean signal of reference microphone, respectively. L CE is the cross-entropy loss for speaker classification. N denotes the number of speakers. p i is the true class label for speaker i, andp i represents the predicted probability corresponding to i-th speaker. L MSE is the mean squared error for target DOA estimation. M is the number of azimuth directions, here M = 181.d i and d i are the predicted and ground-truth DOA coding of the target speaker. Based on the likelihood-based coding in [21], the desired ground-truth values d i are defined as follows: where θ is true target speaker's angle and θ i is one of 181 azimuth directions. σ is the parameter to control the width of the Gaussian curves. d(·, ·) is the azimuth angular distance. After target speaker localizer is pretrained, we optimize the whole L-SpEx network by using the following loss, whereŝ is the extracted signal via Eq. 10, andq is predicted probability from the speaker encoder Enc speaker . γ is also a scale factor to balance the two objectives.

EXPERIMENTS AND DISCUSSION
To facilitate the evaluation, we introduce a multi-channel reverberated version of the Libri2Mix 1 dataset which we refer to as MC-Libri2Mix. The original Libri2Mix is a clean 2talker mixture corpus generated from the LibriSpeech corpus by mixing two randomly selected utterances. Libri2Mix contains training data with 64,700 utterances (270 hours, 1,172 speakers), development data with 3,000 utterances (11 hours, 40 speakers), and test data with 3,000 utterances (11 hours, 40 speakers). The average duration of the utterances is 14.8s. The room impulse responses (RIRs) in MC-Libri2Mix is simulated using pyroomacoustics 2 package. For the room configurations, the length and width of each room are randomly drawn in the range [5,10]m, and the height is selected in the range [3,4]m. The reverberation time (RT 60 ) of the reverberant data ranges from 200ms to 600ms. In MC-Libri2Mix, we consider a linear array with four microphones, where the microphone-to-microphone distance is 5cm [20]. Target speakers are placed in the frontal plane and are at least 15 • apart from each other. The distance of speaker and the center of microphone is from 0.75m to 2m.
Unlike in blind speech separation, we set that the speakers in each 2-talker mixed speech acted as the target speaker in turn, and the corresponding auxiliary reference speech is randomly selected from original LibriSpeech corpus. In practice, the training set (127,056 examples, 1,172 speakers) and Table 1. SDR (dB) and SI-SDR (dB) in a comparative study on the MC-Libri2Mix dataset under open condition. "m" and "cm" represent the real-value mask and complex-value mask, respectively. "Pretrained Speaker Localizer" indicates the extracted signalŝ b that is derived from the output spectrumB t,f of the pretrained target speaker localizer in Fig. 2

Experimental setup
We train all systems for 70 epochs on the 4-channel mixture segments and their corresponding reference utterances. The learning rate is initialized to 1e −4 and decays by 0.5 if the accuracy of validation set is not improved in 2 consecutive epochs. Early stopping is applied if no best model is found in the validation set for 5 consecutive epochs. Adam is used as the optimizer. For feature extraction, STFT is performed with a 8k Hz sampling rate and a 25ms window length with a 10ms stride, and the feature dimension is 257. For mask estimation network, we use three BLSTM layers with 512 cells, and the dimension of speaker embedding is 256. The speaker embedding is inserted between first and second BLSTM layer. For the loss configuration, we used α = γ = 0.5, β = 10 to balance the loss. The parameter σ in Gaussian curve is 6.

Results and analysis
We compare L-SpEx with the mask-based MVDR baseline systems on MC-Libri2Mix in terms of SDR and SI-SDR. From Table 1, we conclude: 1) Our L-SpEx approach achieves the best performance under the open condition. Compared to the "Mask MVDR (cm)" system, L-SpEx leads to 20.7% and 19.0% relative improvement in terms of SDR and SI-SDR measure, respectively. The improvements mainly come from the augmented spatial features through the proposed target speaker localizer module.
2) The proposed L-SpEx with DF beam achieves 0.94 dB and 0.89 dB performance gain over the "Mask MVDR (cm)" baseline in terms of SDR and SI-SDR. This result proves that the spatial feature derived from beamforming outputs can let network attend to the signal from the target direction and ignore nontarget signals.
3) The result of L-SpEx system with DF beam and DF angle further show the effectiveness of spatial feature. This shows that DF angle and DF beam represents different aspects of spatial cues of target speaker, and the two features complement each other. 4) The result evaluated on the output of pre-trained speaker localizer is worse than the mask-based MVDR systems. The reason is that the DOA estimation and beamforming have something in common, but they are still two separate tasks and one mask is hard to optimize. We further report the speaker extraction performance on different angle distance mixture speech in Table 2. From Table 2, we find that the mixture speech with smaller angle difference (i.e., < 45 • ) is more difficult to extract target speaker. It is observed that our L-SpEx system achieves the SI-SDR from 5.82 dB to 7.06 dB, which is even higher than the performance of "Mask MVDR (cm)" on large angle distance (i.e., > 45 • ). Furthermore, we report the extraction performance with different and same gender mixture speech, separately in Table 3. We observe that separating same-gender mixture is a more challenging task, as the spectral cues between each speaker are similar. When we incorporate the direction-related spatial information (i.e., DF beam and DF angle ) to increase the discrimination between speakers, the extraction process starts to be easier. Specifically, our L-SpEx system improve 1.19 dB SI-SDR performance compared with the spectral-only "Mask MVDR (cm)" system.

CONCLUSIONS
In this paper, we proposed an end-to-end localized target speaker extraction on pure speech cues, that is called L-SpEx, to eliminate the assumption of known target angle in existing studies. We took the advantages of an target speaker's enrolled utterance to design speaker localizer for estimating direction-related spatial cues. Experiments showed that the extracted spatial cues and the enrolled utterance input let extraction network works better, as target speech can be extracted based on speaker and direction aspects.