{'fraction_non_alphanumeric': 0.04133545310015898, 'fraction_numerical': 0.010731319554848967, 'mean_word_length': 4.665353222628765, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Multi-Target Filter and Detector for Speaker Diarization

A good representation of a target speaker usually helps to extract important information about the speaker and detect the corresponding temporal regions in a multi-speaker conversation. In this paper, we propose a neural architecture that simultaneously extracts speaker embeddings consistent with the speaker diarization objective and detects the presence of each speaker frame by frame, regardless of the number of speakers in the conversation. To this end, a residual network (ResNet) and a dual-path recurrent neural network (DPRNN) are integrated into a unified structure. When tested on the 2-speaker CALLHOME corpus, our proposed model outperforms most methods published so far. Evaluated in a more challenging case of concurrent speakers ranging from two to seven, our system also achieves relative diarization error rate reductions of 26.35% and 6.4% over two typical baselines, namely the traditional x-vector clustering system and the attention-based system.


Introduction
Speaker diarization is the process of determining when individual speakers are active in a recording. The aim is to generate a diary of the presence of each speaker at each point in time. It also plays an integral role in the pre-processing stage of automatic speech recognition (ASR) and understanding. This technique has been widely used for speech processing in various scenarios, such as conference conversations, broadcast news, debates, and cocktail parties [1]. However, the resilience of speaker diarization remains weak due to the challenges posed by variations in recording channel, environment, reverberation, ambient noise, and the number of speakers [2].
For nearly a decade, researchers have tackled diarization problems using probabilistic models [3] or neural networks [4]. Many methods involve two steps, segmentation and clustering. In the segmentation step, a 1.5-second sliding window (with 50% overlap) is typically used to divide a session into a sequence of short segments. Then, a speaker model is used to extracted the speaker representation (e.g., the x-vector [5,6,7,8], i-vector [9,10], or d-vector [11,12]) of each segment. During the clustering process, segments with highly homogeneous characteristics form a group. Different clustering techniques have been used according to various similarity measures, such as probabilistic linear discriminant analysis (PLDA) and cosine similarity [13,14,15,10]. For example, agglomerative hierarchical clustering (AHC) and spectral clustering (SC) were used in [6,10] and [16,17], respectively. The unbounded interleaved-state recurrent neural network (UIS-RNN), stemming from the Gaussian mixture model (GMM) [18,19] and hidden Markov model (HMM) [7], was used in [12]. In addition, some post-processing methods, such as Variational Bayes (VB) [20] and the LSTM-based method [21], have been applied to refine the initial diarization results.
Recently, some research [22,23,24,25] has focused on end-to-end speaker diarization. Fujita et al. [22] reformulated the diarization task as a multi-label classification problem and utilized the permutation-invariant training (PIT) [26] technique in model training. Moreover, in [27], reliable speaker embeddings were derived by a selector to help a voice activity detector diarize a session. Self-attention [28,29] and frame selection [30] were also used in end-to-end speaker diarization.
The traditional two-step method cannot well handle overlapping speech in a session. Target speaker voice activity detection (TS-VAD) [31], originally designed to handle multi-speaker ASR, achieved good performance on overlapping speech processing. It relies on a x-vector/SC procedure [5,8] to provide first-stage timestamps of each "target" (active) speaker's speech, which are used to extract the first-stage i-vector for each target speaker from frames where only a single speaker is active. Finally, TS-VAD uses the i-vectors of all speakers and MFCCs to generate diarization results. Unfortunately, TS-VAD can only be applied to sessions with a fixed number of speakers, because its neural structure contains tensor concatenation of speaker embeddings. Inspired by the dualpath recurrent neural network (DPRNN) [32,33], an outstanding model in speech separation, we propose a unified structure called Multi-Target Filter and Detector (MTFAD) that can handle conversations with various speaker numbers with a single model. Furthermore, finding that the quality of speaker representations has an impact on the diarization performance, we extend TS-VAD by building with a neural filter that can directly extract speaker representations suitable for the diarization task.
The main contribution of this paper is threefold. First, we greatly extend the practical scope of TS-VAD while inheriting its excellent performance in the speaker diarization task. In this sense, MTFAD has an advantage over TS-VAD because MT-FAD does not set a limit on the number of speakers in a session. In addition to expanding practical use, since training data of different speaker numbers can be used together to train a single model, the model is more powerful than multiple separate models, each trained with training data of a specific number of speakers. Second, we design a filter that can be jointly trained with the diarization model to extract speaker embeddings. With this filter, we not only get better diarization performance, but also avoid pre-training the i-vector extractor. Third, unlike some previous studies, such as [6,12], which deal with only nonoverlapping speech, our model also performs well on data containing overlapping speech regions.  is an elementwise product operator of MFCCs and each mask (labels of speaker occurrences) generated from the first stage.
generate first-stage prediction timestamps for each speaker, and then these timestamps can be used to obtain the i-vector for each speaker. The i-vector is extracted by a pre-trained speaker model (i-vector extractor) from the frame-level MFCCs corresponding only to the target speaker. The second step requires two inputs, i.e., the frame-level MFCCs and the i-vectors of all target speakers. Similar to the structure shown in Fig. 1, the MFCCs are first passed through a four-layer convolutional neural network (CNN) and then respectively concatenated to each speaker's i-vector to form the input of a BiLSTM. The output of the BiLSTM for each speaker is concatenated and fed into another BiLSTM for final classification. The spec of the second BiLSTM depends on the number of speakers in a session and leads to the limitation of TS-VAD. Therefore, TS-VAD can only be applied to a task with a fixed number of speakers.

MTFAD
Like TS-VAD, the proposed MTFAD system also adopts a twostep process, and the speaker timestamps are also provided by an first-stage speaker diarization step. As shown in Fig. 1, MT-FAD consists of a detector and a filter.

MTFAD detector: BiLSTM/DPRNN/Linear
DPRNN is a common method for speaker separation tasks. As shown in Fig. 2, the main feature of DPRNN is the local and global data chunk formulation in the dual-path BiLSTM module. The model first partitions the input tensor Xinput into chunks with or without overlap through a sliding window of length T and concatenates them to form a 3-dimensional tensor of shape D×T ×N , where D is the number of channels, and N  Figure 2: A DPRNN module contains an intra-block and an inter-block. Operator + denotes the addition of two tensors. Xinput, Xintra, and Xinter have the same shape. In case of cascaded DRPNN modules, the preceding Xinter is the direct Xinput of the next block.
is the number of sliding windows. The 3-dimensional tensor is converted by the "overlap-add" operation into consecutive outputs. The BiLSTM module goes through two dimensions: the size of the chunks (intra-block RNN) and the number of chunks (inter-block RNN). As a result, the RNNs can see information around and far away from the current time frame.
The dual-path BiLSTM module consists of an intra-block and an inter-block in order. Both blocks contain BiLSTM, Linear, and LayerNorm in a row. The output of each block is a residual addition of the input and output of LayerNorm. The only difference between these two blocks is their input. Xinput is segmented into N intra-chunks (D × T ). All intra-chunks, i.e., Xinput, are input to the intra-block. The output of the intrablock, Xintra, is resegmented into T inter-chunks (D × N ), which are input to the inter-block. The output of the inter-block, Xinter, is used as the input of the next dual-path BiLSTM module. By recursively processing data through the intra-block and inter-block, DPRNN provides good speech separation results.
Inspired by DPRNN, we design a detector that can handle different numbers of speakers. In speech separation, the length of each session is different, so the number of blocks (N ) in the DPRNN framework is a variable. As shown in Fig. 1, by treating SDEs as intra-block (D × T ) and the number of speakers as the variable N , we can replace the second BiLSTM in TS-VAD with DPRNN to obtain the detector in MTFAD. The SDEs are generated in the same way as in the TS-VAD framework. Instead of forming a 2-dimensional shape of T × N D by concatenating along dimension D, we combine N matrices of where D, T and N denote the number of hidden channels, the number of frames in a session, and the number of target speakers, respectively. The D × T × N multi-SDE tensor is the input to the dual-path BiLSTM module.
Each SDE with size D × T is the detection embedding of a speaker. Therefore, we can regard it as the "intra-speaker" detection embedding, which is a reflection of intra-chunk in DPRNN. The intra-block can gather information on whether the target speaker is active. As for the D ×N matrix, it contains the information whether each target speaker is active in the specific time frame, which can be viewed as "inter-speaker" detection embedding, a reflection of inter-chunk in DPRNN. Therefore, we re-segment the D × T × N multi-SDE tensor into N matrices of D × T as the input of the intra-block and re-segment the  2). Therefore, the detector of MTFAD does not take the number of speakers as a structural parameter and can handle the task with any number of speakers in a unified structure.

MTFAD filter: z-vector (diari"z"ation vector)
We argue that the quality of speaker embeddings is critical for diarization performance. Therefore, we design a filter to specifically extract speaker embeddings suitable for diarization. The filter requires the speaker timestamps in the session, which are provided by the x-vector/AHC diarization step. With the speaker timestamps and the MFCCs of the session, the filter generates the corresponding speaker embeddings using ResNet and Attentive Statistic Pooling [34], as shown in Fig. 1. The output of the filter is called the z-vector. These z-vectors can be used as detector inputs instead of x-vectors or i-vectors. By integrating the filter with the MTFAD detector, speaker embedding extraction and detection is combined into an end-to-end process. In addition, since the filter and detector are trained jointly, the z-vector is expected to be more suitable for the diarization problem then the x-vector and i-vector.

Experiments
Two corpora were involved in our experiments: one is simulated from the Switchboard and NIST SRE datasets, and the other is CALLHOME. All overlapping regions were counted in the performance evaluation. The results were evaluated by diarization error rate (DER) and Jaccard error rate (JER). JER, introduced by DIHARD II, is based on the Jaccard index [35].

SWB+SRE Simulated Corpus
We utilized the additional Switchboard corpus and the NIST SRE dataset. The total number of speakers in the 683 hours of data from SRE and Switchboard was 6,392. We simulated the training data by Algorithm 1 in [28]. To achieve an overlap ratio of 20%, with two, three, and four participants, we chose parameters β for three, six, and nine seconds in the algorithm, resulting in 137, 226 and 320 hours of data, respectively. In the training phase, the ground-truth Rich Transcription Time Marked (RTTM) was used as first-stage diarization. In the inference phase, we used x-vector/AHC to produce the firststage RTTM files on the simulated set. Each speaker's representation vector was extracted from a cascade of time frames in which only he or she was active. In both phases, the i-vector and x-vector speaker representations were obtained by Kaldi [36] pre-trained extractors. For the z-vector case, the first-stage RTTM was used as the input to the diarization models.

Results and discussion
First, we investigated the impact of the quality of speaker representations on diarization performance. Filter in MTFAD was removed, and the z-vectors were replaced with the x-vectors (cf. Fig. 1). During MTFAD training, the x-vectors were extracted from the target speakers' utterances in the unmixed Switchboard and NIST-SRE datasets (denoted as T), or derived from the speaker timestamps labeled by the first-stage diarization of x-vector/AHC (denoted as I). The ratio of |T |/(|T |+|I|) represents the extent to which true speaker representations are used for training. As shown in Table 1, we can see that this ratio hardly affects performance. However, regardless of the ratio, all Ideal test conditions outperform their Threshold and Oracle counterparts. Although Ideal is a cheating condition, these results demonstrate the importance of accurate speaker representations. Therefore, we conclude that extracting better speaker embeddings will be the key to producing better results.
Then, we compared the effects of different speaker embedding models, including z-vector, i-vector, and x-vector. As shown in Table 2, we can find that while the MTFAD model with i-vector or x-vector can improve the performance over the baseline x-vector/AHC method, the MTFAD model with zvector performs the best. The results confirm that the z-vector jointly trained with the MTFAD model is more effective than the x-vector and i-vector obtained from pre-trained models, and better speaker embedding yields better diarization performance.

CALLHOME
CALLHOME, also known as NIST SRE 2000 CALLHOME (LDC2001S97), is a telephone dataset that contains conversations in multiple languages. A total of 500 conversations are recorded at a sampling rate of 8kHz. The number of speakers in each conversation varies from two to seven. Since the CALL-HOME dataset was too small to train our model, we used the SWB+SRE dataset for pre-training.
In the training phase, we pre-trained the MTFAD models on the SWB+SRE dataset. Because there is no official development set defined in the CALLHOME task, we followed the instructions in the Kaldi recipe to split the set equally into two Table 3: Results (%) on CALLHOME (assuming the number of speakers is known). Both TS-VAD and MTFAD are based on the first-stage diarization of x-vector/AHC and i-vector. For each specific number of speakers, MTFAD* and TS-VAD were trained on the same training data. MTFAD was trained with all training data containing different numbers of speakers.

Method
Oracle  parts. CALLHOME-1 was used for fine-tuning the pre-trained models, and CALLHOME-2 was used for evaluation. We determined the parameters in x-vector/AHC based on the performance of CALLHOME-1 (the development set). In the inference phase, we used x-vector/AHC to produce the first-stage RTTM files on CALLHOME-2 (the evaluation set). Like the SWB+SRE task, we obtained the i-vector and x-vector speaker representations according to the RTTM files.
Since the number of speakers in a session in CALLHOME varies from two to seven, we had to pre-train and fine-tune the TS-VAD models separately for each number of speakers. For this purpose, the SWB+SRE and CALLHOME-1 datasets were split into 2-speaker, 3-speaker and 4-speaker subsets for training the corresponding TS-VAD models. For each TS-VAD model, we also trained a corresponding MTFAD* model using the same training data and procedure for comparison.

Results and discussion
First, we compared MTFAD with TS-VAD. The results evaluated on CALLHOME-2 are shown in Tables 3. The experiments were conducted under the 2-speaker, 3-speaker, and 4speaker conditions, and the number of speakers was assumed to be known. All MTFAD and TS-VAD systems are based on the first-stage diarization of x-vector/AHC. From the table, we can see that MTFAD* always outperforms the corresponding TS-VAD and x-vector/AHC baselines under the same condition. Moreover, MTFAD achieves better results than MTFAD*, because MTFAD was trained with all training data containing different numbers of speakers, while each MTFAD* model was trained with only a subset of training data containing a specific number of speakers. MTFAD also outperforms SA-EEND-EDA [29]. These results demonstrate the advantage of the MT-FAD detector: MTFAD performs better because its detector is Table 5: DER (%) on the 2-speaker CALLHOME task. rel. % represents the relative reduction rate over the baseline xvector/AHC.

Method
DER rel. % x-vector/AHC [29] 8.93 -BLSTM-EEND [24] 23.07 -158.3 SA-EEND [28] 10.99 -23.1 SA-EEND-EDA [29] 8.35 6.5 SA-EEND-EDA + Frame Selection [30] 7.84 12.2 MTFAD 7.82 12.4 able to utilize all data in training. Next, we compared the effects of different speaker embedding models, including z-vector, i-vector, and x-vector. From Table 4, we can see that all the three MTFAD models outperform not only the x-vector/AHC baseline but also the strong SA-EEND-EDA model [29]. Furthermore, the results again confirm that the z-vector-based MTFAD is better than the ivector-based and x-vector-based MTFAD under both Threshold and Oracle conditions. We also notice that greater improvements are seen under the Oracle condition. This is because that when the number of speakers is correct in the first-stage diarization, the model can estimate a more accurate z-vector for each speaker. In contrast, under the Threshold condition, the wrong number of speakers predicted in the first stage x-vector/AHC may cause some z-vectors to not match the actual speakers, thus leading to small improvements in DER and JER. The results in Tables 3 and 4 show that with the well-designed filter and detector, the MTFAD model is a flexible and effective diarization model that can extract more accurate speaker vectors to handle conversations with different numbers of speakers.
Finally, we compared the proposed MTFAD model with more existing models. Since most end-to-end models were only evaluated in 2-speaker experiments, we compared different models in the 2-speaker CALLHOME task. Table 5 shows the results. It is clear that MTFAD outperforms all models, with a 12.2% relative reduction in DER over the baseline xvector/AHC [29]. From Tables 4 and 5, we conclude that the MTFAD model outperforms all models compared in this paper in both 2-speaker and multi-speaker tasks.

Conclusions and Future Work
In this paper, we have proposed the MTFAD model, which is developed from a DPRNN-based detector and a z-vector filter, for speaker diarization. The DPRNN structure in the MTFAD detector allows the model to handle conversations with varying numbers of speakers and use data with any number of speakers in training. The detector addresses the weaknesses of TS-VAD and maintains its strengths. The z-vector filter dedicated to diarization also improves performance compared to the traditional i-vector and x-vector.
Currently, the performance bottleneck comes from the misprediction of the first-stage x-vector/AHC. We plan to retrofit the original MTFAD so that it can automatically correct the estimated number of speakers in a conversation provided by the first stage, or even drop the first stage. In addition, the current post-filter is a simple median filter, and we want to replace it with stronger filters such as VB and HMM. Furthermore, we would like to incorporate reconstruction losses commonly used in speech separation to enhance diarization.