{'fraction_non_alphanumeric': 0.02856982802083917, 'fraction_numerical': 0.01075569996078651, 'mean_word_length': 4.807417046193884, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Speaker Diarization with Session-Level Speaker Embedding Refinement Using Graph Neural Networks

Deep speaker embedding models have been commonly used as a building block for speaker diarization systems; however, the speaker embedding model is usually trained according to a global loss defined on the training data, which could be suboptimal for distinguishing speakers locally in a specific meeting session. In this work we present the first use of graph neural networks (GNNs) for the speaker diarization problem, utilizing a GNN to refine speaker embeddings locally using the structural information between speech segments inside each session. The speaker embeddings extracted by a pre-trained model are remapped into a new embedding space, in which the different speakers within a single session are better separated. The model is trained for linkage prediction in a supervised manner by minimizing the difference between the affinity matrix constructed by the refined embeddings and the ground-truth adjacency matrix. Spectral clustering is then applied on top of the refined embeddings. We show that the clustering performance of the refined speaker embeddings outperforms the original embeddings significantly on both simulated and real meeting data, and our system achieves the state-of-the-art result on the NIST SRE 2000 CALLHOME database.


INTRODUCTION
Speaker diarization is the problem of "who spoke when". A typical speaker diarization system usually contains multiple steps. First, the non-speech parts are filtered out by voice activity detection (VAD). Second, the speech parts are split into small homogeneous segments either uniformly or according to the detected speaker change points. Third, each segment is mapped into a fixed dimensional embedding, such as i-vector [1], xvector [2] or d-vector [3,4,5,6,7]. Finally clustering methods or end-to-end approaches are applied to generate the diarization results [8,9,10,11,12,13]. Usually a classifier needs to be trained for similarity scoring for i-vectors and x-vectors, while similarities between d-vectors can usually be measured by simple distance metrics, e.g. cosine or Euclidean distance.
Commonly-used clustering methods for speaker diarization include K-means [14], agglomerative hierarchical clustering * Initial work by first author done as an intern at Microsoft. Fig. 1. Overview of the proposed method. A graph is constructed for each session using speaker embeddings extracted by a pre-trained speaker embedding model. A GNN model is applied to remap the original embedding into another embedding space which is trained according to a loss function defined by the difference between the refined affinity matrix and the ground-truth adjacency matrix.
(AHC) [15], spectral clustering (SC) [16] and affinity propagation [17]. Although deep learning methods driven by large scale datasets have dominated the fields of speaker and speech recognition, it is still non-trivial to design an end-to-end objective function for the speaker diarization problems which is permutation invariant in terms of both the speaker order and speaker number. Most recently there have been several end-to-end approaches that either utilize a factored generative model [13] or are trained according to a permutation-free loss [18].
In this paper, we consider a task upstream of speaker diarization. We suggest that the speaker diarization models could be further improved by locally refining the speaker embedding for each session. The speaker embedding models are usually designed to generalize well across a large range of speakers with different characteristics. However, speaker diarization is a simpler task compared with speaker recognition in the perspective of distinguishing speakers: we only care about separating several speakers for each session.
Our approach is to refine speaker embeddings by improving their performance for similarity measurement between speech segments. Obviously, if we could correctly predict for each pair of speech segments whether they belong to the same speaker or not, the diarization problem would be solved. Our method utilizes local structural information existing among the speaker embedding spaces for each session. The structural information is expressed by a graph built upon the speech segments for each session. The task of predicting similarities between pairs of nodes is analogous to the link prediction task on graphs. Our model can be seen as a neural link predictor [19,20,21] consisting of an encoding component and a scoring component. The encoding component includes GNN [22,23,24,25] layers to map speaker embeddings into another embedding space. Fully connected layers or cosine similarity is used as the scoring component. To train the model, a loss function is defined as the difference between the affinity matrix constructed by the refined embeddings and the ground-truth adjacency matrix. By minimizing this loss the model will be able to refine the original embeddings so that different speakers could be better separated within each session.
Very recently (and in parallel to this work), Lin et al. [26] proposed the use of LSTMs to improve similarity measurement for speaker diarization. Unlike their work, we utilize the structural information in the embedding space instead of utilizing the temporal information in label sequences. Also our model not only outputs better similarity measurement but also refined speaker embeddings, which could be potentially fed into nonclustering-based methods. Our model also achieves better performance with the same experimental setup. Experiments on both simulated and real meeting data show that the performance of speaker number detection and speaker diarization with the refined embeddings outperforms the original embeddings significantly, and our system achieves the state-of-the-art result on the NIST SRE 2000 CALLHOME database.

Building graphs of speech segments
A graph is built for each session using the pretrained speaker embeddings. Each node represents an audio segment which could be word-level, utterance-level, or extracted by a sliding window with a fixed sliding step. Speaker embedding of each segment is extracted as the node features. The weight of edges between nodes is represented by the PLDA scores or cosine similarities between the corresponding x-vectors and d-vectors, respectively. We only keep edges of weight larger than a threshold, which is treated as a hyperparameter.
Formally, each meeting session can be represented as a graph G(V, E, A), where V is the set of nodes (speech segments), E is the set of edges, and A ∈ N × N is the affinity otherwise. The matrix of d-vectors X ∈ R N ×D can be treated as node features for graph G, where N is the total number of segments and D is the dimension of the embedding space. The features of each node v i are represented by a d-vector x i , i ∈ 1, 2, . . . , N . The goal of speaker diarization can be for-mulated as the prediction of y i for each segment embedding x i such that y i is equal to y j if and only if x i and x j belong to the same speakers, i, j ∈ {1, 2, . . . , N }.

Graph neural networks
We apply several variants of GNNs falling into the framework of message passing neural networks (MPNNs) [25]. Under the MPNN framework, the convolutional operator is expressed as a message passing scheme: where x i is the feature of node i in the current layer with dimension D, x i is the node feature in the next layer with dimension D , e i,j is the edge feature from node i to node j, γ(·) and φ(·) are the update function and message function, respectively, parameterized by Θ, and denotes the aggregation function, e.g., sum, mean, max, etc.
One GNN variant we apply is the graph convolutional network (GCN) described by [24] in which the function corresponds to taking a certain weighted average of neighboring nodes. The updating scheme between two layers is: where L =D −1/2ÂD−1/2 is the normalized affinity matrix added by self connection,Â = A + I N ,D denotes the degree matrix ofÂ, W ∈ R D ×D is a layer-specific trainable weight matrix, and σ(·) is a nonlinear function.

Model design
Our model consists of two components: an encoding component and a scoring component. GNN layers are applied as the encoding component. No nonlinear functions are applied between the GNN layers. Because the speaker embeddings are already well trained, adding nonlinearity may lose information inside the original embeddings and result in worse results. The encoding component generates the refined embeddings, of which every pair are concatenated as inputs to the scoring component.
Due to the distinct nature of x-vectors and d-vectors, different scoring components are designed. For d-vectors, it is common to leave the complexity to the model and use a simple distance metric, e.g. cosine distance. Since the original d-vectors are trained for being comparable by a simple distance metric, we keep using this distance metric for scoring the refined embeddings. For x-vectors, on the other hand, an additional classifier, e.g. PLDA, is usually trained to measure the similarity. In our model, we add fully connected layers with nonlinear functions after the GNN layers, which is actually a classifier on top of the refined embeddings. This classifier is trained together with the GNN layers.

Loss function
We train the GNN model w.r.t. a loss function that is defined as the difference between the affinity matrix constructed by the refined embedding and the ground-truth adjacency matrix. This is actually a binary classification of linkage for all pairs of segments inside each session. For the x-vector based system, we simply applied the binary cross entropy (BCE) loss on all pairs of segments for each session.
However, this simple loss function does not work for dvectors. Due to the simplicity of the distance metric used, the gradient would be very noisy if training the model w.r.t. the exact match between the cosine similarities and ground-truth 1s or 0s. Instead we applied the following loss function: in which hist loss corresponds to the histogram loss [27], · nuclear is the nuclear norm between A and the ground-truth adjacency matrix A gt , and α is a scalar value hyperparameter. Histogram loss tries to ensure the distance between pairs of segments belonging to different speakers is larger than the distance between those belonging to the same speakers. The nuclear norm, which is the sum of singular values of a matrix [28] depends on the rank match between A and A gt , instead of an exact match. The combination of these two loss functions achieves similarity scores that stay in a reasonable range.

Spectral clustering
We use spectral clustering as our backend method and follow the standard steps described in [16]. However, to detect the speaker number, we first perform eigen-decomposition on the refined affinity matrix and then determine the speaker number by the number of eigenvalues that are larger than a threshold, which we treat as a tunable parameter. As shown in Section 4.2, this method outperforms the commonly used method based on maximal eigengap.

Datasets and implementation details
To compare our x-vector-based systems with other methods, we follow the evaluation steps described in the callhome diarization/v2 receipe of Kaldi [29]. We used the pretrained x-vector model and PLDA scoring model trained on augmented Switchboard and NIST SRE datasets [30]. Results on the commonly used dataset NIST SRE 2000 CALLHOME (LDC2001S97) Disk 8 are reported.
Our model architecture includes two GCN [24] layers followed by two fully connected layers. Input x-vector is 128dimensional. The dimension of output embedding from the GCN layers is 64, of which every pair is concatenated as a 128dimensional input to the following fully connected layers. The first fully connected layers is 64-dimensional with the ELU activation function [31] and the last layer has 1 dimension followed by a sigmoid function. The GCN layers are implemented with the PyTorch Geometric library [32].
Similar to previous work [13,26], we conducted 5-fold cross validation. The dataset contains 500 sessions in total, which is uniformly split into 5 subsets of 100 sessions for evaluation. In each turn, the model is trained for 50 epochs on 400 sessions with an initial learning rate of 0.001 reduced by a factor of 10 after 40 epochs. The optimal threshold for speaker number detection on the training set is used for evaluation on the testing set. A graph is built for each session on which we connect a pair of nodes if their PLDA score is higher than 0.2. The model is trained in a session-by-session fashion that each batch contains a single graph constructed from one session. All the hyperparameters are tuned on a validation set split from the training set for each turn. Results are shown in Table 1 and summarized in the following section.

Speaker diarization results
We compare speaker diarization error rate (DER) of our model with several recent works. We achieve state-of-the-art performance resulting in a DER of 7.24%, which outperforms both the baselines and all recent works. For fairness, we only include the results without system fusion of [26]. We only include the result with in-domain training of [13]. The DER of [13] with external data is 7.6% which is still worse than our result. Also the speaker embedding model of [13] was trained on a much larger dataset than the one we are using, which may also influence the results.

Datasets and implementation details
For d-vector-based system, we trained the model on simulated conversations and evaluated it on real meetings. The d-vector extraction model is trained and fake meetings are simulated from the VoxCeleb2 dataset [35]. For each simulated meeting, we randomly select 2-15 speakers from the training set of Vox-Celeb2 as the meeting participants. And for each speaker, 2-60 speech segments with duration of 1.5s are sampled. Moreover, we use 45 sessions of in-house real meetings as another testing set. On average, each meeting contains 6 speakers, ranging from 2 to 16. The average duration of meeting sessions is about 30 minutes.
To build the graphs, we only connect a pair of speech segments if the cosine similarity between their d-vectors is larger than a threshold 0.2, which is tuned on the validation set. Our model contains two GCN layers, of which the hidden dimension and output dimension are both equal to the input dimension of 128. We used 150 bins for the histogram loss. The DER reported only includes speaker confusion.

Speaker number detection results
We first evaluate the refined embedding for speaker number detection. We search for the optimal threshold for this task through simulated conversations on the validation set. The threshold which achieves the lowest mean speaker number detection error across all simulated sessions is chosen for evaluation on testing set. As shown in Figure 2, the minimal error achieved by the refined embeddings is lower than the original embeddings with a threshold of 3.0 and 2.0, respectively. The curved slope of the refined embeddings after the optimal threshold is lower than the original embeddings, which indicates that speaker number detection with the refined embedding is more robust. With the optimal threshold tuned on the validation set we evaluate the performance of speaker number detection on two test sets: fake conversations simulated from the testing set of VoxCeleb2 and an in-house meeting dataset, which are indomain and out-of-domain datasets respectively. As shown in Table 2, the performance with refined embeddings outperforms the original embeddings on both datasets with both the eigengap-based method [12] and the threshold-based method.
Intriguingly, while the threshold-based methods with original and refined embeddings outperform the eigengap-based methods on the in-domain dataset (VoxCeleb2 Test), the perfor-mance of the threshold-based method with original embeddings degrades on the out-of-domain (in-house meeting) dataset. This is likely because original embeddings are less robust with respect to the threshold, while the refined embeddings are robust even with a threshold tuned on a different dataset. Consequently the threshold-based method with refined embeddings significantly outperforms other approaches on both in-domain and out-of-domain datasets.

Speaker diarization results
We compare the performance of spectral clustering on speaker diarization using both original and refined embeddings. The results in Table 2 are consistent with the results on speaker number detection. Here again using the threshold-based model using the original embeddings actually performs worse than the eigengap-based method for the out-of-domain dataset. However the model using refined embeddings and the thresholdbased method for speaker number detection achieves a 73.7% relative DER reduction compared to the model with the original embeddings. This indicates that our system should be generalizable to real world applications.

CONCLUSION
In this paper, we present the first use of GNNs for speaker diarization. A graph of speech segments is built for each meeting session and a GNN model is trained to refine the speaker embedding utilizing the structural information in the graph. Our model achieves the state-of-the-art performance on a public dataset, while experiments on both simulated and real meetings show that spectral clustering based on the refined embeddings can achieve much better performance than original embeddings in terms of speaker number detection and clustering.

ACKNOWLEDGEMENT
The authors would like to thank Tianyan Zhou and Yong Zhao for providing the d-vector extraction model, and Chun-Hao Chang and Zhuo Chen for helpful discussions. Rudzicz is a CIFAR Chair in AI.