{'fraction_non_alphanumeric': 0.035874213836477986, 'fraction_numerical': 0.013735849056603773, 'mean_word_length': 4.59729653618699, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Token-level Speaker Change Detection Using Speaker Difference and Speech Content via Continuous Integrate-and-fire

In multi-talker scenarios such as meetings and conversations, speech processing systems are usually required to segment the audio and then transcribe each segmentation. These two stages are addressed separately by speaker change detection (SCD) and automatic speech recognition (ASR). Most previous SCD systems rely solely on speaker information and ignore the importance of speech content. In this paper, we propose a novel SCD system that considers both cues of speaker difference and speech content. These two cues are converted into token-level representations by the continuous integrate-and-fire (CIF) mechanism and then combined for detecting speaker changes on the token acoustic boundaries. We evaluate the performance of our approach on a public real-recorded meeting dataset, AISHELL-4. The experiment results show that our method outperforms a competitive frame-level baseline system by 2.45% equal coverage-purity (ECP). In addition, we demonstrate the importance of speech content and speaker difference to the SCD task, and the advantages of conducting SCD on the token acoustic boundaries compared with conducting SCD frame by frame.


Introduction
Speaker change detection (SCD) is the task of locating the time points when a different speaker starts to speak. The SCD system is often applied as a submodule of speaker diarization, which determines "who spoke when" or is used as a front-end of automatic speech recognition (ASR). The performance of SCD will greatly affect the subsequent models.
Many different SCD systems have been proposed in the literatures. The distance-based methods [1][2][3][4][5] are studied earliest, which calculate the distance between features in the adjacent windows, and once the distance exceeds the threshold, a speaker change is detected. In addition, the model-based methods [6][7][8][9] segment the input audio into fix-length segments that are assumed to contain only one speaker, and models are trained to extract speaker embeddings for each segment. Then the distance between adjacent speaker embeddings is used to decide whether speaker change happens. More recently, some end-to-end SCD systems [10][11][12][13] are proposed to predict the speaker change at the end of the neural network without relying on a distance metric. Yin  which predicts each frame as a change point or not [10]. Leda et al. extract speaker embeddings for adjacent segments with a shared siamese network, then a fully connected (FC) classifier decides whether there is a speaker change between these two segments [11]. Marek et al. propose a Convolutional Neural Networks (CNN) based system to the SCD task [13].
The SCD systems mentioned above only rely on acoustic information such as pitch [14], speaker embeddings [11], and silence [15]. However, the speech content is another critical cue for the SCD task [16]. There has been little effort in exploiting speech content for the SCD task. Meng et al. formulate text-based SCD as a binary sentence-pair classification problem, which judges whether the speaker is changing between consecutive sentence pairs [17]. Jin et al. investigate the way to integrate content and acoustic cues with sequence to sequence model to improve speaker diarization performance [18]. India et al. prove that the combination of content and acoustic features outperforms the cosine distance-based baseline where only acoustic data is used [19]. Anidjar et al. propose a hybrid framework for the SCD problem that is learned by content information and speech signals [20]. Most of the existing methods combining speaker and content information rely on the acoustic boundaries from the manual annotation or the forced-alignment results produced by the ASR, which leads to complex processes and time-consuming annotations.
In this paper, we propose an end-to-end SCD system that combines speech content and speaker difference to detect speaker changes between tokens. We construct a joint model for ASR, speaker identification (SID) and SCD. The ASR part follows the continuous integrate-and-fire (CIF) based encoderdecoder [21] structure, which can generate speech representations and token acoustic boundaries. The SID part uses the located token acoustic boundaries by the CIF to generate tokenlevel speaker embeddings. The SCD part calculates the speaker difference between the token-level speaker embeddings and then combines speaker difference with speech content to detect speaker change. We train and evaluate the proposed model on a public real-recorded dataset, AISHELL-4. The experiment results of our method show a performance improvement compared to a competitive open-sourced frame-level baseline. Further experiments demonstrate the advantages of using speaker difference and speech content for the SCD task.

CIF-based ASR
The continuous integrate-and-fire (CIF) based ASR follows an encoder-decoder framework connected by the CIF mechanism, a soft and monotonic alignment [21]. The blue part in Fig.1 illustrates the structure of the CIF-based ASR. The encoder trans- forms the acoustic feature sequence x = (x1, x2, ..., xT ) to a encoded output h = (h1, h2, ..., hU ), where T and U are the total length of the input sequence and the encoded output, respectively. A weight estimator calculates the information weights α = (α1, α2, ..., αU ) for h at each time step. αu scales the amount of information contained in hu. In the training stage, a scaling operation is applied to the raw weights α to ensure that the sum of the weights α is equal to the length of targetsȳ = (ȳ1,ȳ2, ...,ȳS).
The CIF processes the frame-level speech representations h and information weights α in a left-to-right manner. The information weight α u is accumulated to α a u . Meanwhile, the tokenlevel speech representations are integrated frame by frame.
Once the accumulated weight α a u reaches a threshold β, a token-level acoustic boundary is located, and these time steps are called fired time steps, denoted as u * . At each fired time step, the information weight αu * is divided into two parts, αu1 and αu2. αu1 is used for completing the current integration of token-level representation ci. The sum of information weights consumed in a complete integration is 1. αu2 remains for the next integration. A complete integration spans from the last fired time step to the current fired time step.
After firing the current integration results ci (Eq. 6), the accumulated weight and token-level representation state will be reset as follows: The process described from Eq. 2 to Eq. 8 will be repeated until the last time step U . Then the CIF finishes transferring the encoded output h into token-level representations c = (c1, c2, ..., cS), and these fired time steps are located as token acoustic boundaries. Since we apply the scaling operation in Eq. 1 and the threshold β is set to 1, the length of c is equal to the length of target labelsȳ. For the convenience of the following description, we summarize the process of integration and fire formulated from Eq. 1 to Eq. 8 as follow: The decoder receives the fired token-level representations c and predicts the label sequenceȳ = (ȳ1,ȳ2, ...,ȳS) in an autoregressive manner. Pr During inference, the scaling operation in Eq. 1 is skipped. To alleviate this mismatch between training and inference, the quantity loss and tail handling are adapted. For more details of the CIF-based ASR, we refer readers to [21].

Overview
In this paper, we propose to use speaker difference and speech content to conduct a token-level speaker change detection (SCD). The model body is a joint framework for automatic speech recognition (ASR), speaker identification (SID) and SCD. The three parts are distinguished by different colors in Fig. 1. The ASR part transfers the input feature sequence x to the target token sequence y while providing the token acoustic boundaries contained in the information weights α and speech representations c and o. The SID part conducts a token-level speaker classification, and then generates token-level speaker embeddings m. The SCD part captures the speaker difference d between the speaker embeddings m, and combines it with the speech content l to detect speaker changes between tokens.

Model Structure
The details of the model body is shown in Fig.1. Given the input feature sequence x = (x1, x2, ..., xT ), the CIF-based ASR predicts the token sequence as described in Section 2.
The SID consists of a speaker encoder, a classifier, and the CIF module. The speaker encoder transfers x to frame-level speaker representations z = (z1, z2, ..., zU ), and it has the same temporal downsampling as the ASR encoder to make the length of their output sequence consistent. Then the CIF mechanism uses the frame-level speaker representations z and the information weights α provided by the ASR part to generate token-level speaker representations e. e = (e1, e2, ..., eS) = CIF(z, α) In the calculation process of the CIF, the fired time steps and the length of output sequence S are dependent on the information weights α, so the token-level speaker representations and speech representations can be one-by-one correspondence. The speaker classifier stacks two full-connected (FC) layers. The first layer transfers e to the token-level speaker embeddings m = (m1, m2, ..., mS). The second layer projects m to the speaker probability distribution v = (v1, v2, ..., vS).
The SCD part consists of a speech content extractor (SCE), a speaker difference extractor (SDE), and a jointer. The SCE stacks a FC layer and uni-directional transformer layers with masked multi-head self-attention [22]. It pre-processes the speech presentations from the CIF output c and ASR decoder output o, and generates the speech content l = (l1, l2, ..., lS). The SDE stacks a 1d-convolution layer and a feed-forward network (FFN) layer to calculate the speaker difference d = (d1, d2, ..., dS) between the token-level speaker embeddings m. Using 1D convolution with kernel size 3, the SDE calculates the difference between the speaker embeddings of the last token, the current token and the next token. Then the speaker difference and the speech content are concatenated and fed into the jointer. The FC layer based jointer combines these two cues to predict speaker change probability p = (p1, p2, ..., pS).

Training
Before the joint training of the proposed model, we pre-train the CIF-based ASR part and speaker encoder, respectively. The loss used in the pre-training of ASR is consistent with our previous work [21]. The pre-training of the speaker encoder is an utterance-level speaker classification. We add an extra average pooling and an output FC layer upon the speaker encoder to perform the pre-training. The AMSoftmax [23] loss is used to capture more discriminative speaker embeddings.
During the joint training stage, the parameters of ASR part are frozen. The loss is the interpolation of a token-level AM-Softmax loss for speaker classification and a token-level binary cross-entropy (BCE) loss for speaker changes detection. Ljoint = AMS(v,v) + BCE(p,p) (12) wherev andp are token-level speaker identity label and speaker change label, respectively.

AISHELL-4 [24]
is a public real-recorded conversation speech dataset in conference scenario. It contains 118 hours of audio recorded by an 8-channel microphone array. All our experiments only use the first channel. Each of the total 211 meeting sessions contains 4 to 8 speakers. As the speech recognition for the overlapped speech is beyond the scope of this paper, we exclude the overlapped speech of the AISHELL-4. The excluded rules are that (1) one to four consecutive intervals annotated in the original annotation file form a sentence. (2) these sentences containing overlapped speech for more than 1s will be discarded. (3) these sentences containing overlapped speech that accounts for more than 10% of the duration of any interval will be discarded. (4) these sentences containing more than 10s silence will be discarded. The training and test set are processed independently. Then we extract five meeting sessions as the development set from the original training set, and the remainder is used as our training set. The test set is further excluded these sentences containing no speaker change. The details of the processed data are shown in Table 1. In order to facilitate readers to reproduce our work, we release the data processing scripts 1 .

Evaluating Metric
All our experiments are evaluated on the equal coverage-purity (ECP). The formula of the purity and the coverage [25] are as follows: where R and H are the set of segments cut by the reference and hypothesized change points, respectively. These points exceed a threshold θ are detected as speaker changes. |s| is the duration of segment s. r ∩ h is the intersection of segments r and h.

Experimental Setup
We extract input features using the same setup as [21]. A convolutional layer with 1/2 temporal downsampling is used as the front-end of the ASR, and the filter number is set to 64.
The ASR encoder consists of 15 Conformer [26] encoder layers with 8 attention heads, 400 attention dimensions and 1600 FFN dimensions. There is a max-pooling layer for 1/2 temporal downsampling after the fifth and tenth layers, respectively. The weight estimator consists of a 1-dimensional convolutional layer and a FC layer. The kernel size of the convolutional layer is set to 3, and the filter number is 400. The FC layer has one output unit with sigmoid activation. The ASR decoder consists of 2 Transformer [21] layers with 8 attention heads, 400 attention dimensions and 1600 FFN dimensions. The speaker encoder has the same architecture as the 18-layer ResNet in [27] except for not the final average pooling layer and halving the channels of all the convolutional layers. Both the speaker encoder and the ASR encoder have a 1/8 temporal downsampling. The speaker decoder stacks two FC layers. The first layer has 128 units with ReLU activation, the second layer has 36 (total speaker number in the training set) output units with softmax nonlinearity. The speech content extractor consists of an FC layer and transformer layers. The FC layer projects the inputs to 400 dimensions. The structure of transformer layers is consistent with the ASR decoder. The speaker difference extractor consists of a 1d-convolutional layer with kernel size 3 and an FFN layer with 512 hidden dimensions. The jointer consists of a hidden FC layer with 528 units and an output FC layer with 1 unit. The threshold β in these two CIF is set to 1.
For the pre-training of ASR, the learning rate warms up for the first 1k iterations to a peak of 10 −3 and holds on for the next 40k iterations, and then linearly decays for the remainder. For the pre-training of the speaker encoder, the SGD optimizer is used. The learning rate warms up for the first 1k iterations to a peak of 10 −4 and holds on for the next 5k iterations. During the joint training stage, the learning rate warms up for the first 1k iterations to a peak of 10 −3 , and holds on for the remainder.

Baseline results
We build two baseline systems with the Pyannote1.1 toolkit [28]. The baseline system A1 with Bi-LSTM follows the structure in [10], which addresses the SCD as a binary sequence labeling (BSL) task. The input MFCC feature sequence is encoded by the Bi-LSTM. Then a multi-layer perceptron (MLP) projects the encoded sequence to a frame-level score sequence between 0 and 1. For a fair comparison, the baseline A2 replaces the Bi-LSTM with the ResNet18, the structure used in our speaker encoder. The results of these two baseline systems are shown in the A1 and A2 of Table 2.

Evaluation with token-level speaker difference
The B1 to B4 in Table 2 are four variants of the proposed model. All the four experiments ablate the use of the speech content. B1 feeds the frame-level speaker representations z into the SDE, and skips the convolutional layer in the SDE. Compared with B1, B2 uses the convolutional layer in the SDE to capture the speaker difference. In contrast, B3 and B4 feed the token-level speaker embeddings m into the SDE, and then the jointer detects speaker changes on the acoustic boundaries. Comparing B2 with B1, we find that using convolution in the SDE to capture the speaker difference between the framelevel speaker representations z brings little improvement. Comparing B4 with B3, both of which use the token-level speaker embeddings m, the use of speaker difference brings a 5.52% ECP improvement. These results indicate that calculating the speaker difference between token-level speaker embeddings is more useful to the SCD task. Moreover, compared with the frame-by-frame speaker change detection, detecting speaker changes on the token acoustic boundaries can greatly reduce the number of candidates to be processed. From the results of B1 to B4, we could find that the token-level speaker difference information (B4) obtains the best performance. Thus, all subsequent experiments use token-level speaker difference for the token-level SCD.

SCD with the speaker difference and the speech content
Another advantage of using token-level speaker difference is the ability to combine it with token-level speech content at the  Figure 2: Coverage-purity measured on baseline methods and our method with speaker difference and speech content.
same granularity. B6 in Table 2 shows the result of the proposed method with both token-level speaker difference and speech content. The results of B4 and B5 represent that our method ablates speech content or speaker difference, respectively. We find that using one of the speaker difference or speech content in our method has outperformed these two baseline systems (A1 and A2), and B6 shows that combining these two cues can bring further performance improvement, and outperforms the baseline system with ResNet18 by 2.45% ECP.
In Fig 2, we plot the coverage-purity curves of B4, B5, B6 and these two baseline systems (A1 and A2). All curves are obtained by varying the threshold θ. The dashed line is diagonal, where purity is equal to coverage. Among the five systems, area under the curve of B6 is the largest, and B6 performs better in most of the varing thresholds.

Evaluation with different speaker context settings
In the previous experiments, all convolutional layers in the SDE have the context of [−1, 1], which means that the model refers to the speaker embedding of the previous token and the next token when determining whether there is a speaker change between current token and next token. We then attempt to improve the proposed method by more speaker contexts. The results of our method with different speaker contexts are shown from C1 to C3 in Table 2. The best ECP performance is achieved by the [−1, 1] context. After our analysis of the corpus, we find a lot of speaker change points where one person says only one token inside another person's speech. For such cases, [−1, 1] context is enough for the SCD, and more contexts bring confusion.

Conclusions
In this paper, we propose to use both speaker difference and speech content to detect speaker changes on the located token acoustic boundaries by the CIF. The experiments demonstrate the advantages of using the speaker difference and the speech content for the SCD task, and show performance improvement of detecting speaker changes on the token acoustic boundaries. Specifically, our token-level SCD system using both speaker difference and speech content outperforms the competitive frame-level baseline by 2.45% ECP. In the future, using the CIF to integrate the other information contained in speech with content for more speech processing tasks will be a potential research direction.