{'fraction_non_alphanumeric': 0.05592973759362035, 'fraction_numerical': 0.017565601594913518, 'mean_word_length': 4.634486945962355, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
CS-Rep: Making Speaker Verification Networks Embracing Re-parameterization

Automatic speaker verification (ASV) systems, which determine whether two speeches are from the same speaker, mainly focus on verification accuracy while ignoring inference speed. However, in real applications, both inference speed and verification accuracy are essential. This study proposes cross-sequential re-parameterization (CS-Rep), a novel topology re-parameterization strategy for multi-type networks, to increase the inference speed and verification accuracy of models. CS-Rep solves the problem that existing re-parameterization methods are unsuitable for typical ASV backbones. When a model applies CS-Rep, the training-period network utilizes a multi-branch topology to capture speaker information, whereas the inference-period model converts to a time-delay neural network (TDNN)-like plain backbone with stacked TDNN layers to achieve the fast inference speed. Based on CS-Rep, an improved TDNN with friendly test and deployment called Rep-TDNN is proposed. Compared with the state-of-the-art model ECAPA-TDNN, which is highly recognized in the industry, Rep-TDNN increases the actual inference speed by about 50% and reduces the EER by 10%. The code will be released.


INTRODUCTION
Automatic speaker verification (ASV) systems determine whether two speeches are from the same speaker or not. The front-end models of ASV systems to extract speaker embeddings from speech utterances to represent the corresponding speaker, such as i-vector [1], dvector [2], and x-vector [3]. The back-end functions of ASV systems calculate similarity scores between two embeddings, e.g., probabilistic linear discriminant analysis(PLDA) [4] and cosine similarity.
Currently, ASV systems are widely used in many real-world applications, such as criminal investigation, payment [5], and real-time meeting recordings [6]. In those realistic scenarios, the inference speed is becoming increasingly important.
Recently, the use of multi-branch topology (e.g., ResNet [7] has a convolutional branch and a shortcut branch) [8,9] to improve the theoretical efficiency and accuracy of models has been reported. These designs capture multi-scale information through multi-type Junhai Xu is the corresponding author.
Thanks to NSFC of China (No.61876131, No. U1936102), Key R&D Program of Tianjin (No.19ZXZNGX00030). Regional Innovation Cooperation Project of Sichuan (Grant No.22QYCX0082) convolution kernels with a small number of channels. Additionally, many studies [10,11] have used multi-branch designs to strengthen the time-delay neural network [3](TDNN). ECAPA-TDNN [10] with multi-branch topology has exhibited excellent performance in recent ASV competitions [12,13,14,15,16]. Unfortunately, the actual inference speed of the multi-branch designs is typically much slower than its theoretical speed. The main reason is that the multi-branch designs increase the memory access cost (MAC) and are unfriendly for parallel computing [17]. Therefore, this topology does not solve the contradiction of speed and performance in the real applications.
To improve the actual inference speed for the multi-branch models, Ding et al. [18] proposed a re-parameterization approach to convert the multi-branch network to a plain topology in the inferenceperiod. However, their strategy is based on the structure of "convbn-activation," but the mainstream TDNN-based ASV models rely on the "conv-activation-bn" structure to realize good verification accuracy (we confirm that this is reasonable in Section 4.2). Many successful implementations of ASV have adopted the structure, such as the x-vector [3] (implemented by Kaldi [19]), E-TDNN [20], and ECAPA-TDNN [10], etc. In those ASV models, since their convolutional layer is not adjacent to the batch normalization (BN) [21] layer, they cannot be re-weighted through Ding's method.
In this study, we propose the cross-sequential re-parameterization (CS-Rep, Fig. 1) to increase the inference speed and verification accuracy of ASV systems, especially for the models based on the "conv-activation-bn" 1 architecture. Specifically, we build a multibranch network with the "conv-activation-bn" structure during the training period. In the inference period, CS-Rep adjusts the order of modules of the sequential layer to the "bn-conv-activation" losslessly, causing the BN to be adjacent to the convolutional layer. However, the "bn-first" 2 case will generate influences on each channel of the convolutional layer. We propose the "bn-first" reparameterization method to model these influences by the discrete convolution operator. Then, CS-Rep adopts this approach to convert the multi-branch network to a TDNN-like plain topology while maintaining the multi-scale information capturing ability. Therefore, the model adopted CS-Rep achieves an efficient architecture with friendly parallel computing.
The contributions of this study are listed as follows.
(1) We pro- 1 The sequential layer [22] is a layer including multiple modules by order. For example, the "conv-activation-bn" is a sequential layer with the order of 1-D (TDNN) or 2-D convolutional layer, activation function, and BN layer. Fig. 1 (a) shows the diagram of the sequential layer with three branches. 2 The "bn-first" means the case that the BN layer precedes the convolutional layer in the sequential layer. e.g., the "bn-conv-activation." pose CS-Rep to increase the inference speed, decrease the parameters, and reduce the floating-point operations (FLOPs) of networks with the "conv-activation-bn" structure while maintaining the accuracy.
(2) We also combine the "bn-first" and "conv-first" [18] reparameterization as an out-of-the-box method, which can be conveniently used for all types of sequential layers ("conv-first" and "bnfirst") to improve the inference speed. (3) Based on CS-Rep, we construct Rep-TDNN. It achieves the state-of-the-art performance (1.09% EER on VoxCeleb1-test, without any data augmentation), and its inference speed is 50% faster than the state-of-the-art model ECAPA-TDNN 3 . The remainder of the paper is organized as follows. Related works are briefly introduced in Section 2, the CS-Rep method and the novel Rep-TDNN model are detailed in Section 3, experimental results are presented and analyzed in Section 4, and the conclusions are given in Section 5.

RELATED WORK
Time-delay neural networks: The TDNN-based x-vector [3] system extracts robust embeddings by capturing the long-term temporal relations of utterances through the one-dimensional convolutional layer [23]. Several improved versions of TDNN have been proposed, such as E-TDNN [24] with stable performance and satisfactory inference speed, and ECAPA-TDNN [10] with compelling accuracy. Squeeze-excitation block: Squeeze-excitation (SE) block [25] has been proved successful in modeling global channel interdependencies in speech, bringing improved accuracy to ASV [10]. Multi-branch topology: Multi-branch topology [8,9] involves multiple branches of different types in one sequential layer, so it helps to capture important information from multiple scales. In ASV, many solid models have adopted it, such as InceptionNet [26], Res2Net [9], ECAPA-TDNN [10], and HS-Net [11]. However, multi-branch topology increases the complexity of network topology, leading to negative impacts on the inference speed. RepVGG: RepVGG [18] proposed a structural re-parameterization technique to improve the performance and reduce the computational complexity during inference. Their model applied the plain topology in the inference period and the multi-branch structure in the training period. But there are two limitations in their method: (1) the re-parameterization method proposed in RepVGG is not suitable for ASV because the approach bases on the "conv-bnactivation" structure, but mainstream ASV models adopt the "convactivation-bn" structure; (2) their work lacks the condition of the re-parameterization in TDNNs.

PROPOSED METHODS
This section will describe the proposed CS-Rep strategy through a three-branch TDNN. CS-Rep will be used to develop Rep-TDNN with an extremely efficient architecture and high performance.

Multi-branch training structure
The three-branch structure [18] is introduced to build a TDNN block with excellent speaker information extraction. In our TDNN block, one sequential layer consists of three branches ( Fig. 1 (a)): a TDNN branch T DN N (3) with context = 3, a TDNN branch T DN N (1) with context = 1, and a shortcut connection, respectively. The operation within one sequential layer can be defined as: where Mj∈R B×N×T is the output matrix of the j-th sequential layer, B is the mini-batch size, N is the number of the feature dimension, T is the number of frames, ReLU is the ReLU function, and BNj is the batch normalization (BN) layer of the j-th sequential layer.

Cross sequential re-parameterization method
This subsection describes how to convert a multi-branch network based on the "conv-activation-bn" structure to a plain network using the cross sequential re-parameterization function (CS-Rep). Fig. 1 and Fig. 2 display the topology diagram and parameter diagram for the network during the processing of CS-Rep. The four steps of the approach are as follows: Step 1: Cross-sequential converting. We distribute BNj−1 from sequential-layerj−1 to the head of the branches of sequentiallayerj, causing the model with the "conv-activation-bn" structure to the "bn-conv-activation" structure. This transform leads to the BN and TDNN layers contiguous and allowing them to be merged. This strategy is called cross-sequential transformation (CST). The diagram of CST is displayed in Fig. 1 (b). After utilizing CST, the sequential layer of Eq. (1) can be transformed to   Then we define that W (C) j ∈ R No×N i ×C is the weight of TDNN (TDNN j ) from the j-th sequential layer, Ni and No are the numbers of input and output channels respectively, C is the number of the context of TDNN, and " * " is the discrete convolution operator. Eq.
(2) can be re-written as: Step 2: Merge BNj−1 into T DN Nj. The BN [21]in TDNN is where no ∈ {1, 2, ..., No} denotes the no-th output channel, and µ µ µ, σ σ σ, γ γ γ, and β β β represent the mean of mini-batch, the variance of mini-batch, and the scale and shift parameters, respectively. And µn o , σn o , γn o , and βn o are the elements of vectors µ µ µ, σ σ σ, γ γ γ, and β β β, respectively. In the "bn-first" (BN is before the TDNN operator) re-parameterization, the condition that BN is before the convolutional layer will generate influences on each convolutional layer channel. The influence can be calculated as the bias of the convolutional layer through the discrete convolution operator. The "bn-first" re-parameterization converts every TDNN branch and its preceding BN into a TDNN (W) with a bias vector (b) to reduce the depth of network ( Fig. 1 (c)), and it can be defined as: where j denotes the j-th sequential layer, W j and b j are the weight and bias of the new TDNN branch through the "bn-first" re-weight, and W j,(no,:,:) is tensor slice of W j and b j,(no) is the element of b j . Fig. 2 (b) displays the weight diagram. Then, we generalize a hot-swap algorithm for the "bn-first" (Eq. (5)) and "conv-first" [18] re-parameterization methods. The algorithm can be written as:  For the re-weight of the identity branch, it can be converted to a TDNN branch with context = 1 (W id ): where w id no,n i ,: is the tensor slice of W id , Ni and No are the numbers of input and output channels of TDNN, and ni ∈ {1, 2, ..., Ni} and no ∈ {1, 2, ..., No}. The diagram of building a TDNN layer equivalent to the shortcut connection is shown in the gray branch of Fig. 2(b). After converting the shortcut branch to the TDNN branch, it can be re-weighted by the "bn-first" re-parameterization approach.
Step 3: Pad the weight of branches to the same dimension. As for the TDNN branches with context = 1, we can use zeropadding on its weight to pad to context = 3 (Fig. 2 (c)). After step 3 ( Fig. 1 (c)), the sequential-layerj is where W (1→3) j means that the TDNN weight is changed from context = 1 to context = 3 through zero-padding. b j is the bias of the corresponding branch, and the superscript of b j is only used to distinguish the branch uniformly while it does not mean padding.
Step 4: Merge all branches to one TDNN layer. Based on the linear additivity of the discrete convolution operator [18], the weight and bias of every branch can be added to one branch to implement the merge-branch-convert easily. The final model only constructed by TDNN and ReLU (Fig.1(d) and Fig.2(d)),which can be written as: ...

Proposed model
Based on CS-Rep in Section 3.2, we propose an improved TDNN called Rep-TDNN, which has excellent performance, fast inference speed, few parameters, and fewer floating-point operations (FLOPs). The overview of Rep-TDNN is displayed in Fig. 3. The frame-level model consists of four blocks, including TDNN layers, LeakyReLU activation functions, BN, and SE-Block.
In the training period ( Fig. 3 (right)), each block has one head TDNN layer (the first TDNN layer in the gray box) that integrates feature information and four TDNN sequential layers with three branches (a branch with context = 3, a branch with context = 1, and a shortcut branch; describes in Section 3.1) with the "convactivation-bn" structure. The context C head of each head TDNN layer is [5,1,1,5]. SE-block is adopted in each block to model the channel attention. Statistics pooling aggregates frame-level features to segment-level features passed to two fully connected (FC) layers. AAM-Softmax [27] loss is to choose for loss function.
In the inference period, the CS-Rep method is adopted to convert the model from the multi-branch topology to the plain topology ( Fig.  3 (left)). The parameters and FLOPs of Rep-TDNN (CS-Rep) are only 6.9e+6 and 1.4e+9, which are the same as for E-TDNN [10,23].

Experimental details
Dataset: To investigate the performance of the models, we created a benchmark on the VoxCeleb [28,29] dataset. We implemented some successful architectures, including E-TDNN [23], ResNet34 [10], and ECAPA-TDNN [10], as our baselines. Models were trained on the VoxCeleb2 development sets without data augmentation, and tested on the VoxCeleb1 test set, VoxCeleb1-E, and VoxCeleb1-H [29]. 161-dimensional spectrograms were generated through hamming sliding window with a width of 20 ms and a step of 10 ms. The cepstral mean and variance normalization (CMVN) was applied. No voice activity detection was adopted. Implementation details: Models were trained by AAM-Softmax loss with m = 0.25 and s = 30. In the training stage, the minibatch size of 64 was used. Each training speech sample was randomly sampled for 300 frames from recordings. Stochastic gradient descent (SGD) with momentum 0.9, weight decay 1e-5, and initial learning rate 0.1 was utilized. Full-length utterances were adopted in the testing stage to extract embeddings. The GPU type and CUDA version were NVIDIA 2080Ti and 10.2. Adaptive score normalization (AS-Norm) [30] with cosine similarity was applied for all models. L2-normalized speaker embeddings of each training speaker were chosen as the imposter cohort with a size of 1000. Evaluation metrics: We utilized the equal error rate (EER) and minimum detection cost function (minDCF) from NIST SRE10 [31] as the performance metrics. Moreover, we adopted the actual inference speed (frame-numbers/processing-time) to compare the speed of different models. It is more objective than the indirect speed metrics parameters and FLOPs, which can avoid the bad case in which a model with few parameters and complex topology has a low FLOPs but the actual inference speed may be slow [17].

Experimental results
The results of the state-of-the-art baseline on VoxCeleb1-test, VoxCeleb1-E, and VoxCeleb1-H are shown in Table 1. The results of the inference speed are the average of 5 runs.
In Table 1, the E-TDNN with the "conv-activation-bn" structure obtained approximately 1.6%, 1.6%, and 2.8% EER on the above three evaluation sets, respectively, whereas the E-TDNN with the "conv-bn-activation" structure only achieved 2.1%, 1.9%, and 3.4% EER, respectively. These results confirm that the "conv-activationbn" architecture is necessary for high-accuracy ASV networks.
Compared with the best E-TDNN, the EER results of Rep-TDNN were much lower, relatively reduced by 32.6%, 24.3%, and 18.0% on three evaluation conditions, respectively. Satisfactory performances showed the strong discrimination of speakers for Rep-TDNN. As shown in Table 1, Rep-TDNN obtained approximately 1.2% EER and 2.2% EER on VoxCeleb1-E and VoxCeleb1-H, respectively, which are both hard evaluation test sets. Compared with the results with ECAPA-TDNN, Rep-TDNN had 10.7% and 11.9% reductions of EER on two hard test sets and a 47.9% increase of inference speed. We can find that compared to ECAPA-TDNN, Rep-TDNN was more efficient and more robust in hard conditions. Furthermore, CS-Rep can increase the inference speed for the model with negligible influence on the accuracy. Compared with the Rep-TDNN without CS-Rep, the inference speed of the model with CS-Rep was greatly improved (by 58%) while EERs stayed the same. In Table 1, Rep-TDNN through CS-Rep can process 92,903 frames per second, whereas Rep-TDNN with original topology only can process 58,877 frames per second. The results show that CS-Rep is an essential component for developing ASV models with the fast inference speed and high accuracy.
Generally, Rep-TDNN showed relatively improvements of approximately 50% in inference speed and 10% in performance compared with state-of-the-art ASV models. Hence, it has great potential for the industry, saving spends for test and deployment.

CONCLUSION
This study presents an upgraded re-parameterization approach called CS-Rep to re-parameterize models with a multi-branch design more efficiently. It can be used to reduce the size of parameters, reduce inference time, and improve the performance of ASV systems. We also integrate the "bn-first" and "conv-first" of re-parameterization equations into a plug-and-play module. Based on CS-Rep, this work proposes Rep-TDNN, which obtains 18% -32.6% relative reduction in EER compared with solid baselines. The positive results show that Rep-TDNN reduces the inference time by approximately 50% and improves the performance by approximately 10% compared with the state-of-the-art model ECAPA-TDNN.