{'fraction_non_alphanumeric': 0.04419748355368206, 'fraction_numerical': 0.01903302037427349, 'mean_word_length': 4.901997738409348, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Robust Text-Dependent Speaker Verification via Character-Level Information Preservation for the SdSV Challenge 2020

This paper describes our submission to Task 1 of the Short-duration Speaker Verification (SdSV) challenge 2020. Task 1 is a text-dependent speaker verification task, where both the speaker and phrase are required to be verified. The submitted systems were composed of TDNN-based and ResNet-based front-end architectures, in which the frame-level features were aggregated with various pooling methods (e.g., statistical, self-attentive, ghostVLAD pooling). Although the conventional pooling methods provide embeddings with a sufficient amount of speaker-dependent information, our experiments show that these embeddings often lack phrase-dependent information. To mitigate this problem, we propose a new pooling and score compensation methods that leverage a CTC-based automatic speech recognition (ASR) model for taking the lexical content into account. Both methods showed improvement over the conventional techniques, and the best performance was achieved by fusing all the experimented systems, which showed 0.0785% MinDCF and 2.23% EER on the challenge's evaluation subset.


Introduction
This paper presents our submission to Task 1 of the Shortduration Speaker Verification (SdSV) Challenge 2020. The main purpose of this challenge is to evaluate new techniques for speaker verification in a short duration scenario [1]. The evaluation dataset used for the SdSV Challenge 2020 was derived from the multi-purpose DeepMine dataset [2,3]. We submitted the systems to Task 1 of the SdSV Challenge 2020, which was focused on text-dependent speaker verification (TD-SV).
During the past decade, there has been significant improvement in the field of text-independent speaker verification (TI-SV), which is mainly attributed to the development of deep neural network (DNN) based speaker embedding. These mechanisms have been developed through a variety of methods such as deeper architectures [4,5,6,7], pooling strategies [6,8,9,10], and different objective functions [11,12,13,14,15,16]. However, these techniques are optimized to discriminate only the speaker and may be ineffective in the TD-SV task in which the lexical context, as well as the speaker, is considered [17].
On the other hand, there have also been various efforts to boost performance in the TD-SV. In [18], Larcher et al. used an HMM-based system named HiLAM to model each speaker and each senone state. H. Zeinali et al. [17] proposed a straightforward HMM-based extension of the i-vector approach [19], which allows i-vectors to contain sufficient text-dependent information. In [20], Y. Lei et al. used DNN to estimate the posteriors of the frames for calculation of sufficient statistics and E. Variani et al. [21] extracted frame-level representations termed d-vector through a hidden layer of DNN for the TD-SV task. In [22], Matejka et al. employed bottle-neck DNN features concatenated to other acoustic features to improve the performance, and Zhang et al. [23] proposed an attention aggregation-based end-to-end TD-SV system which takes the speaker and phonetic information into account.
In this paper, we focus on preserving the character-level information. Overall, our contributions are as follows: • Character-level pooling: We introduce the aggregation method using the estimated frame-level posterior obtained from an automatic speech recognition (ASR) model. Experiments show that this method is valid and effective in the TD-SV through the results on the challenge's progress and evaluation subsets. • Score compensation: We propose the score compensation method where the probability of pass-phrase is estimated. Our experiments show that the usage of the proposed score compensation significantly enhances performance in the TD-SV even if the embedding network is trained only to classify the speaker. Furthermore, the fusion of different systems we employed produces the best performance on the challenge's trial subsets.
The rest of this paper is organized as follows: Section 2 describes all components of our systems and Section 3 presents the experimental conditions and results on the challenge's trial subsets we submitted. Finally, we conclude in section 4.

System components description 2.1. Front-end
In our systems, we used two types of front-end networks: TDNN-based [6] and ResNet-based [5] architectures.
TDNN-based architecture. The configuration of TDNN-based systems is shown in Table 1. The usage for each system is described in Section 2.2 and 2.6. The input acoustic feature used in this architectures was log Mel-filterbank energies calculated from 20ms windows with a 10ms hop size and extracted via the Librosa toolkit [24]. We selected 512 and 580 speaker embedding dimensions for statistics pooling and character-level pooling respectively. Our implementation and speaker embedding network training was done using Tensorflow toolkit [25].
ResNet-based architecture. We used Thin ResNet34 architecture recently proposed in [5] (Table 2). Compared to the original ResNet [4], it has only a quarter of channels in each residual block. In this architecture, we used 257-dimensional short-time Fourier transform (STFT) with 200-300 frames crop as input acoustic feature and chose 512 dimensions for embedding. For implementation, we used Pytorch toolkit [26] and developed the systems based on the architectures in [15].

Pooling methods
In the TI-SV task, various pooling mechanisms have been proposed such as statistics pooling [6], self-attentive pooling [8], learnable dictionary encoding (LDE) pooling [9], mutual information neural estimate (MINE) based pooling [10]. In this work, we employed a variety of pooling methods proposed in the TI-SV task. On top of that, we propose a pooling strategy suitable for the TD-SV, i.e., character-level pooling, which leverages a frame-level probability distribution of each character estimated from the end-to-end based ASR model. The pooling methods we used are as follows: • Statistics Pooling (SP) [6] • Self-Attentive Pooling (SAP) [8] • GhostVLAD Pooling (GVP) [9] • Character-Level Pooling (CLP) Character-level pooling. To extract the utterance-level representation appropriate for the TD-SV, we exploit the character posterior probabilities of each frame-level feature. The probability of a character given a frame-level feature, i.e., the posterior, is denoted by: where the set C = {c k | c k is k th character, 1 ≤ k ≤ K}, and hi is the i th frame-level feature with D1 dimensions where 1 ≤ i ≤ T . K indicates the number of symbols in the character set, and T is the number of segment frames. To estimate π k,i , we leveraged the decoder outputs of the end-to-end based ASR model, termed Jasper, proposed by [27]. Since this model was trained by using the Connectionist Temporal Classification (CTC) loss, our character set consisted of a total of 29 symbols including all alphabets (a-z), the space symbol, the apostrophe symbol and the blank symbol used by the CTC loss. Then, the aggregation for character-level representation is as follows: where τ is a constant added to avoid divergence. All the character-level representations are concatenated as v, and then it's passed through the locally-connected layer, which has Kpart fully-connected layers for reducing dimensions and taking character-level affine transformation.
e = e1 T | . . . | eK T T (5) Where Wk and bk indicate trainable parameters with D2 × D1 and D2 dimensions respectively and f (·) means a non-linear activation function. Finally, we can obtain an utterance-level embedding e (See Table 1).

Objective functions
In our work, we made use of various objective functions conventionally used in speaker embedding training. Some variants of softmax-based classification loss were employed in our systems. Also, we used end-to-end based losses which directly optimize distance metrics such as Euclidean or Cosine distance. The objective functions used in the systems are as follows: • Standard Softmax • Additive Margin Softmax (AM-Softmax) [12,13] • Additive Angular Margin Softmax (AAM-Softmax) [14] • Angular Prototypical Loss (A-Prototypical) [15] • Generalized End-to-End Loss (GE2E) [16]

Back-end
In the back-end module, we only used cosine similarity as a scoring method between the two speaker embeddings. No Linear Discriminant Analysis (LDA), Within-Class Covariance Normalization (WCCN), and Probabilistic Linear Discriminant Analysis (PLDA) was applied in this work.

Score normalization
To minimize the domain mismatch (e.g., languages, recording environments, etc.) between the training and the evaluation set and to normalize the distribution of scores during fusion between different models, we used the Adaptive Symmetric Score Normalization (AS-Norm) [28]. We selected speaker-phrase dependent models in DeepMine Task1 Train Partition (i.e., indomain training data) as cohort set and used the most similar top 300 scoring files to calculate normalization variables of enrollment and test sets respectively.

Score compensation
Since the TI-SV approaches focus on minimizing the withinspeaker variability, the embedding vectors may lack crucial information on the lexical content. Thus such embedding vectors are improper for the TD-SV experiments. To complement the lost contextual discriminability, we introduce a score compensation method. Firstly, we define the posterior of the phrase as follows: where the set U = {uj | uj is j th phrase, 1 ≤ j ≤ M }, M is the number of phrase types in the TD-SV's dataset, X is an acoustic feature such as MFCC. We estimate the posterior p(U = uj |X) using softmax layers of TDNN-based network. The architecture of this network is identical to the configuration of TDNN-based architecture for CLP but composed of smaller size layers to prevent overfitting (See Table 1). For training the network, we used DeepMine Task1 Train Partition which includes 10 types of phrases. Finally, we compute the compensation factor and the total score between X and Y as follows: where s phr X,Y is compensation factor,s spk X,Y is the normalized (AS-Norm) score between embeddings of X and Y, α is a scale factor, and sX,Y is the total score between X and Y.

Training condition
According to the fixed training condition of the challenge, we used the designated datasets for training our systems and utilized RSR2015 dataset [18] as a validation set for monitoring. The training set for each system was the combination of different datasets and each training dataset is described as follows.
DeepMine (Task 1 Train Partition). This is the main dataset, i.e., in-domain data, of the SdSV Challenge. It contains 101,063 utterances from 963 speakers, which have five Persian and five English phrases. We used it for training (1) speaker embedding network and (2) estimating the posterior of phrase, and also as (3) cohort set to calculate parameters of AS-Norm.
VoxCeleb1 & 2. We used the development sets of Vox-Celeb1 [29] and VoxCeleb2 [30], which consist of 148,642 and 1,092,009 utterances from 1,211 and 5,994 speakers respectively. In our systems, they were used to train the speaker embedding networks.
LibriSpeech. To train the CTC-based ASR model, namely Jasper, which was utilized in character-level pooling and for estimating the posterior of phrase, we used the train-clean/other sets of LibriSpeech corpus [31], which comprise 281,241 utterances from 2,338 speakers. Additionally, in some systems, we employed them for training speaker embedding networks.

Trial condition
According to the trial condition of the challenge, the enrollment was accomplished using three utterances of a specific phrase for each model and among four types of trials in the TD-SV task, only Target-Correct, where the target speaker utters the correct pass-phrase, was considered as target and the rest was an imposter. The trial set was divided into two subsets: a progress subset (30%), and an evaluation subset (70%). The progress subset was used to monitor progress on the leaderboard, while the evaluation subset was used for the official results.

Analysis
We analyzed two experimental scenarios. First, we verified the feasibility and effectiveness of the character-level pooling strategy for the TD-SV task through the results on the progress and evaluation subsets (Table 3). In the second experiment, we applied AS-Norm and score compensation to all the systems we used in the first experiment, to further boost the performance. Also, we fused different systems and confirmed the best primary system and single system on the progress and evaluation subsets in terms of MinDCF, which was the main metric for the challenge (Table 4). No preprocessing such as data augmentation or VAD was applied to the training and trial data.
Analysis of character-level pooling strategy (Table 3). Each subsystem (1-15) was composed of different front-ends, pooling techniques, objectives, and training datasets described in Section 2 and 3.1. Among them, system (5) utilized text-dependent training (TDT), which was jointly trained by combined classes of speaker and phrase (i.e., speakers × phrases classes). In systems of (5-16), phrase information was not considered, since they were trained for TI-SV. For the reasons stated in Section 2.2, the character-level pooling methods (systems 1-4) showed improved performances compared with other systems in terms of 0.3554 MinDCF and 8.48% EER on the challenge's evaluation subsets (See Table 3).
Results using AS-Norm & score compensation (Table 4). We used AS-Norm and score compensation described in Sections 2.6 and 2.7 respectively, for improvement of performances. As you can see in Table 4, the performance of all systems was improved significantly. In particular, the performances of systems that didn't consider the lexical context (5-15) increased greatly, compared to the character-level pooling systems (1-4), which showed minor improvement. The best performance of a single system was 0.1307 MinDCF and 3.18% EER on the evaluation subsets. From these results, we could interpret that the better the speaker is distinguished, the higher the performance can be achieved, when score compensation was applied. Finally, we performed the fusion by computing the equal-weighted sum of the scores of different systems, which were TDNN-based (18), ResNet-based (19), and all systems (20). Overall, the performance of ResNet-based systems outperformed TDNN-based systems in the case of both single systems and fusions, and the best primary system was the fusion of all systems. It obtained 0.0785 MinDCF and 2.23% EER on the evaluation subset.

Conclusions
In this paper, we described our submission to Task 1 of the SdSV Challenge 2020. We propose a new pooling and score compensation methods that leverage a CTC-based end-to-end ASR model for taking the lexical content into account. Our systems contained two front-end architectures and acoustic features, and various pooling methods including our proposal, and different objective functions. Experiments show that the usage of the proposed character-level pooling and score compensation methods significantly enhances text-dependent speaker verification performance. Finally, the best performance of the primary system was obtained through the fusion of all the experimented systems, which showed 0.0785% MinDCF and 2.23% EER on the challenge's evaluation subset.

Acknowledgements
This work was supported by the research fund of Signal Intelligence Research Center supervised by Defense Acquisition Program Administration and Agency for Defense Development of Korea.