{'fraction_non_alphanumeric': 0.03079683786798584, 'fraction_numerical': 0.00960279353993889, 'mean_word_length': 4.949221004039238, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
SAMO: Speaker Attractor Multi-Center One-Class Learning for Voice Anti-Spoofing

Voice anti-spoofing systems are crucial auxiliaries for automatic speaker verification (ASV) systems. A major challenge is caused by unseen attacks empowered by advanced speech synthesis technologies. Our previous research on one-class learning has improved the generalization ability to unseen attacks by compacting the bona fide speech in the embedding space. However, such compactness lacks consideration of the diversity of speakers. In this work, we propose speaker attractor multi-center one-class learning (SAMO), which clusters bona fide speech around a number of speaker attractors and pushes away spoofing attacks from all the attractors in a high-dimensional embedding space. For training, we propose an algorithm for the co-optimization of bona fide speech clustering and bona fide/spoof classification. For inference, we propose strategies to enable anti-spoofing for speakers without enrollment. Our proposed system outperforms existing state-of-the-art single systems with a relative improvement of 38% on equal error rate (EER) on the ASVspoof2019 LA evaluation set.


INTRODUCTION
Automatic speaker verification (ASV) systems are essential for voice-based authentication that can recognize individuals through speech. With the rise of advanced voice conversion and speech synthesis techniques, ASV technologies are susceptible to possible spoofing attacks, especially logical access (LA) attacks with synthetic speech [1,2]. To tackle such vulnerability in ASV systems, a growing focus has been paid to spoofing countermeasure (i.e., CM or anti-spoofing) systems that often instrument deep learning models to differentiate speech spoken by humans (i.e., bona fide speech) and synthesized speech (i.e., spoofing attacks) [2].
Existing research on speech anti-spoofing investigated different embedding extractions. Recent end-to-end models have shifted from hand-crafted features such as LFCC [3] and CQCC [4,5] to raw waveforms, achieving state-of-the-art performance [6,7,8,9]. Other efforts have focused on training strategies, such as data augmentation [10,11] and multi-task learning [12]. Despite the impressive results, the main challenge remains on the generalization ability to unseen attacks, i.e., spoofing attacks generated by speech synthesis techniques that are not used in generating training data [1]. While this is a pressing issue due to the rapid development of speech synthesis techniques, most cutting-edge anti-spoofing systems do not explicitly address this problem. * This work is supported in part by a New York State Center of Excellence in Data Science award and synergistic activities funded by the National Science Foundation (NSF) under grant DGE-1922591. The authors would also like to thank Fei Jiang for the initial discussion of the multi-center idea. To address this generalization problem, in our prior work, we proposed the one-class learning approach [13] that tries to compact the bona fide speech into a cluster in the learned embedding space while pushing away spoofing attacks from this cluster. This approach has shown significant improvement in the generalization ability to unseen spoofing attacks [13]. However, due to the variety of timbre and speaking traits of different speakers, we have found that the bona fide speech of different speakers naturally forms multiple clusters in the embedding space. We argue that compacting different speakers into one cluster according to our original formulation [13] may have caused some misclassification of the spoofing attacks [14]. For instance, as shown in Figure 1(a), unseen spoofing attacks may lie in between multiple naturally separated clusters of bona fide speech, and compacting bona fide speech into a cluster is likely to coerce the spoofing attack samples into the cluster as well. It would make more sense to maintain the natural clustering of bona fide speech while pushing away spoofing attacks, as shown in (b).
In this work, we propose speaker attractor multi-center one-class learning (SAMO) to model speaker diversity while maintaining the generalization ability brought by one-class learning. We aim to find a speech representation for anti-spoofing that not only discriminates bona fide speech from spoofing attacks but also clusters bona fide speech according to speakers. After training, this model can be deployed to test speakers with or without enrollment utterances: With enrollment, the test utterance can be compared to the enrollment utterances for being classified into bona fide or spoofing attacks; Without enrollment, the test utterance can be compared to the learned attractors from training. The proposed system achieves notable improvements over state-of-the-art single systems on the ASVspoof2019 LA dataset, suggesting the effectiveness of our proposed SAMO scheme.
Our contributions are as follows: 1) We propose a novel multi-center one-class learning objective for anti-spoofing to respect the acoustic diversity of bona fide speech of different speakers, while discriminating bona fide and synthetic speech under the one-class learning paradigm. 2) We propose an algorithm with different optimization schedules to co-optimize the abovementioned aspects of the objective. 3) The proposed approach achieves the flexibility of deploying to speakers with or without enrollment utterances. Our work is reproducible with code at https://github.com/sivannavis/samo.

Speaker attributes in anti-spoofing
Existing research explored the idea of leveraging speaker attributes in anti-spoofing. There are mainly three categories. Some investigated the person-of-interest approach, where in [15], a pre-trained anti-spoofing model is calibrated on bona fide and fake speech of the known identity. In [16], anti-spoofing is formulated as a speaker verification problem, and one-class strategies are employed to show promising in-the-wild generalization ability. However, they are limited to known identities whose bona fide speech is available during testing. Some used speaker embedding from speaker verification to assist anti-spoofing, either by concatenating the speaker embedding and the spoof embedding [17] or in a multi-objective learning fashion [12,18]. Some designed a joint system to integrate anti-spoofing with speaker verification, i.e., spoofing-aware speaker verification to optimize both objectives [19,20].
We propose a novel way to leverage the speaker attributes without requiring a reference set of bona fide speech for known identities in the test scenario. Our method also has better simplicity with only one speech embedding extractor.

Preliminary work on one-class learning
One-class classification is to address the classification scenario where the normal class is well represented in the training set but the anomaly class is not [21]. With the distribution mismatch between training and test for spoofing attacks, specifically unseen synthetic attacks, the anti-spoofing problem fits the one-class classification setting. The idea of one-class learning goes beyond one-class classification by jointly learning the embedding space and classifier. It was proposed in [13] to learn an embedding space for anti-spoofing using neural networks. It uses the One-Class-Softmax (OC-Softmax) loss function as the training objective: wherexi ∈ R D represents the normalized speech embedding corresponding to the i-th utterance in a batch of N samples, and yi ∈ {0, 1} is the label. yi = 0 and yi = 1 indicate that the utterance is bona fide and spoofing attack, respectively. my i refers to the boundary margins for the bona fide and spoof, respectively. m0 > m1.
Around the center vectorŵ ∈ R D , m0 compacts the distribution of the bona fide speech, while m1 pushes away the spoofing attacks, and α is a scale factor. While this approach significantly improves the generalization ability to unseen attacks, it does not consider the acoustic diversity of bona fide speech from different speakers, channels, and recording environments. Such diversity naturally forms multiple clusters of bona fide speech. By squeezing bona fide speech embeddings into a single cluster, OC-Softmax faces the risk of misclassifying unseen spoofing attacks that would naturally lie in between those dense regions of bona fide speech.

METHODOLOGY
In this section, we describe our proposed method SAMO to address the abovementioned limitation of OC-Softmax [13]. The idea is to compact bona fide speech into multiple clusters instead of one, while pushing away spoofing attacks from all clusters. These clusters are formed based on speaker identity during training, and the cluster centers are called attractors. During inference, we measure the cosine similarity between the test utterance and the nearest speaker attractor (if no enrollment data) or the target speaker's enrollment speech (if there is enrollment data) to classify it as bona fide or spoof.

Speaker attractors
Extensive studies have manifested the intrinsic speaker diversity of bona fide speech in speech processing tasks, such as source separation [22,23], text-to-speech [24], and speaker diarization [25]. This diversity naturally forms multiple clusters of bona fide speech in speech embedding spaces, where utterances from the same speaker tend to be close to each other. Leveraging this idea, we define a speaker attractor as a speaker-specific anchor in the embedding space to attract bona fide speech embeddings of the same speaker during training. To compute and update such anchors, we simply average the embeddings of each speaker's bona fide speech as a speaker attractor. Through training, the network weights and the embedding space are updated, and so do the attractors. However, the update of attractors does not happen in each iteration of the training; instead, it happens on a schedule explained in Section 3.3.
Ideally, these attractors respect the natural clustering effect caused by the acoustic diversity of speakers while achieving good coverage of bona fide distribution for one-class learning. In the next section, we propose a loss function to achieve this goal. After training, the learned attractors are fixed during inference. If the test speaker has no enrollment utterances, then the test utterance can be compared to these speaker attractors through the inner product to compute a score indicating whether it is bona fide or spoof. If the test speaker has enrollment utterances, then the test utterance can be compared to the enrollment utterances for the classification. The inference score is defined as wherexi ∈ R D is the normalized speech embedding of the i-th test utterance, si is the corresponding speaker, andŵs i ∈ R D is the average embedding of their enrolled utterances if any.ŵs are learned attractors of speaker s in the training set.

Loss function for multi-center one-class learning
In Eq. (1), bona fide speech embeddings are compacted to a single cluster around the centerŵ. Here we replace this single cluster with multiple clusters, each of which is around one attractor corresponding to one speaker in the training set, leading to the proposed SAMO loss function: where di is calculated by Herexi is the normalized speech embedding of the i-th utterance in the batch and yi is the corresponding label. si is the corresponding speaker andŵs i is the normalized speaker attractor. α is the scale factor. m0 and m1 are the margins for the bona fide and spoof classes, respectively. N is the number of utterances in the batch. di is calculated as the cosine similarity to measure how similar the feature embeddings are to speaker attractors. As in Eq. (4), for a bona fide sample, the similarity is calculated between its normalized embeddingxi and its speaker attractorŵs i . Alternatively, when a sample belongs to the spoofing class, the similarity is betweenxi and its nearest speaker attractor among all training speakers Strain. By minimizing this loss, the model can learn an embedding space that compacts speech utterances belonging to the same speaker and pushes away spoofing attacks from all speaker attractors. This is different from OC-Softmax in Eq. (1), where all the training utterances are attracted to or pushed away from only one centerŵ. It is noted that the idea of multi-center has been employed in other tasks, such as face recognition [26] and language recognition [27] to better capture the local neighbor relationship of the embedding space. However, their centers are randomly assigned to samples of each class. Our proposed SAMO leverages the auxiliary speaker information in defining the clusters and only uses multiple centers for the bona fide class to enhance diversified compactness for one-class learning in the context of anti-spoofing.

SAMO training algorithm
The proposed SAMO loss essentially has two sub-objectives: compacting the bona fide utterances spoken by the same speaker and pushing away spoofing utterances from all speaker attractors. Both sub-objectives use the attractors in the calculation. By definition in Section 3.1, each attractor is computed as the average embedding of the bona fide speech of each speaker, which changes as the embedding space changes. However, if the attractors are updated every time the embedding space updates (i.e., a mini-batch in network training), it may harm the convergence of the second sub-objective, i.e., pushing away spoofing utterances from all bona fide clusters. This is because any update of the attractors will change the optimization directions for the spoofing utterances. Empirically, we do find that frequently updating the attractors makes it difficult for the loss to converge.
We propose to update speaker attractors every M epochs during training. The relative frequency of updates between the network weights (hence embedding space) and attractors is thus M times the number of mini-batches in each epoch. In this way, the attractors serve as relatively stable anchors for attracting bona fide utterances and pushing away spoofed ones. In Section 4, we study how M affects the system performance. At the beginning of training, we initialize speaker attractors as one-hot vectors in the embedding space, each representing the optimization direction of a speaker. The training process is described in Algorithm 1.

Experimental setup
As mentioned in Section 3.1, the SAMO system, after training, can be used for test speakers with or without enrollment data. In our experiments, we test the systems in both scenarios. Dataset. We use utterances from ASVspoof2019 LA [28] as our data in all experiments. We use the same train/dev/eval splits as ASVspoof2019 LA. All attack types in the evaluation set are different from those in the training and development sets. Regarding speakers, we only keep the target speakers in the development and evaluation sets and discard the non-target speakers from these sets; The non-target speakers do not have enrollment data to support the scenario where enrollment data is used during inference, hence are adverse to our comparison between the two scenarios, both of which our method is able to deal with. Details of our dataset are shown in Table 1.
Evaluation metrics. We use equal error rate (EER) and the minimum tandem detection cost function (min t-DCF) as the metrics for all experiments. The EER metric is defined as the identical false acceptance rate and false rejection rate with a certain threshold of the inference score. Along with EER, min t-DCF evaluates all types of errors in speaker verification when the CM system is integrated into the "common" automatic speaker verification (ASV) system provided by the ASVspoof2019 challenge [29]. As the ASV system is fixed, it can be viewed as a metric for the CM system when deployed to a speaker verification application [30].
Comparison methods. We compare SAMO with two state-ofthe-art anti-spoofing methods: AASIST [9] and OC-Softmax [13]. AASIST is a model based on graph neural networks with integrated spectro-temporal attention. It uses the softmax loss in training. We train AASIST with its reported configurations in [9]. In [13], OC-Softmax uses a backbone network based on deep residual network ResNet-18 with one-class learning loss mentioned in Section 2.2.
To remove the effect of backbone network architecture, we change the network backbone to AASIST but maintain the OC-Softmax loss function, using α = 20, m0 = 0.5, and m1 = −0.2.
Implementation details. For all experiments, we use the architecture of AASIST [9] as our speech embedding network. A 160-d embedding vector is computed from the last hidden layer. For our proposed SAMO, we set the update interval M = 3; the scale factor Table 2. Comparison of our proposed SAMO with Softmax and OC-Softmax on the target-only portion of the ASVspoof2019 LA evaluation set. All the systems use AASIST [9] backbone. The average (best) results across 3 trials with random training seeds are shown.

Results and discussions
Comparison with state-of-the-art methods. Table 2 shows the EER and min t-DCF comparisons between SAMO and the two comparison methods on the target-only portion of the ASVSpoof2019 LA evaluation set. We report the average results with 3 different random seeds for all models and report the best results in parentheses. We can see that: 1) The results of our proposed method SAMO with two inference strategies both demonstrate advantages over other comparison methods with notable margins, e.g., SAMO with enrollment decreases EER of Softmax and OC-Softmax by relative 38% (1.08% vs 1.74%) and 14% (1.08% vs 1.25 %), respectively. 2) Compared to Softmax, OC-Softmax improves the system performance by 28% relatively, which agrees with the advantages of one-class learning over the binary classification reported in [13].
3) Compared to OC-Softmax, SAMO further improves the performance by 14% relatively, indicating the advantage brought by the multi-center modeling of bona fide speech in SAMO. Ablation studies. Table 3 investigates the effects of important components and parameters of our proposed SAMO system. We report average results for all setups with 3 random seeds. To see the effects of leveraging bona fide speech data in speaker attractors, Setup 2 initializes the speaker attractors as one-hot vectors in the embedding space without further updates during training. Results show that this setup does not learn a meaningful embedding space. Setup 3 initializes the attractors as one-hot vectors first, then only updates attractors once at the beginning of the second epoch. Compared with Setup 2, this setup shows that simply including bona fide speaker information in attractors without any training schedule helps with learning a useful embedding space. Compared with SAMO, it also shows that a proper update schedule is essential for achieving Embedding visualizations. To comprehend the effects of speaker attractors in the embedding space, we pick three speakers in the training set and transform their 160-D embedding vectors to 2-D in Figure 2 with t-SNE [32]. From the visualization, we can see that the bona fide utterances are grouped in a small region at the right side of the figure, while the spoofing attacks are pushed away from this region, occupying a large area on the left. This shows that SAMO follows the general one-class learning framework. Within the bona fide utterance region, we can also see that utterances of the three speakers are generally clustered according to speaker identity. This shows that SAMO models the bona fide class as speaker clusters. Both observations show that the SAMO algorithm performs as expected as the objective function Eq. (3).
The current training set only contains 20 speakers. We believe that with a larger variety of speakers in the training set, the benefit of SAMO could be demonstrated even more since the speaker attractors will better represent the bona fide embedding space.

CONCLUSION
We proposed speaker attractor multi-center one-class learning (SAMO) as a novel representation learning framework for voice anti-spoofing. It models speaker diversity in bona fide speech to build compact speaker-specific clusters in the embedding space. The approach is to compact bonafide speech around speaker attractors and isolate spoofing attacks from all attractors. Experiments show that SAMO improves anti-spoofing performance over stateof-the-art methods, including AASIST and OC-Softmax. For future work, we plan to extend the SAMO idea to model other speech attributes, such as device [33] and codec variations [34].