{'fraction_non_alphanumeric': 0.02980144469853715, 'fraction_numerical': 0.021634179099059857, 'mean_word_length': 5.0803354667843745, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
FMFCC-A: A Challenging Mandarin Dataset for Synthetic Speech Detection

As increasing development of text-to-speech (TTS) and voice conversion (VC) technologies, the detection of synthetic speech has been suffered dramatically. In order to promote the development of synthetic speech detection model against Mandarin TTS and VC technologies, we have constructed a challenging Mandarin dataset and organized the accompanying audio track of the first fake media forensic challenge of China Society of Image and Graphics (FMFCC-A). The FMFCC-A dataset is by far the largest publicly-available Mandarin dataset for synthetic speech detection, which contains 40,000 synthesized Mandarin utterances that generated by 11 Mandarin TTS systems and two Mandarin VC systems, and 10,000 genuine Mandarin utterances collected from 58 speakers. The FMFCC-A dataset is divided into the training, development and evaluation sets, which are used for the research of detection of synthesized Mandarin speech under various previously unknown speech synthesis systems or audio post-processing operations. In addition to describing the construction of the FMFCC-A dataset, we provide a detailed analysis of two baseline methods and the top-performing submissions from the FMFCC-A, which illustrates the usefulness and challenge of FMFCC-A dataset. We hope that the FMFCC-A dataset can fill the gap of lack of Mandarin datasets for synthetic speech detection.


Introduction
Synthetic speech attacks refer to a situation that an attacker makes use of textto-speech (TTS) or voice conversion (VC) technologies to synthesize a speaker's voice to cheat the automatic speaker verification (ASV) systems [17]. With the advancements in synthetic speech technologies, the state-of-the-art TTS and VC systems achieve such a high level of naturalness that even humans have difficulties to distinguish synthetic speech from genuine speech, which imposes a arXiv:2110.09441v1 [cs.SD] 18 Oct 2021 significant threat to the reliability of ASV systems [14]. Therefore, it is crucial to develop a synthetic speech detection model that can efficiently discriminate between genuine and synthetic speech [7].
To protect ASV systems from the increasing development of speech synthesis technologies, researchers from across the globe create fake speech detection challenges and release synthetic speech datasets. Automatic speaker verification spoofing and countermeasures challenge, called ASVspoof, has been launched in 2015, 2017, 2019 and 2021, and four speech datasets (ASVspoof2015 dataset [17], ASVspoof2017 dataset [7], ASVspoof2019 dataset [14], ASVspoof2021 dataset [18]) have been released by the organizers. These ASVspoof datasets are important milestones in the synthetic speech detection community, which make the research community could study and propose methodologies to solve synthetic speech attacks to ASV systems. One of the most cited versions is ASVspoof2015 dataset [17], which contains not only genuine utterances but also synthesized utterances generated by 10 different speech synthesis systems (seven VC systems and three TTS systems), and in total more than 260,000 utterances are provided. ASVspoof2019 LA dataset [14] contains genuine utterances and synthesized utterances generated using 17 different TTS and VC systems in total 121,000 utterances. In addition, in order to make a speech dataset containing the utterances synthesized by the latest speech synthesis technologies, Reimao et al. [10] released a Fake or Real (FoR) dataset in 2019. However, it is important to note that the TTS and VC systems utilized in existing speech datasets are outdated compared to the current state-of-the-art speech synthesis systems, and all utterances are in English. A new speech dataset containing the synthesized speech by the latest speech synthesis technologies and spoken in non-English is urgently needed to be proposed.
In order to facilitate the development of synthetic speech detection for synthesized Mandarin utterances, we have constructed a challenging Mandarin dataset and organized the accompanying audio track of the first fake media forensic challenge of China Society of Image and Graphics (FMFCC-A). The FMFCC-A dataset is focused on the latest Mandarin speech synthesis technologies, where the synthesized Mandarin speech is generated by not only open-source tools but also commercial systems. The main contributions of this work are summarized as follows: 1) We constructed the FMFCC-A dataset for the research of detection of synthesized Mandarin speech and divided it as the training, development and evaluation sets, while four kinds of audio post-processing operations were conducted on valuation dataset. 2) To demonstrate it is possible to use the FMFCC-A dataset to train deep learning based classifiers, we adopted two neural network based synthetic speech detection methods as the baselines. 3) We provided a detailed analysis of top-performing submissions from the FMFCC-A, which are focused upon detection of Mandarin TTS and VC speech under various previously unknown speech synthesis systems or audio post-processing operations. To verify and reproduce the experiments presented in the paper, all of our source codes and datasets are now available via GitHub: https://github.com/Amforever/FMFCC-A. The remainder of this paper is organized as follows. In the next section, we formally introduced the construction of the FMFCC-A dataset and two baseline methods. In Section 3, the evaluation metrics and FMFCC-A schedule are described. In Section 4, we present a detailed analysis of two baseline methods and the top-performing submissions from the FMFCC-A. Finally, conclusions and potential future works are drawn in the last section.

Dataset and Baselines
The objective of this paper is to construct a challenging Mandarin dataset for synthetic speech detection and validate the usefulness and challenge of the dataset. In this section, we explain the construction of the FMFCC-A dataset and two detection baselines in detail.

FMFCC-A Dataset and Partitions
The FMFCC-A dataset contains not only genuine utterances but also a mountain of synthesized utterances, which are spoken in Mandarin. The total number of utterances of the FMFCC-A dataset is 50,000, which includes 10,000 genuine utterances and 40,000 synthesized utterances. The genuine utterances, i.e. speech recordings from humans, were randomly collected from 58 speakers, which covers a good variety of speaker ages and genders. The synthesized utterances were generated from state-of-the-art Mandarin speech synthesis systems. Specifically, the synthesized utterances of the FMFCC-A dataset were generated according to 11 Mandarin TTS systems, i.e., Alibaba TTS, Biaobei TTS, Blackberry TTS, FastSpeech [11], Iflytek TTS, IBM Waston TTS, Lingyun TTS, Tacotron [16], Baidu TTS, Sibichi TTS and GAN-TTS [3], and two Mandarin VC systems, i.e.,

Audio
Post-processing Evaluation dataset Speech corpus Ada-In VC [4] and Medium VC [5]. The 13 Mandarin speech synthesis systems are noted as A01 to A13 in this paper (as in Table 1), where A04 [11], A06 [4], A09 [16], A12 [3] and A13 [8] are open-source speech synthesis tools and A01, A02, A03, A05, A07, A08, A10 and A11 are commercial speech synthesis systems, For five open-source synthesis tools, we generated synthesized Mandarin utterances by implementing their source codes, while we sent API requests for commercial speech synthesis systems to get synthesized Mandarin utterances. For each utterance in the FMFCC-A dataset, the duration is randomly set in the range between two seconds and 10 seconds, the sampling rate of 16 kHz, 16-bit quantization and is stored in mono WAV format. More details about the utterances in the FMFCC-A dataset are shown in Table 1.
As with most researches of synthetic speech detection, the FMFCC-A dataset is partitioned into three disjoint datasets, namely training, development and evaluation sets. More details about the utterances and the used speech synthesis systems in training, development and evaluation datasets are shown in Table  2. The training dataset includes 6,000 synthesized utterances and 4,000 genuine utterances. As illustrated in Table 2, each synthesized utterance in training dataset is generated by one of the five speech synthesis systems (A01 -A05). The development dataset includes 17,000 synthesized utterances and 3,000 genuine utterances. In order to generalize the performance of synthetic speech detection models against previously unknown speech synthesis systems, we also produced synthetic speech with five additional speech synthesis systems (A06 -A09). The evaluation set of FMFCC-A dataset is designed to evaluate and analyze the synthetic speech detection models under previously unknown speech synthesis systems and audio post-processing operations, so the construction of evaluation dataset is more complicated than training and development sets (as shown in Fig. 1), The evaluation dataset is comprised of 17,000 synthesized utterances and 3,000 genuine utterances. On the one hand, to evaluate the synthetic speech detection models against previously unknown speech synthesis systems, the synthesized utterances are generated according to more diverse speech synthesis systems which include the same nine speech synthesis systems (A01 -A09) used in the development dataset and additional four speech synthesis systems (A10 -A13). On the other hand, in order to evaluate the performance of synthetic speech detection models under various audio post-processing operations, 50 percent of the evaluation dataset are randomly selected to undergone compression-decompression operation or additive Gaussian noise operation. Specifically, we selected 12.5 percent of the evaluation dataset and transformed them (WAV files) into MPEG-1 audio layer 3 (MP3) with 96 kilobits per second (Kbps) and converted MP3 files back to WAV using FFmpeg software 1 . In the same way, advanced audio coding (AAC) compression of 64 Kbps and decompression were conducted on another 12.5 percent of the evaluation dataset. For random Gaussian noise addition, we selected 12.5 percent of the evaluation dataset and process the speech with 0.01 random Gaussian noise addition, and process another 12.5 percent of the evaluation dataset with 0.002 random Gaussian noise addition. Finally, 50 percent of the evaluation dataset are undergone a compression-decompression operations or additive Gaussian noise operations and the remaining 50 percent of the evaluation dataset are original without any audio post-processing operations.

Baseline Methods
The whole flowchart of the neural networks based synthetic speech detection models is illustrated in Fig. 2. Firstly, acoustic features are extracted from the speech signal. Then, several different designed convolution blocks are stacked to extract contextual representations of the input acoustic feature. Finally, the fully connected layers map the deep features on the label space of speech and the probability of being synthetic speech is calculated by a softmax layer. In recent years, many acoustic features have been proposed for synthetic speech detection, while Mel-frequency cepstral coefficient (MFCC) [9] and constant-Q cepstral coefficient (CQCC) [13] are two of the most used acoustic features. In addition, light convolutional neural network (LCNN) [2] and residual convolutional neural network (ResNet) [1] are the popular neural network architectures used as deep representation learning and classification for synthetic speech detection. Nowadays, several end-to-end synthetic speech detection models based on neural networks with mere the raw speech waveform have been proposed [12,6], which could achieve satisfactory performance.
In order to demonstrate it is possible to train deep learning based classifiers using the FMFCC-A dataset, we adopted two neural network based synthetic speech detection methods as the baselines. The first baseline method (B01) consists of 34 layers ResNet and CQCC feature [13]. The parameters of extraction of the CQCC feature are similar to those in [14]. The second baseline method (B02) adopted Res-TSSDNet architecture [6] and the raw speech waveform, which is an end-to-end synthetic speech detection method. Two baseline methods are implemented in PyTorch 1.1.0 2 , which are trained on the training set of FMFCC-A dataset, and tested on the development and evaluation sets respectively.

Evaluation Metrics
Two 'threshold-free' evaluation metrics are used to estimate performances of synthetic speech detection methods on the FMFCC-A dataset, which are Logloss [15] and equal error rate (EER) [7]. The Log-loss [15] is one of the major metrics to assess the performance of a classification problem. It is indicative of how close the prediction probability is to the corresponding true value (0 or 1 in the case of binary classification). The more predicted probability diverges from the actual value, the higher is the log-loss value. For binary classification, Log-loss can be expressed as follows: where i is the index of utterances and it corresponding label is y i ; p i is the prediction probability and ln refers to the natural logarithm of a number. To avoid the case of ln(0), ln(max(p), ) is computed where is a positive real number closing to zero. The lower Log-loss indicates that the detection models have performed better. The EER is a commonly used metric for the datasets that are heavily unbalanced for two-class classification (one genuine utterance versus many synthesized utterances) [7]. Suppose P fa (θ) and P fr (θ) stand for the false acceptance rate and false rejection rate respectively, which are defined as : Num{synthesized utterances with score > θ} Num{total synthesized utterances} , P fr (θ) = Num{genuine utterances with score ≤ θ} Num{total genuine utterances} , where Num{·} denotes the number of utterances in the set. The EER corresponds to the threshold θ EER at which the false acceptance rate equals to false rejection rate i.e. EER = P fa (θ EER ) = P fr (θ EER ). In this paper, the EER is estimated using the Bosaris toolkit 3 .

FMFCC-A Schedule
FMFCC-A focuses on the research of detection of synthesized Mandarin speech, which is run in two phases: the preliminary phase and the main evaluation phase. Participants can use training and development sets of the FMFCC-A dataset to construct and optimize their synthetic speech detection models, while the use of other external speech datasets is also allowed. Participants should assign each utterance with a real-valued and finite score range in [0, 1] which reflected the relative probability that the test utterance was genuine. The Log-loss [15] was adopted as the metric for evaluating the performance of submissions, and in Formula (1) is set to 10e −9 .
During the preliminary phase, the training and development sets of FMFCC-A dataset were available for each participant, and a standard protocol file comprising a list of filenames and corresponding labels of utterances from the training dataset was provided. However, the label information of speech from the development dataset was not provided. Each participant team might make up to 3 submissions per day and the corresponding validation performance of the submission would be provided by the organizer. The focus of the preliminary phase was upon the training and optimizing synthetic speech detection methods that were robust to previously unknown speech synthesis systems. During the main evaluation phase, participants were allowed only a single submission for which the result would be determined by the utterances from the evaluation dataset. Given the relevance to realistic scenarios, the focus of the main evaluation phase was upon evaluating generalization of synthetic speech detection methods to previously unknown speech synthesis systems and audio post-processing operations.

Preliminary Phase of FMFCC-A
In the preliminary phase of the FMFCC-A, a total of 48 valid submissions were received. Since the aim of the FMFCC-A is the scientific analysis of different Mandarin synthetic speech detection methods, we assign anonymized team IDs (T01 to T48) to participators, which are corresponding to the order ranking in the preliminary phase. Table 3 shows results in terms of the Log-loss and EERs for all submissions as well as two baseline methods (B01 and B02), pooled over all speech synthesis systems in the development dataset. As we can see from Table 3, 27 of 48 participating teams submitted results that outperformed the B01 in terms of the Log-loss, and 13 participating teams' results achieved lower EERs than that of B01. The top two performing submissions, T01 and T02, achieve Log-loss of 0.000083 and 0.000223, and both EERs are equal to 0, which are performing results. However, it observes that monotonic increases in the Log-loss that are not always mirrored by monotonic increases in the EER. The participating teams T15, T23, T30, and T31 deliver low EERs, however, they have high Log-loss values which lead them to a low ranking in the preliminary phase of FMFCC-A. There is a great spread both in Log-loss and EERs, however, the performance of the top nine performing systems is much narrower. Top nine performing submissions achieve Log-loss below 0.1 and EERs below 3% even when almost half of the development dataset are generated by previously unknown speech synthesis systems, which deliver a substantial improvement over other submissions. After communications with participants, we knew that some of the participants used the idea of pseudolabel learning methods. They uploaded result files, which contained scores of all utterances on the development dataset, to get the feedback from the organizers and sent the feedback to their neural networks to make the neural networks more effectively learn for difficult samples. The performance of submission on preliminary phase of FMFCC-A demonstrates that the training set of FMFCC-A dataset could be used to train deep learning based classifiers, while the development set of FMFCC-A dataset could be further optimized the parameters of synthetic speech detection models.

Main Evaluation Phase of FMFCC-A
In the main evaluation phase, the top 10 participants in the preliminary phase were required to run their submissions on the evaluation set of the FMFCC-A dataset. Note that the eighth-place team (T08) in the preliminary phase, unfortunately, did not run their submission despite repeated warnings from the organizers. Therefore, we excluded them in the main evaluation phase and asked the eleventh-place participating team (T11) in the preliminary phase to participate in the main evaluation phase. The Log-loss and EER results of 12 synthetic speech detection methods on the main evaluation phase are illustrated in Table  4.
As can be seen in Table 4, the T06, which ranks sixth place on the preliminary phase, achieves the first place in the main evaluation phase with a Log-loss of 0.351378 and EER of 13.59%. The T01 that achieves the best performance in the preliminary phase ranks second place in the main evaluation phase according to the Log-loss. However, T01 delivers the lowest EER of 9.50% among all submissions. In addition, all top ten performing submissions achieve better performance than two baseline methods in the main evaluation phase. The Logloss of all synthetic speech detection models on the main evaluation phase are ranged between 0.351378 and 2.286764, and EER are ranged between 9.50 and 33.37, which are increased a lot compared to those on the preliminary phase. This observation indicates that the evaluation set of the FMFCC-A dataset has dramatically decreased the performances of submissions and baselines, which illustrates the challenge of the FMFCC-A dataset for synthetic speech detection methods. To evaluate the performance of submissions against previously un-known speech synthesis systems and audio post-processing operations, we give a detailed description in the following. Table 5. EERs (%) of detection methods as measured on the evaluation set of the FMFCC-A dataset subjected to each speech synthesis system and pooled performances.

Performance on Unknown Speech Synthesis Systems
While the participants' rankings of FMFCC-A are based on pooled performance (the synthetic speech from all speech synthesis systems in the dataset are considered for evaluation), it is also meaningful to present results when decomposed by each speech synthesis system. As described in Section 2.1, we classify the evaluation set of the FMFCC-A dataset into three parts according to whether the speech synthesis systems are used in training and development datasets. The known speech synthesis systems (KSSS) set includes A01 to A05 which are presented in both of the training and development datasets. The first unknown speech synthesis systems (USSS1) set includes A06 to A09 which are not presented in the training dataset but used in the development dataset. The second unknown speech synthesis systems (USSS2) set includes A10 to A13 which are neither used in training dataset nor development dataset. The EER results when decomposed separately for each of the speech synthesis systems are presented in Table 5, while the pooled detection results for KSSS, USSS1, and USSS2 (named as Pool1, Pool2 and Pool3 respectively) are also presented.
As shown in Table 5, each line represents the EERs of one detection method against various speech synthesis systems. The speech synthesis systems A06, A12 and A13 seriously degrade the performances of detection methods, which showed being challenging to be detected. In fact, the A06 and A13 are Mandarin VC systems and A12 is a generative adversarial network (GAN) based Mandarin TTS system, which are fundamentally different from other Mandarin TTS systems.
Speech synthesis systems Therefore, the detection methods that trained on the training dataset achieve poor performance for A06, A12 and A13. Although A07, A08, A10, and A11 are also threat detection methods, they are easier to detect than A06, A12 and A13. One reason may be that A07, A08, A10, and A11 are pipeline TTS techniques that are similar mechanism with TTS techniques of A01 to A05. In addition, EERs of Pool1, Pool2 and Pool3 are 8.89 %, 13.99 % and 26.61 % respectively, which means that the threat of previously unknown speech synthesis systems to detection methods are obviously compared to the known speech synthesis systems.
In order to show the variation of Log-loss results on the evaluation dataset for synthetic speech detection methods, we illustrate boxplots Log-loss results in Fig. 3. The similar trends can be observed in the boxplots of Log-loss. There is a greater spread in Log-loss for attacks A06, A12 and A13, which shown they are a greater threat to detection methods. The difference between performances of detection methods against A01 to A05 is much narrower. Through the analysis of detection methods for known attacks (KSSS) and unknown attacks (USSS1 and USSS2), it indicates the importance of improving the detection models against previously unknown speech synthesis systems.

Performance on Audio Post-processing Operations
Considering the relevance to realistic scenarios, it is also very important for the detection models to detecting synthesized Mandarin utterances that have under- gone various audio post-processing operations. In this section, we analyze the results of detection models for the synthesized Mandarin utterances undergone each audio post-processing operation on the evaluation set of the FMFCC-A dataset. As described in Section 2.1, we classified the evaluation dataset into five parts according to whether utterances have undergone post-processing operations. Firstly, the 50 percent of the evaluation dataset were not undergone any audio post-processing operations as original set (Original). Secondly, the 12.5 percent of the evaluation dataset were processed by MP3 compressiondecompression with 96 Kbps (CD-MP3). Thirdly, the 12.5 percent of the evaluation dataset were processed by AAC compression-decompression with 64 Kbps (CD-AAC). Next, the 12.5 percent of the evaluation dataset were added with 0.01 random Gaussian noise (NA-01). Finally, the remaining 12.5 percent of the evaluation dataset were added with 0.002 random Gaussian noise (NA-002). Table 6 shows the EERs of detection models for detecting original and audio post-processing utterances in the evaluation dataset. As shown in Table 6, the average EER of detection models on the original dataset is 17.63%, which is much lower than that applied with audio postprocessing operations. The performance of detection models on utterances with 0.002 random Gaussian noise is better than that on utterances with 0.01 random Gaussian noise, expect for T11, T10 and B01. Specifically, the average EER of detection models on NA-002 is 21.98% which is lower than that 22.95% on NA-01. It indicates that the higher level Gaussian noise added to utterances, the more difficult to be detected. As shown in the last two columns of Table  6, the average EERs of detection models on the MP3 and AAC compressiondecompression utterances are 22.19% and 24.02% respectively, which are much higher than that on original utterances. It is demonstrated that there is great performance degradation of detection methods due to the lower bit-rate AAC compression-decompression operations. In a word, it is shown that all four audio post-processing operations degrade the performance of synthetic speech detection methods.

Conclusion
With the improvement of Mandarin TTS and VC technologies, the need for an up-to-date Mandarin dataset that can be used in the research of synthetic speech detection also increases. In this paper, we introduce the FMFCC-A dataset and provide a detailed analysis of two baseline methods and top-performing submissions from the FMFCC-A. The FMFCC-A dataset contains 10,000 genuine utterances collected collected from 58 speakers and 40,000 synthesized utterances generated by 11 Mandarin TTS systems and two Mandarin VC systems. We divide the FMFCC-A dataset as the training, development and evaluation sets, and conduct four kinds of audio post-processing operations on valuation dataset. The training and development datasets are used for training and optimizing the synthetic speech detection models, while the evaluation dataset focus upon evaluating the generalization of synthetic speech detection methods to previously unknown speech synthesis systems and audio post-processing operations. Through the detailed analysis of two baseline methods and the top-performing submissions from the FMFCC-A, we demonstrate the usefulness and challenge of FMFCC-A dataset. At the same time, we also observe that previously unknown speech synthesis systems and audio post-processing operations would significantly degrade the performance of synthetic speech detection.
There are two main areas of future work regarding the FMFCC-A dataset. Although the FMFCC-A dataset is already a good source for the research of detection of synthesized Mandarin speech, it can be improved by aggregating more genuine and synthetic utterances in Mandarin. In addition, more realistic audio post-processing operations will be introduced to enhance the practical application of FMFCC-A dataset. With the FMFCC-A dataset created and published, we hope that the research community can use it to improve the detection of synthesized Mandarin speech.