{'fraction_non_alphanumeric': 0.031123439667128987, 'fraction_numerical': 0.011761442441054091, 'mean_word_length': 4.9787728026534, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Unified Source-Filter GAN: Unified Source-filter Network Based On Factorization of Quasi-Periodic Parallel WaveGAN

We propose a unified approach to data-driven source-filter modeling using a single neural network for developing a neural vocoder capable of generating high-quality synthetic speech waveforms while retaining flexibility of the source-filter model to control their voice characteristics. Our proposed network called unified source-filter generative adversarial networks (uSFGAN) is developed by factorizing quasi-periodic parallel WaveGAN (QPPWG), one of the neural vocoders based on a single neural network, into a source excitation generation network and a vocal tract resonance filtering network by additionally implementing a regularization loss. Moreover, inspired by neural source filter (NSF), only a sinusoidal waveform is additionally used as the simplest clue to generate a periodic source excitation waveform while minimizing the effect of approximations in the source filter model. The experimental results demonstrate that uSFGAN outperforms conventional neural vocoders, such as QPPWG and NSF in both speech quality and pitch controllability.


Introduction
Currently, neural vocoders  usually achieve very highfidelity speech generation by directly modeling raw speech waveforms using advanced neural networks without ad hoc designs. On the other hand, because of the data-driven nature and unified network architecture [27,28], the speech components controllability of the neural vocoders are usually inferior to the conventional source-filter vocoders [29,30]. Therefore, it is desired to develop a neural vocoder capable of high-fidelity and controllable speech generation.
To improve the controllability, there have been proposed many generation models integrating conventional parametricbased source-filter models with deep neural network architectures [21][22][23][24][25][26]. For example, neural source-filter (NSF) [21,22] realizes speech generation based on non-autoregressive modeling by non-linear filtering of parametrically generated source excitation signals with multiple dilated convolutional layers. LPCNet [23] adopts a WaveRNN [19]-like architecture to generate residual signals while a linear filtering process is applied to generate speech waveforms as in the conventional linear predictive coding (LPC) vocoder [31,32]. Generative adversarial network (GAN) based neural homomorphic vocoder (NHV) [24] first develops neural-based linear-time-variant (LTV) filters with the input pulse trains and white noise to generate mixed source excitations, and then a trainable causal finite impulse response (FIR) filter is applied to the excitations for generating the output waveforms. Although these hybrid neural vocoders have successfully improved the controllability by integrating parametric-based approaches, the synthetic speech quality of these vocoders tends to be inferior to that of data-driven unified neural vocoders. Moreover, there is still room for improvements in the controllability.
To achieve high-fidelity and high-controllability speech generation, we propose a GAN-based framework to introduce the source-filter model with fewer ad hoc designs into a single neural network. The generator is designed by factorizing quasi-periodic Parallel WaveGAN (QPPWG) [28] into two cascaded networks corresponding to the source excitation generation and resonance filtering, and these two networks are jointly optimized in the training stage. Only a sinusoidal waveform is additionally used as the simplest clue to generate a periodic source excitation waveform while minimizing the effect of approximations in the source filter model. Moreover, to generate reasonable source excitation signals, an additional auxiliary loss is applied to the source excitation network. The main contributions of this paper are summarized as follows: • We propose a unified framework for neural vocoders attaining an interpretable and tractable source-filter-like architecture, making it possible to well models excitation generation and resonance filtering while keeping the simplicity of training.
• The proposed method achieves better fundamental frequency (F0) controllability than the conventional neural vocoders, such as QPPWG and NSF, while attaining high-fidelity speech generation even in F0 transformation scenarios.

Related work
This chapter describes non-AR neural vocoders: Parallel Wave-GAN (PWG) [3], QPPWG [28], and NSF [21]. PWG and QP-PWG are the basis of our method, and we use QPPWG as one of the baseline methods. NSF is a semi-parametric neural vocoder based on the source-filter model, which is used as the other baseline method.

Parallel WaveGAN (PWG)
PWG is a GAN-based method for generating raw waveforms.
It is a compact model without an autoregressive structure or a causal mechanism, and can achieve fast speech generation with high fidelity. The model consists of two networks, generator (G) and discriminator (D). The WaveNet-based generator, which is conditioned by auxiliary features, learns to make the discriminator recognize the generated sample as real. This process can be written as follows: where z is random noise distributed from Gaussian distribution. Note that all auxiliary features of the generator are omitted in this paper for simplicity. The discriminator learns to identify the generated sample as f ake and the natural sample as real. This process can be written as follows: where x denotes the natural samples and p data denotes the data distribution of the natural samples. PWG also adopts multi-STFT loss [3] as an auxiliary loss Laux(G) to improve the training stability. In conclusion, the final loss function of the generator can be written as a weighted sum of Laux and L adv as follows: LG where λ adv is hyperparameter for weight and empirically set to 4.0 in this paper.

Quasi-periodic parallel WaveGAN (QPPWG)
Although PWG achieves high-fidelity speech generation, the fully data-driven nature makes PWG lack the explicit controllability of each speech component especially when unseen auxiliary features are given such as F0 outside the F0 range of training data. To alleviate this issue, in QPPWG pitch-dependent dilated convolution neural networks (PDCNNs), which dynamically changes dilation sizes adapting to pitch, are introduced to PWG. PDCNN facilitates QPPWG to capture the very longterm dependencies of periodic components and makes the pitch of the QPPWG-generated speech more consistent with the auxiliary F0.
For the ordinary dilated convolution neural networks, the dilation size d, is predefined and time-invariant. The dilation sizes dt of PDCNNs are dynamically defined at each time step t as follows: where fs is the sampling rate, f0,t is an F0 value at time t, and a is a hyperparameter called dense factor, which determines the sparsity of the PDCNNs, and empirically set to 4.0 in this paper.

Neural source-filter (NSF)
NSF is a neural vocoder based on source-filter model, and divided into three modules: a condition module, a source module, and a filter module. An F0 sequence f0,1, · · · , f0,T and a spectral feature sequence (e.g. mel-spectrogram) are used as an input of NSF. The condition module upsamples the input features and extracts feature embeddings for the resonance filtering. In the source module, by treating f0,t as the instantaneous frequency, the fixed number of sinusoidal wave basis signals is generated, where the h-th basis signal e (h) t is given by where fs is sampling frequency, φ ∈ [−π, π] is a random initial phase, nt ∼ N (0, σ 2 ) is a Gaussian noise, and h is the scale factor of the fundamental frequency. These signals are merged using a feed-forward network to output the source excitations. The filter module modulates the source signal using multiple stages of dilated convolution and affine transformations similar to those in ClariNet [20]. NSF adopts multi-resolution STFT loss to learn the difference between the output and the target waveform in the spectral domain. Unlike that of PWG, it calculates the mean square error (MSE) for the log power spectrum.

Proposed method: unified source-filter GAN (uSFGAN)
Our proposed method, uSFGAN is based on QPPWG, but differs on the generator in several ways: (1) the generator is explicitly split into a source-network and a filter-network; (2) a sinusoidal signal is used as an additional input of the sourcenetwork, and (3) a regularization term for the output of the source-network is added to the auxiliary loss. On the other hand, the discriminator is the same as that of QPPWG.

Network architecture
As the proposed architecture shown in Fig.1, the generator of uSFGAN receives random noise z sampled from Gaussian distribution, an F0 sequence f , and an auxiliary feature sequence c as the input, where f and c are assumed to be extracted per frame. A sinusoidal signal v = v1, · · · , vT is first generated on the basis of upsampled f as follows: where f0,t is the instantaneous frequency at time t, and fs is the sampling frequency. Unlike QPPWG adopting only noise inputs, the sinusoidal signal input is used to make the estimation of the harmonic components easier and improve the learning efficiency of the proposed source-network. Then, v is combined with z as a two-channel input of the source-network. The source-network performs the pitch-dependent dilated convolution conditioned on the upsampled auxiliary features c to output the source excitation signalê. The generated source excitation signal is used as the input to the filter-network and is also used to calculate the spectral envelope regularization loss, which is used for the auxiliary loss, as mentioned in Section 3.2. In the filter-network, non-causal dilated convolution with fixed dilation sizes is performed. The output waveform is input to the discriminator and also used to calculate the multi-resolution STFT auxiliary loss.

Spectral envelope regularization loss
To encourage the source-network to output a reasonable source excitation signal, one constraint is imposed on the output of the source-network. As in the traditional source-filter vocoders, such as STRAIGHT [29] and WORLD [30], we assume that the spectral structure of the source excitation signal consists of harmonic components and stochastic components, and its spectral envelope is flat, i.e., the power of spectral envelope is constant over all frequency. We adopt regularization to satisfy this assumption on the spectral envelope of the source excitation signal in the output of the source-network. We use a simplified algorithm of cheaptrick [33] to extract the spectral envelope from the output signal of source-network. The original algorithm of cheaptrick is composed of three steps: (1) F0 adaptive windowing and calculation of log power spectrum, (2) F0 adaptive smoothing in the spectral domain, and (3) F0 adaptive liftering in the cepstrum domain. In order to speed up the spectral envelope estimation process, we apply several modifications to the cheaptrick algorithm. First, we directly use the F0 values given as the auxiliary feature f rather than extracting F0 values from the output signal. Those F0 values are further rounded to integers, and then, the corresponding windows and liftering functions are obtained in advance. Moreover, the step (2) is omitted because this process requires a relatively large processing time and the F0 adaptive spectral envelope extraction is still performed by the F0 adaptive liftering in the step (3). Although these modification causes slight degradation of the spectral envelope estimation accuracy, it doesn't cause any significant issues as the precise spectral envelope estimation is not necessary for the regularization.
The spectral envelope regularization loss is given by whereÊ (n) k is the k-th frequency component of log power spectral envelope extracted fromê at n-th time frame by the simplified cheaptrick algorithm. Note that when this loss reaches to 0, linear power values of the spectral envelope are 1 over all frequency and time frames.

Training criteria
The same adversarial losses as QPPWG but different auxiliary losses are adopted in uSFGAN training. Our method uses two types of auxiliary losses: multi-resolution STFT loss and the spectral envelope regularization loss. The STFT loss is defined as follows: where Y n k andŶ n k are the k-th STFT component at the n-th time frame of a natural waveform and the output waveform. Re and Im denote real part and imaginary part, respectively. This STFT loss is different from that of PWG [3], but the same as that of NSF [21,22]. Finally, our auxiliary loss is represented as follows: where M is the number of STFT losses using various STFT parameters, and λreg is a hyperparameter balancing the two auxiliary losses and empirically set to 1.0 in this paper.

Experimental evaluation 4.1. Experimental conditions
To investigate the effectiveness of our proposed method, we compared four different models: publicly available pretrained NSF (hn-sinc-nsf-9 [34]) model referred to as PT-NSF, NSF with WORLD features referred to as WORLD-NSF, QPPWG, uSFGAN, and uSFGAN without the spectral envelope regularization loss. We adopted F0 conversion to evaluate the controllability.
For the training data, we used 4000 utterances from CMU-ARCTIC database [35] consisting of more than 1000 utterances each from four speakers: slt, bdl, clb, and rms. We used a set of 264 utterances consisting of 66 utterances from each speaker as validation data and another set of 264 utterances as test data. The sampling frequency was set to 16000 Hz by downsampling.
WORLD-NSF used the same architecture as PT-NSF. QP-PWG used 10 adaptive blocks and 10 fixed blocks as the optimized setting. uSFGAN used 30 adaptive blocks for the sourcenetwork, and 30 fixed blocks for the filter-network. uSFGAN was trained with the RAdam optimizer [36] ( = 10 −6 ) with 400 k iterations as in QPPWG. The generator of uSFGAN is trained with only auxiliary loss in the first 100 k iterations, and then trained with the adversarial loss as well as the auxiliary loss in the remaining 300 k iterations. The parameter settings of the multi-resolution STFT loss are shown in Table 1.
In PT-NSF, F0 extracted by YAAPT [37] and mel-spectrum extracted by STFT-based method were used for the input. In the other models, F0, spectral envelope, and aperiodicity extracted by WORLD [30] were used. The window length was set to 64 ms and the shift length was set to 5 ms in all models. The spectral envelope was parameterized into 25-dimensional mel-cepstral coefficients, and aperiodicity was coded into 1dimension. The unvoiced/voiced intervals were represented as a binary feature. For PT-NSF, the target speech used for training was normalized after feature extraction. On the other hand, for the other models, no normalization process was applied to the target speech.

Objective evaluations
As objective evaluation indexes, we used root mean square error of log F0: RMSE [Hz], unvoiced/voiced decision error: U/V   Table 2. It shows that uSFGAN tends to generate speech well conveying the information of the given auxiliary features with higher accuracy than PT-NSF, WORLD-NSF, and QPPWG. In addition, the spectral envelope regularization loss significantly improves the F0 transformation accuracy in uSFGAN. An example of spectrograms and waveforms of the output source excitation signals from uSFGAN with and without the spectral envelope regularization loss are shown in Figs. 2 and 3. We can find that the use of the spectral envelope regularization loss is effective for generating a reasonable source excitation signal of which spectral envelopes are flat and both harmonic components and periodic waveform shape well correspond to F0 values.

Subjective evaluations
We conducted an opinion test on speech quality. Natural speech and synthetic speech from WORLD and three models: PT-NSF, QPPWG, and uSFGAN were evaluated by 10 subjects. We evaluated 160 utterances per each method per F0 scaling factor. The synthetic speech was generated by scaling F0 values by 1.0, 2.0 and 0.5 times.
As the experimental result, mean opinion scores (MOS) are shown in Table 3. It shows that uSFGAN significantly outperforms WORLD, PT-NSF, and QPPWG in both 1.0 × F0 and 0.5 × F0. In 2.0 × F0, although WORLD still achieves the best speech quality, uSFGAN achieves comparable speech quality to PT-NSF and significantly better speech quality than QPPWG.
To investigate the controllability, we conducted an ABX test to evaluate the perceptual accuracy of F0 conversion by 10 subjects. We evaluated 100 utterances per each method per F0 scaling factor. The synthetic speech by WORLD was used as a   reference speech for three models: PT-NSF, QPPWG, and uS-FGAN. The subjects selected a speech sample of which pitch was closer to that of the reference speech.
The experimental results are shown in Table 4. Each entry in the table shows the number of times selected as closer to the reference pitch in each pair comparison from the three models. It shows that uSFGAN significantly outperforms the other models in both 2.0 × F0 and 0.5 × F0. Our samples can be found on our demo page [38].

Conclusions
In this paper, we have proposed a unified neural vocoder framework based on the source-filter model, which is called unified source-filter GAN. In the proposed neural vocoder, a sinusoidal signal as the additional input, the pitch-dependent dilated convolution, and spectral envelope regularization loss have been implemented for factorizing the overall network into the sourcenetwork and the filter-network. The experimental evaluation results have demonstrated that the proposed neural vocoder can significantly improve F0 transformation accuracy while achieving high speech quality. Thus, uSFGAN can be applied to entertaining TTS by combining it with an acoustic model that outputs vocoder features.

Acknowledgements
This work was supported in part by JST, CREST, and JP-MJCR19A3.