{'fraction_non_alphanumeric': 0.02668287318400185, 'fraction_numerical': 0.009087225791514731, 'mean_word_length': 5.292061179898034, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
AILTTS: Adversarial Learning of Intermediate Acoustic Feature for End-to-End Lightweight Text-to-Speech

The quality of end-to-end neural text-to-speech (TTS) systems highly depends on the reliable estimation of intermediate acoustic features from text inputs. To reduce the complexity of the speech generation process, several non-autoregressive TTS systems directly find a mapping relationship between text and waveforms. However, the generation quality of these system is unsatisfactory due to the difficulty in modeling the dynamic nature of prosodic information. In this paper, we propose an effective prosody predictor that successfully replicates the characteristics of prosodic features extracted from mel-spectrograms. Specifically, we introduce a generative model-based conditional discriminator to enable the estimated embeddings have highly informative prosodic features, which significantly enhances the expressiveness of generated speech. Since the estimated embeddings obtained by the proposed method are highly correlated with acoustic features, the time-alignment of input texts and intermediate features is greatly simplified, which results in faster convergence. Our proposed model outperforms several publicly available models based on various objective and subjective evaluation metrics, even using a relatively small number of parameters.


Introduction
The advancement of neural networks has led to significant quality improvement in text-to-speech (TTS) systems. Most neural TTS models consist of a two-stage pipeline architecture: an acoustic model that converts text or phonemes to intermediate features such as mel-spectrograms, and a neural vocoder that synthesizes waveforms by utilizing the intermediate features as condition. Depending on the structural differences that occur while converting text into intermediate features, we can classify neural TTS models as either an autoregressive (AR) or a non-autoregressive (non-AR). AR-based methods [1,2,3] successfully generate high quality speech, but their inference speed is low because they cannot be implemented in parallel. To solve the low inference speed problem of AR-based methods, various types of non-AR TTS models have been proposed [4,5,6,7].
Recently, non-AR TTS models have been extended to contain a single-stage end-to-end structure that incorporates the generative model-based neural vocoder into the training process [5,8,9,10]. This structure trains entire text-to-speech conversion chains under a unified framework. Thus, it enables the flexible usage of intermediate latent domain feature characteristics instead of constraining them to track the characteristics of signal processing-based features such as the mel-spectrogram. In this unified framework, it is more important to reliably estimate trainable intermediate acoustic features from text inputs because the features must have timing information corresponding to the duration of each text. In addition, they are required for conditioning the generative model-based waveform synthesis module. Utilizing the trainable characteristic of intermediate features, several unified TTS models that exclude acoustic feature decoding have been proposed [5,8,10]. However, the synthesized speech quality of such models is unsatisfactory (e.g., monotonous and unnatural) because: 1) it is difficult to train intermediate features to sufficiently represent prosodic variations due to their structural limitations while guiding acoustic features, and 2) the following neural vocoder can not be trained to fully utilize the intermediate features from the entire utterance due to a GPU memory constraint; thus, its receptive field size should be very limited.
In this paper, we propose a method for reliably estimating informative intermediate acoustic features from text inputs. We solve the problems of the insufficient prosodic variation and limited receptive field of the neural vocoder by strengthening the role of the trainable intermediate features. We adopt an effective prosody predictor that powerfully models the complicated distribution of prosody-related embeddings in the reference mel-spectrogram using an adversarial training process. We also show that the obtained prosody embeddings are very effective to perform the likelihood-based internal time alignment between text and intermediate acoustic features. Our alignment method converges quickly during training and always shows robust alignment performance. Our experiments confirm that our method significantly enhances the expressiveness and quality of generated speech. The generated audio samples for the experiments are also available at our demo page 1 .
This paper is organized as follows. Section 2 analyzes the main concepts and problems of current single-stage models. Section 3 proposes our approach to achieve high-quality speech generation, Section 4 presents our experimental results, and Section 5 concludes the paper.

Background
Since the length of intermediate acoustic features differs over time according to the phonetic variation of input text, we need to predict the duration of each text as well as its acoustic characteristics. In common non-AR TTS systems, text-level embeddings are expanded at each time duration to match their time scale with target intermediate acoustic features. With this objective in mind, [4] adopted an external signal processing-based aligner [11] to obtain reference time duration, and [6,12] proposed likelihoodbased aligners that internally estimate the duration by selecting the most probable path from a learned alignment map.
Also, to efficiently synthesize waveforms with neural vocoders, we must provide rich and dynamic prosody information to intermediate acoustic features. From this perspective, sev- eral single-stage models that perform the aforementioned tasks without explicitly decoding target mel-spectrograms have been proposed. VITS [9] adopts variational inference with normalizing flows to model the complicated probabilistic distribution of the latent domain intermediate features. Other single-stage models [5,8,10] simplify the above process by utilizing their own trainable intermediate latent features; however, their generation quality is not on par with two-stage models [5,8].
In this paper, we use LiteTTS [10] as a baseline because it has low complexity and relatively high synthesis quality. LiteTTS adopts a domain transfer technique that approximates text inputs to clone prosody-related information extracted from the target mel-spectrogram during training. Text and prosodyrelated embeddings are combined first, and then intermediate features are derived by upsampling following reference durations given by the external aligner [11]. However, since 1) they simply minimize the L1 distance between the target and predicted prosody-related embeddings, and 2) the following neural vocoder can not be trained receiving the entire intermediate features, the output quality is somewhat over-smoothed and unsatisfactory. To overcome these problems without increasing model parameters, we propose an adversarial training method with a conditional discriminator to reinforce the estimation of the complex distribution of prosody embedding included in the intermediate feature.
We also adopt a likelihood-based internal aligner conditioned on the prosody embedding, which robustly and quickly performs alignment without using an external aligner. estimation module that generates the target of the prosody predictor, a prosody predictor that predicts hpr from text (phonetic) embeddings (h ph ), and an internal aligner including a duration predictor whose target is a learned alignment between text and acoustic features. To simplify the architecture with low complexity, we adopt a lightweight transformer-based encoder [10] for the phoneme encoder, prosody encoder, and prosody predictor.

Prosody predictor with conditional discriminator
We generate the intermediate acoustic feature I by upsampling the combined phoneme-scale embeddings h ph + hpr following the duration given by the internal aligner. Here, the prosody embedding hpr is calculated through mapping the prosody encoder output h pr to phoneme-scale by using attention with h ph . The main purpose of the prosody predictor is to generate an embedding that can be successfully replace the target prosody embedding hpr from input h ph .
Considering the dynamic nature of the prosody embedding, we leverage an effective generative model architecture that includes various types of discriminators. The proposed discriminator distinguishes the target prosody embedding hpr from the predicted embedding h pred and utilizes phonetic information as conditional data. We adopt a projection-based conditional discriminator [13] by conditioning the phonetic embedding h ph as shown in Fig. 2. We apply feature matching loss [14] between the generated and target feature maps to stabilize the training of generative adversarial network (GAN). In Fig. 2, the feature maps are obtained as the output of all the 1-D convolution layers prior to the P ostConv1D layer. Seven feature maps are extracted in total, one from P reConv1D layer and six from three residual 1-D convolutional blocks.
Since audio waveforms are generated by the vocoder using intermediate acoustic features, we designed the conditional discriminator as follows: 1) we use upsampled embeddings instead of original phoneme-scale embeddings as input to the discriminator to match the time-scale with intermediate acoustic features, and 2) since the intermediate features have short-term dependencies lower than the receptive field of the vocoder, we also carefully design the conditional discriminator to have the same receptive field size as the vocoder, which enables the discriminator to efficiently capture the diverse patterns of the input prosody embeddings. Based on the generative and discriminative loss of least-squares GAN [15], we define the total prosody predictor loss by considering both reconstruction loss Lrecon and feature matching loss L f m : where H (·) denotes the upsampled embeddings, and F i denotes the i-th feature map of the discriminator.

Prosody conditioned internal aligner
To learn the time alignment between phonetic and intermediate acoustic features without using an external aligner, we adopt a likelihood-based method [6,12]. That is, we maximize the loglikelihood of monotonic alignments from a probability matrix calculated by the L2-distance between two encoded features with different time scales. Then, phoneme durations (binary matrix) are obtained by selecting the most probable path from the probability matrix. The gap between the two matrices is reduced by minimizing their KL-divergence.
Our proposed aligner utilizes the phoneme-scale embeddings h ph + hpr for the phonetic features and the mel-spectrogram for the acoustic features. Since hpr contains local acoustic information mapped into phoneme-level by a formerly applied attention module, it becomes much easier for the aligner to learn the time relationship between two features than when using only h ph . Following [4], we jointly train the duration predictor which accepts the phonetic embedding h ph with a stop gradient. We show that the accuracy of our text-to-acoustic feature alignment increases when we use our proposed prosody embedding as a condition, which also improves the accuracy of the duration predictor.

Final training loss
The total loss for training our model is defined as follows: L total = Lvar + L align + L pred + Lvoc + Laux, (4) where Lvar includes pitch and energy prediction losses applied to h pr in [10], and L align includes loss functions for the internal aligner, including the duration predictor. L pred and Lvoc represent total losses from the prosody predictor and the vocoder, respectively, in accordance with their GAN structure. Laux is defined by the L1 loss between the target mel-spectrogram and the predicted mel-spectrogram obtained by the output of the auxiliary predictor.

Auxiliary predictor
The role of the auxiliary predictor is to further provide acoustic information to the intermediate feature. Its architecture is almost identical to that marked '*' in Fig. 2, with the exception that the input is the intermediate feature. The output channel of the P ostConv1D layer is set to the dimension of the melspectrogram instead of 128, and layer normalization is applied to the last stage of every residual 1-D convolutional block. In this way, we also match the receptive field with the vocoder because its input is also the intermediate feature, which enables to efficiently provide acoustic information. Since the auxiliary predictor is only used for the training stage, the computational complexity and memory usage are not increased.

Experimental Setup
Dataset and Model Settings We evaluated our model on the LJSpeech dataset [16]. We converted text sequences into phoneme sequences with an open-source tool 2 . To calculate spectrograms, the FFT size, window size, hop size were set to 1024, 1024, and 256, respectively, regarding the sampling rate of 22050Hz. Mel-spectrograms were obtained by applying an 80 band mel filter bank. We adopted Univnet-c16 [17] as a vocoder, which has beneficial lightweight properties by using a locationvariable convolution (LVC) technique [18]. The dimension of all hidden embeddings was set to 256, and the receptive field of the vocoder, auxiliary predictor, and the conditional discriminator was set to 19. Training Configuration and Inference We trained our model with the Adam [19] optimizer using β1 = 0.8, β2 = 0.99 under the learning rate of 2e − 4 and its decaying factor of 0.999. The batch size was set to 24. The entire model was trained in a fully end-to-end manner, except for the prosody and duration predictors. Since the training objectives of two modules (hpr and phoneme duration) do not converge during the early stage, they are jointly trained with other modules after 300k steps. For the prosody predictor loss, only reconstruction loss was applied for the first 50k steps (until 350k steps) without applying feature matching loss and adversarial loss to stabilize the training of the prosody predictor. During inference, the prosody embedding and duration values predicted from the above two modules were used to generate the intermediate acoustic feature. Models for Comparison We compared our model with both single-stage and two-stage models. We first experimented with LiteTTS by replacing the HiFi-GAN V3 [20] vocoder with the  Figure 3: Comparison of the convergence rate between a conventional method (using h ph as input to the aligner) and a prosody conditioning method (using h ph + hpr as input to the aligner). For each iteration, the MCD-DTW values from 10 test samples are averaged.
UnivNet-c16 vocoder for a fair comparison. For the two-stage models, we adopted Tacotron 2 and Glow-TTS as baselines, two popular AR and non-AR TTS models, respectively, combined with the UnivNet-c16 vocoder. Here, we trained the vocoder using 80-dim mel-spectrogram as input, and utilized pretrained weights for the acoustic models 3 .

Results
Audio Quality Table 1   prosody predictor relieves the over-smoothing problem that occurs when using L1 loss only, and enhances the expressiveness of generated speech. As shown in the second ablation experiment, it is demonstrated that providing phoneme-level acoustic information to the aligner helps the model to obtain robust alignment and results in improvement of generation quality. As a result, we conclude that both methods improve the model's generation of various prosody characteristics of speech, and increase the perceptual naturalness of the generated audio. Alignment convergence We also compared the alignment convergence rate of the two experiments from the ablation study of the prosody conditioned aligner. To check this, we calculated the mean mel-cepstral distance (MCD) between synthesized and ground-truth audio samples after synchronizing them in time with dynamic time warping (DTW) [21]. As shown in Fig 3, the MCD-DTW values of the proposed aligner rapidly decrease during the early stage of training (prior to 4k iterations), and continuously show higher performance (lower MCD-DTW values) than the baseline in every iteration. As we also demonstrated in the second ablation study, we verify that our method has robust alignment performance and rapidly converges in the learning process.

Conclusion
In this paper, we propose an adversarial learning method to estimate intermediate acoustic features, which plays a significant role in generating natural and expressive speech in the single-stage lightweight TTS system. Equipped with a prosody conditioned aligner, our proposed model outperforms the Glow-TTS based model (+0.36 MOS) and LiteTTS (+0.22 MOS) while preserving lightweight properties. As future work, we will design a generative model-based stochastic prosody predictor to further extend prosody controllability. In addition, we will extend our proposed aligner to have a capability of generating multi-speaker's speech signals by conditioning speakerdependent features.

Acknowledgments
This work is supported by Clova Voice, NAVER Corp., Seongnam, Korea.