{'fraction_non_alphanumeric': 0.03618894672721232, 'fraction_numerical': 0.014093476575479012, 'mean_word_length': 4.653882279273638, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Guided-TTS 2: A Diffusion Model for High-quality Adaptive Text-to-Speech with Untranscribed Data

We propose Guided-TTS 2, a diffusion-based generative model for high-quality adaptive TTS using untranscribed data. Guided-TTS 2 combines a speaker-conditional diffusion model with a speaker-dependent phoneme classifier for adaptive text-to-speech. We train the speaker-conditional diffusion model on large-scale untranscribed datasets for a classifier-free guidance method and further fine-tune the diffusion model on the reference speech of the target speaker for adaptation, which only takes 40 seconds. We demonstrate that Guided-TTS 2 shows comparable performance to high-quality single-speaker TTS baselines in terms of speech quality and speaker similarity with only a ten-second untranscribed data. We further show that Guided-TTS 2 outperforms adaptive TTS baselines on multi-speaker datasets even with a zero-shot adaptation setting. Guided-TTS 2 can adapt to a wide range of voices only using untranscribed speech, which enables adaptive TTS with the voice of non-human characters such as Gollum in \textit{"The Lord of the Rings"}.


Introduction
Neural speech synthesis models have shown to produce natural samples [39,44,46]. Recent two-stage TTS models as well as end-to-end TTS models generate high-quality speech in real time. [6,12,21,22,23,33,36,37]. However, these high-quality TTS models assume that sufficient amounts of large-scale data for the target speaker are given, which is a large constraint. Adaptive TTS models aim to generate high-quality speech for the target speaker given a limited amount of reference data. In general, the more reference data is given, the better adaptive TTS models adapt to the target speaker, and one of the main challenges is to reduce the amount of reference data while preserving adaptation quality and naturalness.
Most adaptive TTS models learn to generate speech from various speakers using a multi-speaker dataset and adapt to new speakers through various methods. There have been many zero-shot adaptive TTS models using a pre-trained speaker verification model to extract the speaker embedding from a short reference speech or their own speaker encoder to extract more generalizable speaker representation [3,4,9,47]. Fine-tuning based adaptive TTS models have also been proposed to update part of pre-trained multi-speaker TTS models using a small amount of reference speech [4,5,7,49]. Min et al. [29] propose meta-learning based adaptation using short-duration reference speech. However, when the amount of the reference speech is only about 10-seconds-long, existing adaptive TTS models show poor sample quality and speaker similarity compared to single-speaker high-quality TTS models trained on a single-speaker TTS dataset, such as LJSpeech.
In this work, we present Guided-TTS 2, which introduces a diffusion-based generative model for adaptive TTS with a small amount of untranscribed data. We train a speaker-conditional DDPM on large-scale untranscribed datasets to model the distribution of speech for various speakers. Similar to , Guided-TTS 2 performs text-to-speech synthesis by guiding the sampling process of the speaker-conditional DDPM with the pre-trained phoneme classifier. We adapt our pre-trained diffusion model to the target speaker by using the classifier-free guidance method and fine-tuning directly on the reference speech data. Although the fine-tuning approach requires additional training time for adaptation, Guided-TTS 2 significantly reduces the fine-tuning cost to less than a minute on top of our speaker-conditional diffusion model's adaptability.
We demonstrate that Guided-TTS 2 fine-tuned with a ten-second reference speech of LJSpeech obtains comparable sample quality and speaker similarity to the single-speaker high-quality TTS baselines trained on 24-hour LJSpeech. Furthermore, Guided-TTS 2 outperforms the adaptive TTS baselines, YourTTS and Meta-StyleSpeech, for the LibriTTS and VCTK datasets. As our speakerconditional DDPM adapts without transcript, Guided-TTS 2 enables adaptive TTS with audio clips that are difficult to transcribe, such as the voice of Gollum in "The Lord of The Rings". We highly encourage readers to listen to various samples of Guided-TTS 2 shown in Fig. 1 on our demo page. 3 2 Background 2.1 Score-based Generative Model Song et al. [43] introduce a unified approach of denoising diffusion probabilistic model (DDPM) [15,40] and score matching with Langevin dynamics (SMLD) [41,42]. We follow the formulation of Grad-TTS [32], which applies Song et al. [43] to TTS. Popov et al. [32] define the diffusion process, which can convert any data distribution to standard normal distribution where β t is the pre-defined noise schedule β t = β 0 + (β T − β 0 )t and W t is the Wiener process. Anderson [1] proves that the reverse process, which follows the reverse trajectory of the diffusion process, can be formulated in SDE. Popov et al. [32] use the following equation, which is a discretized version of the reverse SDE, during sampling to generate data X 0 from standard Gaussian noise X T where N is the number of steps in the discretized reverse process, and z t is the standard Gaussian noise. As Popov et al. [32] set T to 1, the size of one step is 1 N , and t ∈ { 1 N , 2 N , ..., 1}. As can be seen in Eq. 2, ∇ Xt log p t (X t ), called the score s(X t ), is required for sampling procedure. To predict the score, data X 0 is first corrupted into X t at an arbitrary timestep t using t ∼ N (0, I) and Eq. 1. Afterwards, a neural network s θ is trained using the following loss to estimate the score where λ(t) = I − e − t 0 βsds . By minimizing this loss, the model s θ is trained to compute the score, which is essential for generating data from noise.

Diffusion Guidance Methods
Recently, classifier guidance [10, 43] and classifier-free guidance [14] are proposed for guiding the diffusion model. The discretized reverse process for each method can be obtained by replacing the unconditional score s(X t ) in Eq. 2 with the modified conditional scoreŝ(X t |y), which is defined differently depending on the approach. Song et al. [43] show that it is possible to generate conditional samples using unconditional diffusion model by leveraging the separately trained classifier. Given the unconditional score and the gradient of the log-density of the classifier ∇ Xt log p t (y|X t ), the conditional score s(X t |y) can be estimated by adding the unconditional score and the classifier gradient. Conditional samples can then be generated by setting the modified conditional score equal to the conditional score as in Eq. 4.ŝ

Classifier Guidance
Classifier guidance can also be utilized to improve conditional diffusion models. Dhariwal and Nichol [10] provide an additional classifier gradient to the conditional score, and the modified conditional score accordingly is as follows: Dhariwal and Nichol [10] show that sample quality can be improved at the expense of diversity if more classifier gradient is provided by increasing the gradient scale γ. By adjusting the gradient scale for the optimal value, they achieve state-of-the-art performance in conditional image generation.
Classifier-free Guidance Unlike classifier guidance, classifier-free guidance [14] does not require a classifier to guide a diffusion model toward a target condition. By learning the conditional and unconditional scores with a single model, we can calculate the classifier gradient by subtracting the unconditional score from the conditional score. During training, the condition y is sometimes replaced with the null label φ, making it possible to jointly train unconditional and conditional models. The modified conditional score for classifier-free guidance is as follows: s(X t |y) = s(X t |y) + γ · (s(X t |y) − s(X t |φ)).
According to GLIDE [30], classifier-free guidance has two main advantages. The first advantage is that classifier-free guidance enables guiding the diffusion model with only a single model. Second, the diffusion model can be guided even toward conditions that are difficult to predict using a classifier such as text. In GLIDE, diffusion model with classifier-free guidance achieves better performance than with CLIP-based [34] classifier guidance in text-to-image generation. Similarly, DALL·E 2 [35] shows powerful performance in text-to-image generation task using CLIP and classifier-free guidance.

Guided-TTS
Guided-TTS [20] is a model that constructs high-quality TTS without any transcript of the target speaker. Kim et al. [20] train an unconditional DDPM using the untranscribed speech and guide it using a phoneme classifier and duration predictor trained on a large-scale multi-speaker ASR dataset. Kim et al. [20] provide speaker embedding extracted from the speaker encoder to the phoneme classifier and duration predictor, making both modules speaker-dependent. These speaker-dependent modules are generalizable to various unconditional models of unseen speakers, enabling a single phoneme classifier and duration predictor to guide multiple unconditional DDPMs. Kim et al. [20] observe that applying the existing classifier guidance methods [43,10] to text-tospeech cannot guarantee accurate pronunciation regardless of the gradient scale. In order to generate speech that faithfully reflects the given text, Kim et al. [20] propose norm-based guidance, a classifier guidance method of scaling the norm of the classifier gradient accordingly to the norm of the score, which is as follows: With the norm-based guidance, Kim et al. [20] show that Guided-TTS is comparable to other TTS models in terms of audio quality and pronunciation accuracy, showing the possibility of building a high-quality TTS model using untranscribed speech. In this section, we present Guided-TTS 2, which aims to adapt new speakers with a small amount of untranscribed data. Extending Guided-TTS to adaptive TTS, we train a speaker-conditional DDPM on multi-speaker untranscribed datasets and guide the diffusion model with the phoneme classifier using classifier guidance. Based on the pronunciation accuracy of the norm-based guidance in Guided-TTS, we focus on the speaker-conditional DDPM for estimating the conditional score for new speakers.
We pre-train our speaker-conditional DDPM on large-scale multi-speaker datasets for speaker classifier-free guidance. For better adaptation to new speakers, we further fine-tune the diffusion model using the untranscribed reference speech of the target speaker. Since Guided-TTS 2 adapts only with the untranscribed speech, Guided-TTS 2 can enable adaptive TTS using challenging reference speeches that may contain auditory contents untranscribable in the desired language.
In Section 3.1, we introduce the speaker-conditional DDPM and speaker classifier-free guidance. We present the details of fine-tuning for adaptation to new speakers in Section 3.2, and describe components of Guided-TTS 2 and sampling process for adaptive TTS in Section 3.3.

Speaker-conditional Diffusion Model
Our speaker-conditional DDPM estimates speaker-conditional score of speech for various speakers without any transcript. For zero-shot adaptation ability to target speakers, we use a pre-trained speaker verification model as our speaker encoder E S like Jia et al. [18], which encodes speaker information of untranscribed data.
Given a mel-spectrogram X = X 0 in a large-scale unlabeled dataset, let us define a forward diffusion process of X t , t ∈ [0, 1] as in Eq. 1. Our speaker-conditional DDPM learns to estimate conditional score ∇ Xt log P θ (X t |S) for speaker S that is required for the reverse denoising process. We use the same U-Net architecture [38] of unconditional DDPM in Guided-TTS [20] as our speaker-conditional DDPM s θ . For conditioning speaker information, we extract a speaker embedding e S = E S (X) ∈ R d from the mel-spectrogram X, and concatenate speaker embedding e S with timestep t for speaker conditioning. The training objective of speaker-conditional DDPM s θ is as follows: We further introduce a classifier-free guidance [14] to augment the speaker condition from the speaker embedding. For speaker classifier-free guidance, as in GLIDE [30], we introduce the null embedding e φ ∈ R d and additionally train the speaker-conditional DDPM to estimate both the conditional and the unconditional score. At every iteration, we replace the speaker embedding e S of each data with a null embedding e φ with a 50% chance to output the unconditional score s θ (X t |φ). As our speaker embedding is set to have a unit norm, we define the null embedding using additional trainable parameter w ∈ R d as e φ = w w for consistency. For zero-shot adaptation, we extract speaker embedding eŜ from the reference untranscribed data of the target speakerŜ and provide speaker embedding eŜ to the diffusion model for adaptation. We augment the speaker condition of the target speaker by setting the classifier-free scale γ S to be larger than 0 and compute the modified conditional scoreŝ θ (X t |Ŝ) as in Eq. 9.

Fine-tuning
The zero-shot adaptation method described above relies on the speaker embedding extracted from the reference data to adapt to the target speaker. To better adapt to the target speaker, we directly fine-tune the pre-trained speaker-conditional DDPM on the reference data to estimate the speaker-conditional score of the target speaker. While most existing adaptive TTS models require more than a minute of reference data for fine-tuning, we found that speaker-conditional DDPM trained on a large-scale multi-speaker dataset can adapt surprisingly well to the target speaker even only with a ten-second reference data. However, if speaker-conditional diffusion overfits to the single reference data, the diffusion model loses the pronunciation ability learned from the multi-speaker dataset and is difficult to use for TTS synthesis. Therefore, we fine-tune the diffusion model for a small number of iterations with a small learning rate so as not to overfit to the single reference data.
Given the reference untranscribed data for the target speakerŜ, we first extract the speaker embedding eŜ from the reference data. Using the reference data, we optimize the diffusion model through the Adam optimizer with a learning rate of 2 · 10 −5 , which is lower than the learning rate 10 −4 used for pre-training. We found that the statistics of the pre-trained optimizer degrade the adaptation performance. Thus, we reuse only the pre-trained weights of the diffusion model and initialize the optimizer for fine-tuning. We fine-tune the speaker-conditional DDPM to estimate only the conditional score for the target speaker as in Eq. 8.
We update the diffusion model by 500 iterations, to the extent that the model adapts well to various target speakers while preserving pronunciation ability. The following training time is about 40 seconds on a NVIDIA RTX 8000 GPU, which is small enough to build a TTS model for the target speaker.

Adaptive Text-to-Speech
In this section, we explain each module of Guided-TTS 2 and describe how Guided-TTS 2 performs adaptive TTS using reference speech. Similar to Kim et al. [20], Guided-TTS 2 consists of a speakerconditional DDPM, a phoneme classifier, a duration predictor, and a speaker encoder, as shown in Fig.  2. The architecture of each module is detailed in Section A.1. The phoneme classifier predicts the phoneme sequence frame-wise from the noisy mel-spectrogram X t , and the duration predictor outputs the duration of each phoneme. All modules use speaker embedding as a condition for generalization to new speakers. In general, we use a fine-tuned diffusion model in Section 3.2 for adaptation except for zero-shot adaptation, which we use the pre-trained speaker-conditional DDPM described in Section 3.1.
Let us assume that the reference data for the target speakerŜ and the phoneme sequence y to generate are given. We first predict the duration of each phoneme with the duration predictor, and expand the phoneme sequence y to frame-level phoneme sequenceŷ with the predicted duration. Then, we sample the noise X 1 having the same length as the frame-level phoneme sequence from the standard Gaussian distribution for the reverse process of DDPM. In order to perform adaptive TTS, we replace s(X t ) in Eq. 2 with the modified conditional scoreŝ θ (X t |ŷ,Ŝ) for the target speakerŜ and the frame-level phoneme sequenceŷ during sampling.
To compute the modified conditional scoreŝ θ (X t |ŷ,Ŝ), we combine the speaker classifier-free guidance in Eq. 9 and the norm-based guidance for text in Eq. 7. At each timestep in the reverse sampling process, we obtainŝ θ (X t |Ŝ) from the speaker-conditional DDPM through the speaker classifier-free guidance, and compute the classifier gradient ∇ Xt log p t (ŷ|X t ,Ŝ) from the pre-trained phoneme classifier. Withŝ θ (X t |Ŝ) and ∇ Xt log p t (ŷ|X t ,Ŝ), we compute the modified conditional score for the target speaker and the phoneme sequence using the norm-based guidance as follows: where γ T is the text gradient scale for the norm-based guidance method. . That is, we use reference data of 11 speakers, 7 women and 4 men from each accent, from VCTK, and reference data of 10 speakers from subset test-clean of LibriTTS. In addition, we perform TTS with Guided-TTS 2 using real-world reference audio. We extract 10-second audio clips of several speakers from Youtube and use them as reference clips. As we do not have licenses for real-world scenarios to re-distribute, we instead provide Youtube links for each reference audio.
Training and Fine-tuning Details We train the model using the same architectures and hyperparameters as Guided-TTS except for the speaker-conditional diffusion model. As in Guided-TTS, we convert text into International Phonetic Alphabet (IPA) phoneme sequences using open-source software [2] and convert the speech into mel-spectrogram with the same hyperparameters as Glow-TTS [21]. We extract the alignment of speech and transcript required for training the phoneme classifier and duration predictor using Montreal Forced Aligner (MFA) [28]. The phoneme classifier, duration predictor, and speaker encoder are all trained in the same way as Guided-TTS.
During diffusion model pre-training, we first train speaker-conditional DDPM using LibriTTS for 1M iterations with batch size 256 on 4 NVIDIA RTX 8000 GPUs. We additionally train the pre-trained DDPM for classifier-free guidance in two different ways: (i) using LibriTTS only and (ii) using LibriTTS and Libri-Light. The former case is expressed as Guided-TTS 2 (LT) and the latter case is expressed and Guided-TTS 2 (LT+LL). For both cases, we train the models for 100K iterations.
Evaluation We use Grad-TTS and Guided-TTS as single-speaker TTS baselines. For Grad-TTS, we use the official implementation and the pre-trained model 4 . We set the temperature and the number of reverse steps of Grad-TTS, Guided-TTS, and Guided-TTS 2 to τ = 1.5 and N = 50. For Guided-TTS 2, we set text gradient scale γ T to 0.3 and speaker gradient scale γ S to 1.0. For the vocoder, we use the official implementation and pre-trained model of universal HiFi-GAN. 5 We set the sampling rate of ground truth audio and the samples from single-speaker TTS models to 22,050Hz.
For comparison to adaptive TTS models, we use the official implementation and pre-trained model of zero-shot adaptive TTS models, YourTTS and Meta-StyleSpeech [29]. 6 Unlike YourTTS, which is an end-to-end TTS model, Meta-StyleSpeech requires a vocoder. Therefore, we train universal HiFi-GAN that matches the frontend of Meta-StyleSpeech using an official implementation of HiFi-GAN. Since YourTTS and Meta-StyleSpeech generate speech with a sampling rate of 16kHz, for a fair comparison, we downsample the speech generated by Guided-TTS 2 to 16kHz and then compare it with baselines.
To show the pronunciation accuracy of our model, we measure the character error rate (CER). The CTC-based conformer of the NEMO toolkit [26] trained in 7K hours of speech is used for measuring the CER. To compare the performance of models according to the speaker classifier-free gradient scale γ S , we measure the CER and speaker encoder cosine similarity (SECS). For the SECS measurement, as in YourTTS, we use speaker embedding extracted from the speaker encoder of the Resemblyzer [27] package. We generate each sentence of the test set 5 times, and use the averaged metrics values for each model.

Comparison to Single Speaker TTS Models
To compare the audio quality and speaker similarity with reference audio, we measure the 5-scale mean opinion score (MOS) and the 5-scale speaker similarity mean opinion score (SMOS) using Amazon Mechanical Turk for LJSpeech. In addition, we check the pronunciation accuracy of each model through CER. We randomly select 50 samples from LJSpeech's test set for evaluation. We use 10 pre-selected reference samples for measuring SMOS. For Guided-TTS 2, we generate 5 sentences per reference audio. In Table 1, we compare Guided-TTS 2 with the high-quality single speaker TTS models, Grad-TTS and Guided-TTS. Unlike the baselines, which fully utilize LJSpeech for training, Guided-TTS 2 uses a 10-second randomly chosen reference clip from LJSpeech. Through MOS and CER, we confirm that both Guided-TTS 2 and Guided-TTS 2 (zero-shot) generate high-quality samples with accurate pronunciation regardless of fine-tuning. In particular, Guided-TTS 2 with fine-tuning shows comparable performance in all metrics, including the metric for speaker similarity, to the existing TTS model. These experimental results demonstrate that we can build a high-quality TTS model, even on par with the high-quality single-speaker model, with only 10-second untranscribed data.
We additionally show that existing adaptive models are inferior to single-speaker TTS models while Guided-TTS 2 achieves competitive performance, which is very challenging for the case of 10-5 HiFi-GAN: https://bit.ly/3w9OHf1 6 YourTTS: https://bit.ly/3wbC6I1, Meta-StyleSpeech: https: //bit.ly/3FJMSsg seconds long reference audio. We compare the performance of the single speaker TTS models, the adaptive TTS models, and Guided-TTS 2 for LJSpeech in Section B.1. While the existing adaptive TTS models are far behind in sample quality and speaker similarity compared to the single-speaker TTS models, Guided-TTS 2 showed comparable performance to single-speaker TTS despite being an adaptive TTS model. Samples of all models are on the demo page.
Amount of Data for Training We do not need any transcript to train the speaker-conditional diffusion model of Guided-TTS 2. Therefore, we can use unlabeled datasets such as Libri-Light, which the existing adaptive TTS models cannot utilize. In order to check the performance according to the amount of training data, we compare Guided-TTS 2 (LT) with Guided-TTS 2 (LT+LL), where the former uses LibriTTS for additional training for classifier-free guidance and the latter uses LibriTTS and Libri-Light together. As shown in Table 1, we observe that LibriTTS is sufficient for fine-tuningbased adaptation. We also find that using more untranscribed data is beneficial for our model in zero-shot performance. From this section, we use Guided-TTS 2 (LT+LL) for evaluation, which uses both LibriTTS and Libri-Light for training.

Comparison to Adaptive TTS Models
In this section, we compare our model with adaptive TTS models given a variety of reference audios from diverse datasets by measuring MOS, SMOS, are CER. We select 55 samples and 50 samples from the test set of VCTK and LibriTTS, respectively. We use the same reference data as YourTTS, 11 reference audios for VCTK, each from 11 respective reference speakers, and 10 reference audios for LibriTTS, each from 10 respective reference speakers. We generate 5 sentences per reference audio.   3.94 ± 0.10 2.36 3.19 ± 0.09 Meta-StyleSpeech (Min et al. (2021)) 3.65 ± 0.13 1.84 3.26 ± 0.10 The results of each model for LibriTTS and VCTK are presented in Table 2. As can be seen from the results, Guided-TTS 2 (zero-shot) outperforms other zero-shot baselines in SMOS. This indicates that Guided-TTS 2 can more robustly generate speech similar to the reference for various speakers than other adaptive TTS models with zero-shot adaptation setting. Moreover, we find that adaptation performance improves with fine-tuning in terms of speaker similarity. We also confirm through MOS and CER that Guided-TTS 2 is superior to adaptive TTS models in sound quality and pronunciation accuracy, whether fine-tuning or not. Samples are available on the demo page.

Analysis
Effect of Fine-tuning Comparing the Guided-TTS 2 (zero-shot) and Guided-TTS 2 results in Table  1 and 2, we observe that fine-tuning improves speaker similarity (SMOS) while losing pronunciation accuracy. In other words, as the model is fine-tuned, the sample gets closer to the reference speech when it comes to prosody and timbre while getting further away from the given text. Since the primary purpose of adaptive TTS is to generate a speech that is close to a given reference speaker, we focus on improving the similarity while minimizing the sacrifice of pronunciation accuracy. We compare the CER and SECS according to the number of iterations for fine-tuning in Section B.2. Effects of Speaker Classifier-free Gradient Scale We also show the performance of our model according to the speaker classifier-free gradient scale γ S for LibriTTS. For the fine-tuning setting and the zero-shot adaptation setting, we measure CER and SECS for multiple gradient scales. We check the effect of gradient scale on pronunciation accuracy through CER and on speaker similarity through SECS between embeddings extracted from the speaker encoder. We explore the speaker gradient scale γ S within [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0]. Fig. 3 presents the performance of each model for various gradient scales. As shown in Fig. 3, as the gradient scale increases, the pronunciation accuracy decreases, while the speaker similarity increases slightly. In particular, the pronunciation accuracy of our model with fine-tuning is sensitive to the gradient scale. We find that if the speaker gradient scale is set to γ S ∈ [1, 3], Guided-TTS 2 can generate a high-quality sample with the target speaker's voice and accurate pronunciation. For convenience, we set the default gradient scale γ S to 1.

Text-to-Speech with Real-world Data
To show the performance of Guided-TTS 2 in a real-world scenario, we adapt Guided-TTS 2 to various speakers using real-world audio clips as reference audio. We extract a 10-second audio clip of the desired speaker from Youtube and use it as reference data. With Guided-TTS 2, we can construct a high-quality TTS model using a wide range of voices such as Gollum in "The Lord of the Rings". We highly recommend readers to listen to the various samples on the demo page.

Conclusion
In this work, we present Guided-TTS 2, a high-quality adaptive TTS model with only 10 seconds of the target speaker's unlabeled reference audio. As Guided-TTS 2 does not require any transcript to train the speaker-conditional diffusion model, we can leverage a large amount of untranscribed data. Using the classifier-free guidance and fine-tuning techniques for the diffusion-based model, Guided-TTS 2 can generate samples comparable to single-speaker TTS models with a short reference speech. We showed that Guided-TTS 2 is on par with the single-speaker TTS models for LJSpeech, and outperforms the adaptive TTS baselines for various datasets regarding sound quality and speaker similarity. We also showed that Guided-TTS 2 can generate samples robustly even with real-world clips such as Youtube clips. We believe that Guided-TTS 2 reduces the burden for high-quality TTS by reducing the amount of the target speaker's data.
Despite the advantages of Guided-TTS 2, some limitations are present. Guided-TTS 2 lacks parameter efficiency for each speaker, as all the parameters of speaker-conditional DDPM are fine-tuned for adaptation. As Chen et al. [5] propose, research on fine-tuning only the minimal amount of parameters without degrading adaptation performance can be considered a future work. In addition, as Guided-TTS 2 uses a pre-trained universal vocoder for raw wave generation, the sample quality of Guided-TTS 2 is bounded by the performance of the universal vocoder. When it comes to adaptation to unseen speakers, the sample quality of the universal vocoder is also degraded with respect to the recording environments. In order not to depend on the performance of the external neural network, research on end-to-end adaptive TTS will also be a promising future work.

A.1 Architectures and Training Details
In this section, we describe the training details and architecture of each model not covered in the main text. For all the modules except for the speaker-conditional diffusion model, we use the same structure from Guided-TTS [20], and also train them the same way. Hyperparameters and training details of each module are in Table 3. Phoneme Classifier The phoneme classifier, for which we use the WaveNet [44] architecture, is trained to classify corrupted mel-spectrograms as phoneme labels frame-wise with cross-entropy loss.
Since the length of the phoneme label and the mel-spectrogram should be the same for frame-wise classification, we use the alignment extracted from the Montreal Forced Aligner (MFA) [28] to match the length.

Duration Predictor
The duration predictor is a module that predicts the duration of each phoneme. We train the duration predictor with the L2 loss between the log-scale of the ground truth duration extracted from MFA and the predicted duration. We use a duration predictor with the same architecture as Glow-TTS [21].
Speaker Encoder We utilize speaker embedding for training the speaker-conditional diffusion model, speaker-dependent phoneme classifier, and duration predictor. Our speaker encoder compresses speaker information of mel-spectrogram into speaker embedding e S and is trained with GE2E [45] loss on the speaker verification dataset.

A.2 Hardware and Sampling Speed
We perform all experiments and evaluations using a NVIDIA RTX 8000 with 48GB memory. To check the sampling speed of Guided-TTS 2, we measure the real-time factor (RTF) for generating 8-second-long speech. When we use the classifier-free guidance method, our speaker-conditional DDPM estimates both the conditional and unconditional scores simultaneously, which makes the synthesis speed slower. If we do not use the classifier-free guidance method, the RTF of Guided-TTS 2 is 0.526. With the classifier-free guidance, the RTF of Guided-TTS 2 is 0.802. Even though the primary goal of Guided-TTS 2 is a high-quality adaptive TTS, Guided-TTS 2 enables real-time synthesis.

A.3 Subjective Evaluation
We evaluate the sample quality and the speaker similarity of each model by measuring MOS and SMOS via Amazon Mechanical Turk (AMT). For MOS and SMOS evaluation, we provide reference audio clips for 1, 3, and 5 points to listeners for each assessment, and we strongly recommend wearing headphones to evaluate. When measuring MOS, we provide listeners with the prompt "How natural (i.e., human-sounding) is this recording? Please focus on the audio quality and the naturalness of pronunciation." and the ground truth transcript, and we ask them to evaluate the sample quality of the audio clips for the transcript on a scale of 1 to 5 points. For measuring SMOS, we provide listeners with the prompt "How similar are the speaker in the reference audio and the speakers in each recording? (Please focus on the tone of voice and speaking style rather than audio quality.)" and reference audio from the target speaker and ask them to rate the speaker similarity between the reference audio and provided audio clips on a scale of 1 to 5. We spent a total of around $900 for the subjective evaluations.

B.1 Comparison on LJSpeech
In  4.14 ± 0.08 0.89 3.66±0.07 YourTTS  3.99 ± 0.10 2.76 3.43±0.09 Meta-StyleSpeech (Min et al. (2021)) 3.84 ± 0.11 1.52 3.45±0.09 The performance of each model is in Table 4. Each metric value of YourTTS and Meta-StyleSpeech indicates a significant performance gap between the existing adaptive baselines and single-speaker TTS models. On the other hand, Guided-TTS 2 outperforms adaptive TTS baselines and is comparable to single-speaker TTS models. These results demonstrate that only 10 seconds of reference audio is enough to construct a high-quality TTS model, which is on par with TTS models trained on a sufficient amount of data from the target speaker.

B.2 Analysis for Fine-tuning
Effect of Length of Reference Audio To analyze the effect of the length of the reference sample on pronunciation accuracy and speaker similarity, we measure CER and SECS with samples generated using reference audios of various lengths. As in the previous section, we conduct experiments on LJSpeech. For the case of reference audio shorter than 10 seconds, we cut audios of LJSpeech to 3 or 5 seconds. For the case of reference audio longer than 10 seconds, we concatenate several audios into 30 or 60 seconds-long samples and use them as reference samples.
The performance of each length model is in Table 5. We find that pronunciation accuracy improves by using longer reference audio until a length of about 10 seconds. In addition, we observe that speaker similarity is maintained well even if the length of the reference sample used is about 3-second-long.
These results indicate that the length of the reference audio is critical for accurate pronunciation. In order to build a high-quality TTS model with shorter reference audio, a method to preserve pronunciation capability is required, which can be a possible future research direction.
Effect of Optimizer and the Number of Fine-tuning Iterations We check the effects of the number of fine-tuning iterations and statistics of the pre-trained optimizer on fine-tuning. For evaluation, we use the same reference samples and sentences of the LibriTTS dataset in Section 5.2 and generated samples in the same way as in Section 5.2 except for whether the optimizer is reused and the number of iterations of fine-training. For comparison, we measure CER and SECS. We observe two results in Table 6. First, as the number of fine-tuning iterations increases, speaker similarity gradually improves, while pronunciation accuracy deteriorates rapidly after a certain number of iterations. Second, using the statistics of the pre-trained optimizer for fine-tuning negatively affects both pronunciation accuracy and speaker similarity. Through this experiment, we find the optimal setting of Guided-TTS 2, which can generate samples as close to the reference audio as possible without pronunciation deterioration.

C Societal Impact
Guided-TTS 2 has an advantage in significantly reducing data required for high-quality adaptive TTS. In addition, Guided-TTS 2 can adapt to not only human voice but also non-human characters such as Gollum, which shows the possibility of extension to TTS for non-human characters in industries such as games and movies. On the other hand, 10-second untranscribed speech for the target speaker is easy to obtain through recording or YouTube clips for celebrities, and the contribution of Guided-TTS 2 that reduces the data required for high-quality adaptive TTS makes a lot of room for misuse. Guided-TTS 2 is likely to be misused as voice phishing for individuals or to have a fatal effect on the security system through voice. Given this potential misuse, we've decided not to release our code. Although we do not release the code, due to the adaptation ability of the diffusion-based model, we expect that the adaptive TTS technology is highly likely to be misused like Deepfake. We leave the research on anti-spoofing that distinguishes generated speech from real audio as future work, considering the potential for misuse of Guided-TTS 2.