{'fraction_non_alphanumeric': 0.04349497149445904, 'fraction_numerical': 0.026455704311062714, 'mean_word_length': 4.634067123782028, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
NU-Wave: A Diffusion Probabilistic Model for Neural Audio Upsampling

In this work, we introduce NU-Wave, the first neural audio upsampling model to produce waveforms of sampling rate 48kHz from coarse 16kHz or 24kHz inputs, while prior works could generate only up to 16kHz. NU-Wave is the first diffusion probabilistic model for audio super-resolution which is engineered based on neural vocoders. NU-Wave generates high-quality audio that achieves high performance in terms of signal-to-noise ratio (SNR), log-spectral distance (LSD), and accuracy of the ABX test. In all cases, NU-Wave outperforms the baseline models despite the substantially smaller model capacity (3.0M parameters) than baselines (5.4-21%). The audio samples of our model are available at https://mindslab-ai.github.io/nuwave, and the code will be made available soon.


Introduction
Audio super-resolution, neural upsampling, or bandwidth extension is the task of generating high sampling rate audio signals with full frequency bandwidth from low sampling rate signals. There have been several works that applied deep neural networks to audio super-resolution [1,2,3,4,5,6,7]. Still, prior works used 16kHz as the target frequency, which is not considered a high-resolution. Since the highest audible frequency of human is 20kHz, high-resolution audio used in multimedia such as movies or musics uses 44.1kHz or 48kHz.
Similar to recent works in the image domain [8,9,10], audio super-resolution works also used deep generative models [4,5], mostly focused on generative adversarial networks (GAN) [11]. On the other hand, prior works in neural vocoders, which is inherently similar to audio super-resolution due to its local conditional structure, adopted a variety of generative models such as autoregressive models [12,13], flow-based models [14,15], and variational autoencoders (VAE) [16]. Recently, diffusion probabilistic models [17,18] were applied to neural vocoders [19,20], with remarkably high perceptual quality. Details of diffusion probabilistic models and their applications are explained in Section 2.1 and 2.2.
In this paper, we introduce NU-Wave, a conditional diffusion probabilistic model for neural audio upsampling. Our contributions are as follows: 1. NU-Wave is the first deep generative neural audio upsampling model to synthesize waveforms of sampling rate 48kHz from coarse 16kHz or 24kHz inputs. 2. We adopted diffusion probabilistic models for the audio upsampling task for the first time by engineering neural vocoders based on diffusion probabilistic models. 3. NU-Wave outperforms previous models [2,5] including the GAN-based model in both quantitative and qualitative metrics with substantially smaller model capacity (5.4-21% of baselines).

Diffusion probabilistic models
Diffusion probabilistic models (diffusion models for brevity) are trending generative models [17,18], which apply iterative Monte-Carlo Markov chain sampling to generate complex data from simple noise distribution such as normal distribution. Markov chain of a diffusion model consists of two processes: the forward process, and the reverse process.
The reverse process is defined as the reverse of the diffusion where the model, which is parametrized by θ, learns to estimate the added Gaussian noise. Starting from yT , which is sampled from Gaussian distribution p(yT ) = N (yT ; 0, I), the reverse process p θ (y0:T ) := p(yT ) T t=1 p θ (yt−1|yt) is defined as the multiplication of transition probabilities: where µ θ (yt, t) and σ 2 t are the model estimated mean and variance of yt−1. Ho et al. suggested to set variance as timestep dependant constant. Latent variable yt−1 is sampled from the distribution p(yt−1|yt) where θ is model estimated noise and z ∼ N (0, I): Diffusion models approximate the probability density of data p θ (y0) := p θ (y0:T )dy1:T . The training objective of the diffusion model is minimizing the variational bound on negative log-likelihood without any auxiliary losses. The variational bound of the diffusion model is represented as KL divergence of the forward process and the reverse process. Ho et al.  reparametrized the variational bound to a simple loss which is connected with denoising score matching and Langevin dynamics [22,23,24]. For a diffused noise ∼ N (0, I), the loss resembles denoising score matching:

Conditional diffusion models as a neural vocoder
Neural vocoders adopted diffusion models [19,20] by modifying them to a conditional generative model for x, where x is a local condition like mel-spectrogram. The probability density of a conditional diffusion model can be written as p θ (y0|x) := p θ (y0:T |x)dy1:T . Furthermore, Chen et al. [20] suggested training the model with the continuous noise level √ᾱ , which is uniformly sampled between adjacent discrete noise levels √ᾱ t and √ᾱ t−1, instead of the timestep t. Continuous noise level training allows different noise schedules during training and sampling, while a discrete timestep trained model is needed the same schedule. Along with the continuous noise level condition, Chen et al. [20] replaced the L2 norm of Eq.(4) to L1 norm for empirical training stability.

Network architecture
In this section, we introduce NU-Wave, a conditional diffusion model for audio super-resolution. Our method directly generates waveform from waveform without feature extraction such as short-time Fourier transform spectrogram or phonetic posteriorgram. Figure 1 illustrates the architecture of the NU-Wave. Based on the recent success of diffusion-based neural vocoders DiffWave [19] and WaveGrad [20], we engineer their architectures to fit audio super-resolution task. To build a downsampled signal conditioned model, NU-Wave model θ : R L × R L/r × R → R L takes the diffused signal yᾱ := √ᾱ y0 + √ 1 −ᾱ , the downsampled signal y d and the noise level √ᾱ as the inputs and its outputˆ := θ (yᾱ, y d , √ᾱ ) estimates diffused noise . Similar to DiffWaveBASE [19], our model has N = 30 residual layers with 64 channels. In each residual layer, bidirectional dilated convolution (Bi-DilConv) with kernel size 3 and dilation cycle [1, 2, . . . , 512] × 3 is used. Noise level embedding and local conditioner are added before and after the main Bi-DilConv to provide conditions of √ᾱ and y d . After similar parts, there are two main modifications from the original DiffWave architecture: noise level embedding and local conditioner. Noise level embedding. DiffWave includes an integer timestep t in the inputs to indicate a discrete noise level [19]. On the other hand, inputs of WaveGrad contain a continuous noise level √ᾱ to have a different number of iterations during inference [20]. We implement the noise level embedding using 128-dimensional vector similar to sinusoidal positional encoding introduced in Transformer [25]: We set a linear scale with C = 50000 instead of 5000 introduced by Chen et al. [20], because the embedding with C = 5000 cannot effectively distinguish among the low noise levels. γ in noise level embedding is set to 1/16. The embedding is fed to the two shared fully connected (FC) layers and a layer-specific FC layer and then added as a bias term to the input of each residual layer. Local conditioner. We modified the local conditioner to fit the audio upsampling task. After various experiments, we discovered that the receptive field for the conditional signal y d needs to be larger than the receptive field for the noisy input yᾱ. For each residual layer, we apply another Bi-DilConv with kernel size 3 for the conditioner and the same dilation cycle as the main Bi-DilConv to make the receptive field of y d nearly twice as large as that of yᾱ. We hypothesize that having a large receptive field for the local conditions provides more useful information for upsampling. While DiffWave upsampled the local condition with transposed convolution [19], we adopt linear interpolation to build a single neural network architecture that could be utilized for different upscaling ratios.

Training objective
We discovered that the L1 norm scale of small timesteps and large timesteps differ by a factor of almost 10. Substituting the L1 norm to log-norm offers stable training results by scaling the losses from different timesteps. Besides, audio superresolution tasks need to substitute conditional factor from melspectrogram x to downsampled signal y d as: Eᾱ,y 0 ,y d , log − θ yᾱ, y d , √ᾱ 1 .
Algorithm 1 illustrates the training procedure for NU-Wave which is similar to the continuous noise level training suggested by Chen et al. [20]. The details of the filtering process are described in Section 4.5. During sampling, we could utilize an untrained noise schedule with a different number of iterations by using continuous noise level training. Thus, T, α1:T , β1:T in Algorithm 1 and Algorithm 2 need not be identical. Take gradient descent step on

Noise schedule
We use linear noise schedule Linear(1 × 10 −6 , 0.006, 1000) during training following the prior works on diffusion models [18,19,20]. We adjust the minimum and the maximum values from the schedule of WaveGrad (Linear(1 × 10 −4 , 0.005, 1000)) [20] to train the model with a large range of noise levels. Based on empirical results, √ᾱ T should be smaller than 0.5 to generate clean final output y0. During inference, the sampling time is proportional to a number of iterations T , thus we reduced it for fast sampling. We tested several manual schedules and found that 8 iterations with β1:8 = [10 −6 , 2×10 −6 , 10 −5 , 10 −4 , 10 −3 , 10 −2 , 10 −1 , 9×10 −1 ] are suitable for our task. Besides we did not report the test results with 1000 iterations as they are similar to that of 8 iterations.

Dataset
We train the model on the VCTK dataset [26] which contains 44 hours of recorded speech from 108 speakers. Only the mic1 recordings of the VCTK dataset are used for experiments and recordings of p280 and p315 are excluded as conventional setup. We divide audio super-resolution task into two parts: Sin-gleSpeaker and MultiSpeaker. For the SingleSpeaker task, we train the model on the first 223 recordings of the first speaker, who is labeled p225, and test it on the last 8 recordings. For the MultiSpeaker task, we train the model on the first 100 speakers and test it on the remaining 8 speakers.

Training
We train our model with the Adam [27] optimizer with the learning rate 3 × 10 −5 . The MultiSpeaker model is trained with two NVIDIA A100 (40GB) GPUs, and the SingleSpeaker model is trained with two NVIDIA V100 (32GB) GPUs. We use the largest batch size that fits memory constraints, which is 36 and 24 for A100s and V100s. We train our model for two upscaling ratios, r = 2, 3. During training, we use the 0.682 seconds patches (32768 − 32768 mod r samples) from the signals as the input, and during testing, we use the full signals.

Evaluation
To evaluate our results quantitatively, we measure the signalto-noise ratio (SNR) and the log-spectral distance (LSD). For a reference signal y and an estimated signalŷ, SNR is defined as SNR(ŷ, y) = 10 log 10 ( y 2 2 / ŷ − y 2 2 ). There are several works reported that SNR is not effective in the upsampling task because it could not measure high-frequency generation [2,5]. On the other hand, LSD could measure high-frequency generation as spectral distance. Let us denote the log-spectral power magnitudes Y andŶ of signals y andŷ, which is defined as Y (τ, k) = log 10 |S(y)| 2 where S is the short-time Fourier transform (STFT) with the Hanning window of size 2048 samples and hop length 512 samples, τ and k are the time frame and the frequency bin of STFT spectrogram. The LSD is calculated as following: For qualitative measurement, we utilize the ABX test to determine whether the ground truth data and the generated data are distinguishable. In a test, human listeners are provided three audio samples called A, B, and X, where A is the reference signal, B is the generated signal, and X is the signal randomly selected between A or B. Listeners classify that X is A or B, and we measure their accuracy. ABX test is hosted on the Amazon Mechanical Turk system, each of SingleSpeaker and MultiSpeaker task is examined by near 500 and 1500 cases. We only tested it for the main upscaling ratio r = 2.

Baselines
We compare our method with linear interpolation, U-Net [28] like model suggested by Kuleshov et al. [2], and MU-GAN [5]. To compare with our waveform-to-waveform model, we choose these models because U-Net is a basic waveform-towaveform model and MU-GAN is a waveform-to-waveform GAN-based model with minimal auxiliary features in audio super-resolution. We implemented and trained baseline models on same the 48kHz VCTK dataset. While U-Net model was implemented as details in Kuleshov et al. [2], we modified number of channels (down: min(2 b+2 , 32), up: min(2 12−b , 128) for block index b = 1, 2, ..., 8) in MU-GAN for stable training. We early-stopped training baseline models based on their LSD metrics Each model's number of parameters is provided in Table  1, NU-Wave only requires 5.4-21% parameters than baselines. Hyperparameters of baseline models might not be fully optimized for the 48kHz dataset.

Results
As illustrated in Figure 2, our model can naturally reconstruct high-frequency elements of sibilants and fricatives, while baselines tend to generate that resemble low-frequency elements by flipping along with downsampled signal's Nyquist frequency. Table 2 shows the quantitative evaluation of our model and the baselines. For all metrics, NU-Wave outperforms the baselines. Our model improves SNR value by 0.18-0.9 dB from the best performing baseline, MU-GAN. The LSD result achieves almost half (r = 2: 43.8-45.0%, r = 3: 55.6-57.3%) of linear interpolation's LSD, while the best baseline model (MU-GAN, r = 2, SingleSpeaker) only achieves 62.2%. We tested our model five times and observed that the standard deviations of the metrics are remarkably small, in the factor of 10 −5 .

Discussion
In this paper, we applied a conditional diffusion model for audio super-resolution. To the best of our knowledge, NU-Wave is the first audio super-resolution model that produces highresolution 48kHz samples from 16kHz or 24kHz speech, and the first model that successfully utilized the diffusion model for the audio upsampling task.
NU-Wave outperforms other models in quantitative and qualitative metrics with fewer parameters. While other baseline models obtained the LSD similar to that of linear interpolation, our model achieved almost half the LSD of linear interpolation. This indicates that NU-Wave can generate more natural highfrequency elements than other models. The ABX test results show that our samples are almost indistinguishable from reference signals. However, the differences between the models were not statistically significant, since the downsampled signals already contain the information up to 12kHz which has the dominant energy within the audible frequency band. Since our model outperforms baselines for both SingleSpeaker and Mul-tiSpeaker, we can claim that our model generates high-quality upsampled speech for both the seen and the unseen speakers.
While our model generates natural sibilants and fricatives, it cannot generate harmonics of vowels as well. In addition, our samples contain slight high-frequency noise. In further studies, we can utilize additional features, such as pitch, speaker embedding, and phonetic posteriorgram [4,7]. We can also apply more sophisticated diffusion models, such as DDIM [29], CAS [30], or VP SDE [31], to reduce upsampling noise.