{'fraction_non_alphanumeric': 0.039650956513400774, 'fraction_numerical': 0.019897396557510667, 'mean_word_length': 4.681830563879052, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Deep Noise Suppression With Non-Intrusive PESQNet Supervision Enabling the Use of Real Training Data

Data-driven speech enhancement employing deep neural networks (DNNs) can provide state-of-the-art performance even in the presence of non-stationary noise. During the training process, most of the speech enhancement neural networks are trained in a fully supervised way with losses requiring noisy speech to be synthesized by clean speech and additive noise. However, in a real implementation, only the noisy speech mixture is available, which leads to the question, how such data could be advantageously employed in training. In this work, we propose an end-to-end non-intrusive PESQNet DNN which estimates perceptual evaluation of speech quality (PESQ) scores, allowing a reference-free loss for real data. As a further novelty, we combine the PESQNet loss with denoising and dereverberation loss terms, and train a complex mask-based fully convolutional recurrent neural network (FCRN) in a"weakly"supervised way, each training cycle employing some synthetic data, some real data, and again synthetic data to keep the PESQNet up-to-date. In a subjective listening test, our proposed framework outperforms the Interspeech 2021 Deep Noise Suppression (DNS) Challenge baseline overall by 0.09 MOS points and in particular by 0.45 background noise MOS points.


Introduction
Speech enhancement aims at improving intelligibility and perceived quality of a speech signal degraded by disturbances, which can include both additive noise and reverberation. The classical solution for single-channel denoising is to estimate a time-frequency (TF) domain mask for spectral weighting [1][2][3][4][5][6].
The Deep Noise Suppression (DNS) Challenge for Interspeech 2020 (DNS1) [18], ICASSP 2021 (DNS2) [19], and Interspeech 2021 (DNS3) [20] addressed a challenging task for joint dereverberation and denoising. Strake et al. [21] achieved a second rank in the DNS1 non-realtime track by training an actually realtime-capable FCRN for complex spectral maskingbased denoising and dereverberation, which we build upon. All of the abovementioned works apply supervised training using synthetic data, where clean speech and noise are prepared separately and are mixed by addition, while reverberation (if considered) is applied by convolution with simulated or recorded room impulse responses (RIRs). However, in a real-world implementation, only the noisy speech mixture is available, leading to the question, how such real data could be advantageously employed in training. A semi-supervised training for speech enhancement deep neural networks (DNNs) is proposed in [22][23][24], which utilize both synthetic and real data. All of these works employ generative adversarial networks (GANs), using a generator to separate the target signal from the mixture, while a discriminator distinguishes between the true clean speech signal and the enhanced signals. Thus, real data can be used for training the generator, while synthetic data, where clean speech must be available, is used for training the discriminator. Furthermore, well-performing speech enhancement GANs usually still use an additional discriminative loss term such as an L1 loss to stabilize generator training [22,23], which can only be employed for synthetic data. Beyond GAN-type losses, to the best of our knowledge, it has not yet been shown how real data can be used to train a speech enhancement DNN.
To allow real data for non-GAN losses, a possible way is to train a network which predicts absolute category rating (ACR) listener scores, as, e.g., perceptual evaluation of speech quality (PESQ) [25] does it. However, the original PESQ is a nondifferentiable function, which cannot be used as an optimization criterion for gradient-based learning. Zhang et al. [26] implemented an approximated gradient descent for the original PESQ, while Martín-Doñas et al. [27] proposed an approximated PESQ formulation, which is differentiable. Zhao et al. [28] proposed a supervised psychoacoustic loss, which outperforms the approximated PESQ [27]. Fu et al. [29] trained an end-to-end DNN approximating the PESQ function (socalled Quality-Net) without having to know any computation details of the original PESQ formulation. The trained Quality-Net is used to provide a differentiable PESQ loss for the training of speech enhancement DNNs. However, as the original PESQ [25], both [27,29] require a clean reference signal, which makes them only suitable for a fully supervised training approach. Furthermore, the Quality-Net is fixed during the enhancement model training, which strongly limits the performance, since the Quality-Net can be fooled by the speech enhancement model (estimated PESQ scores increase while true PESQ scores decrease) after training for several minibatches, as reported by the authors of [29].
In our work, we propose an end-to-end non-intrusive PESQNet modeling the PESQ function, and subsequently we employ it to train a speech enhancement DNN. Our proposed PESQNet can estimate the PESQ score of the enhanced signals without knowing the corresponding clean speech (just like human raters in ACR listening tests), thus, it can serve to provide a reference-free perceptual loss for training with real data. Furthermore, we solved the problems addressed in [29] by training the FCRN and the PESQNet alternatingly, similar to alternating training schemes used in adversarial trainings [22][23][24]. We combine the loss provided by the PESQNet and the successful FCRN loss from [21], thereby training a complex mask-based FCRN in a "weakly" supervised way, employing both synthetic and real data in training.
The rest of the paper is structured as follows: Section 2 introduces the notations, the speech enhancement system, and our novel weakly supervised training scheme with PESQNet. The experimental setup as well as the results and discussion are presented in Section 3. We conclude the work in Section 4.

Deep Noise Suppression (DNS) by an FCRN
We assume the microphone mixture y(n) to be composed of the clean speech signal s(n) reverberated by the room impulse response (RIR) h(n), and disturbed by additive noise d(n) as with s rev (n) and n being the reverberated clean speech component and the discrete-time sample index, respectively, and * denoting a convolution operation. Since we perform a spectrum enhancement, we transfer all the signals to the discrete Fourier transform (DFT) domain (see Fig. 1): with frame index ℓ and frequency bin index k ∈ K = {0, 1, . . . , K −1}, and K being the DFT size. Following [21], we use an FCRN structure delivering a magnitude-bounded complex mask M ℓ (k) ∈ C, with |M ℓ (k)| ∈ [0, 1] to obtain the enhanced speech spectrum: Finally, the enhanced speech spectrumŜ ℓ (k) is subject to an inverse DFT (IDFT), followed by overlap add (OLA) to reconstruct the estimated signalŝ(n). We adopt the fully-supervised loss function operating in the complex spectrum domain proposed in [21], which consists of two loss terms. The utterance-wise joint loss term aims at joint dereverberation and denoising by utilizing the clean speech spectrum S ℓ (k) as target, and is defined as with Lu being the set of frames belonging to the utterance indexed with u, and L = Lu being the number of frames. A second loss term only focuses on denoising by employing the reverberated clean speech spectrum S rev ℓ (k) as target in Finally, the two loss terms (4) and (5) are weighted by β, and combined in the final loss for synthetic data shown in Fig. 1: with β = 0.9, as used in [21].

Novel PESQNet
Furthermore, we propose an end-to-end non-intrusive PESQNet to estimate the PESQ score of an enhanced speech PESQNet loss (7) PESQ(u) Total loss (9) PESQNet PESQ(u) Synth. loss (6) Real loss (8) utterance in the DFT domain. This network shall allow the use of real data to train the FCRN by providing learning guidance without requiring a clean reference signal. Perceptual speech quality estimation delivers a single value (label) for an entire utterance, just as speech emotion recognition (SER) does. Furthermore, models used for SER are always non-intrusive, which is suitable for our reference-free PESQ estimation. So we adapted the SER model proposed in [31], which shows state-of-the-art performance in emotion recognition, to construct our PESQNet. The only difference to the original SER model in [31] is that we change the input to the network from log-mel spectrum to the amplitude spectrum, and the last output layer is changed from four output nodes with softmax activation function to a single output node with a gate function, which limits the estimated PESQ between 1.04 and 4.64, as it is done in [25]. All other settings can be taken from [31]. Note that any other SER model could be used alternatively.
The output of the PESQNet should be as close as possible to its ground truth measured by the original PESQ function [25]. Thus, the loss function for PESQNet training is defined as with PESQ(u) being the true PESQ score of the utterance indexed with u, measured by ITU-T P. 862 [25], see Fig. 1. Note that J PESQNet u (7) can only be employed with synthetic data.

Novel Weakly Supervised Training with PESQNet
The pre-trained PESQNet is employed at the output of the FCRN estimating the PESQ scores of the enhanced speech, which should be maximized. Therefore, we can define a loss function for utterance u, which is minimized during FCRN training as shown in Fig. 1. Please note that J real u (8) is a speech enhancement loss, which for the first time, allows the use of real data beyond GAN-type losses. Furthermore, for synthetic data only, we can explicitly consider joint dereverberation and denoising by combining the synthetic loss (6) with the novel referencefree psychoacoustic loss (8). Thus, our proposed final loss function for utterance u in the weakly supervised training employing both real and synthetic data is with α ∈ [0, 1] being the weighting factor. , evaluated separately on synthetic data without and with reverberation. DNSMOS is adopted from [30]. Best results are in bold font, and the second best are underlined. To prevent a fixed PESQNet from being fooled by the updated FCRN (estimated PESQ scores increase while true PESQ scores decrease [29]), we use an alternating training protocol for adapting the FCRN with (9) and the PESQNet with (7). The FCRN and PESQNet training is illustrated in Fig. 1. A zero-mean and unit-variance normalization based on the statistics collected on the training set is represented by the NORM box in Fig. 1

Datasets, Methods, Training, and Metrics
In this work, signals have a sampling rate of 16 kHz and we apply a periodic Hann window with frame length of 384 with 50% overlap, followed by an FFT with K = 512. We directly adopted the FCRN from [21, Fig. 2] as our DNS model, with exactly the same settings, as we do not focus on network topologies here. Note that the number of input and output frequency bins is Kin = 257 + 3 = 260. The last 3 frequency bins are redundant and only used for compatibility with the two maxpooling and upsampling operations, and are dropped for subsequent processing. With these settings (24 ms frame size and 12 ms frame shift), we measured a real-time factor of r = 0.95 on an Intel Core i5 quad core machine (3.4 GHz clock), meeting both real-time and the algorithmic delay restrictions (24 ms+12 ms = 36 ms ≤ 40 ms) of the Interspeech 2021 DNS Challenge [20].
First, we pre-train the FCRN model using the WSJ0 speech corpus [32] (dubbed D synth WSJ0 ) and loss (6). Secondly, the PESQNet is pre-trained on the same dataset with the enhanced speech spectrumŜ ℓ (k) generated by the pre-trained and fixed FCRN, using loss (7). After the pre-trainings, we propose a two-stage fine-tuning with subsets of the data from DNS3 (denoted as D synth DNS3 ). In a first stage, we fine-tune the pre-trained FCRN with the synthetic loss (6) on the D synth DNS3 dataset and subsequently, we fine-tune the PESQNet on the same dataset with the enhanced speech spectrum generated by the fixed fine-tuned FCRN, again with loss (7).
The second-stage (final) fine-tuning is used to advantageously employ real data for a weakly supervised training. Thus, besides the synthetic dataset D synth DNS3 , we also constructed a real dataset D real DNS1+2 comprising the real recordings from both preliminary and blind test datasets from DNS1 and DNS2, in total 4 hours of training material and 0.4 hours of validation material. Only for the model which is used to process the final blind test data D test DNS3 for a subjective listening test, we employ a real dataset with additional real data from the DNS3 development set (not blind test set of course), denoted as D real DNS1+2+3 . A cycle of our novel training protocol (second stage) is as follows: We first train the FCRN with a fixed PESQNet, employing one minibatch (3 utterances) of real data followed by one minibatch of synthetic data using loss (9). The minibatch with synthetic data not only explicitly considers denoising and dereverberation, but also prevents the FCRN from generating additional artifacts in the enhanced speech spectrum. Then, we fix the FCRN, and train the PESQNet with 50 consecutive minibatches of synthetic data using (7), with the enhanced speech obtained from the current FCRN as shown in Fig. 1. This training protocol is dubbed 1−1−50 . We further explore performance when replacing the real minibatch data by an additional synthetic minibatch, which is dubbed 0−2−50 . The weighting factor α in J total u (9) is set to 0.9 for all trainings, only for the novel training protocol 1 − 1 − 50 we perform an extra experiment with α = 0.8. During second-stage fine-tuning, for either the proposed protocol 1 − 1 − 50 or for the purely synthetic reference protocol 0−2−50 , the FCRN model and the corresponding PESQNet are saved after every 39 protocol runs. Among all the saved FCRN models, we select the one which provides the lowest loss (9) on the respective validation sets of real or synthetic data. In our case, measurements on both real and synthetic validation sets lead to the same finally chosen model.
The pre-training dataset D synth WSJ0 is exactly the same as used in [21], which is synthesized with clean speech from the WSJ0 corpus [32] and noise from DEMAND [33] and QUT [34]. We simulate SNR conditions of 0, 5, and 10 dB according to ITU-T P.56 [35]. No reverberation effects are considered in preparation of the pre-training dataset.
The fine-tuning dataset D synth DNS3 contains 100 hours of training material and 10 hours of validation material with a uniformly sampled SNR between 0 and 40 dB. We reverberate 50% of the files in D synth DNS3 by convolving the clean speech component with simulated RIRs following [21].

Results and Discussion
In Table 1 we evaluate the performance of our novel weakly supervised training by performing an instrumental measurement on the synthetic test set D synth, dev

DNS1
. As references, we report the performance of the DNS3 baseline [17] and of the FCRN employing only the synthetic loss (6) [21], both being trained with the synthetic dataset D synth DNS3 . Finally, the performance is measured for the FCRN trained with our novel loss J total u (9), employing either synthetic data only, or both synthetic and real data. Please note that the results in Table 1 are measured on synthetic data D synth, dev DNS1 , and therefore only serve to prove that our novel proposed framework using synthetic and real data does not deteriorate under synthetic data. We observe that the DNS baseline offers the worst perceptual quality (PESQ, DNSMOS) for the conditions without reverberation. This can be attributed to an insufficient SNR improvement, reflected by an about 2 dB lower ∆SNRseg compared to other methods. Under reverberated conditions, the baseline offers the worst dereverberation performance reflected by the lowest SRMR score, but is good in DNSMOS. The FCRN trained with synthetic loss (6) offers the highest perceptual quality and intelligibility reflected by PESQ and STOI scores, respectively, under both reverberation conditions. Our proposed FCRN/PESQNet trained with losses (9) and (7), but only using synthetic data ( 0 − 2 − 50 protocol) shows good balanced performance with six out of eight 1 st and 2 nd ranked metrics. When including real training data as well (proposed 1−1−50 protocol, with α = 0.9), still six 1 st and 2 nd ranks are obtained, which shows that including real data in training doesn't considerably harm performance on synthetic dataset D synth, dev DNS1 . Further decreasing α in (9) to 0.8 degrades the performance to only three 2 nd ranked metrics.
To evaluate the performance on real data, our actual goal, we measured the DNSMOS score on the real recordings from D real,dev DNS3 as shown in Table 2. Even though only a small amount of real data (4 hours) was added to the training material, our novel loss J total u (9) with α = 0.9 employing both real and synthetic data offers the highest DNSMOS score (3.33) among all methods (except for the oracle experiment). This indicates that by training our FCRN with the help of the non-intrusive PESQNet and the novel J total u (9), real data can be advantageously employed during training, which can improve the DNS performance in a real test environment. The setting with α = 0.8 shows that, for such a benefit, a high weight on the original synthetic loss J synth u (6) is still needed. With α = 0.9, however, we are able to prove that our 1−1−50 protocol with the referencefree PESQNet indeed solves the earlier observed issues re-

Submissions
Overall MOS Noisy 2.77 DNS3 Baseline [17], trained with synthetic data only 3.07 FCRN/PESQNet, losses (9) and (7), 1−1−50 , α = 0. 9 3.16 ported by Fu et al. [29] with their proposed Quality-Net. Accordingly, at this point we already can conclude that with our proposed 1−1−50 protocol and the reference-free PESQNet indeed is able to make use of real data. As an oracle experiment, we trained our FCRN/PESQNet using the novel training protocol 1 − 1 − 50 with α = 0.9, employing the data from D real,dev DNS3 not only for evaluation, but also as the real data part during training. The further increased DNSMOS of 3.36 shows that the additional information from real data can be advantageously used by employing the proposed training protocol.
Comparing to the reference training protocol 0−2−50 employing only synthetic data, the performance improvements of our proposed training protocol 1−1−50 with α = 0.9, reflected by a 0.02 points higher DNSMOS, are only sight. In this work, however, we are limited to a very small amount of real data (4 hours) used in training, when compared to the amount of synthetic data (100 hours). Accordingly, during training, all the real data has to be repeatedly used for several times even during one epoch of synthetic data, which may limit the performance, and especially the generalization capability, on real data. In turn, we believe that the use of much more real data will provide an even more significant performance gain. Table 3 presents the overall MOS scores from the subjective listening test carried out by the DNS3 organizers based on the blind test set D test DNS3 , which contains both synthetic and real data. Our proposed framework outperforms the DNS3 baseline [17], which is trained only with synthetic data, by 0.09 MOS points. To some extent, this also confirms the DNSMOS results shown in Table 2, where our proposed framework also outperforms the DNS3 baseline by 0.09 DNSMOS points. Furthermore, our proposed framework delivers a natural-sounding background noise reflected by the reported background noise MOS of 4.33, leading to an improvement of 0.45 MOS points over the DNS3 baseline.

Conclusions
In this paper we illustrate the benefits obtained form training a complex mask-based FCRN for speech enhancement in a "weakly" supervised way employing both synthetic and real data. We propose a non-intrusive PESQNet, providing a reference-free perceptual loss for the training of speech enhancement neural networks using real data. Furthermore, the PESQNet and the FCRN are trained alternately to keep the PESQNet up-to-date. Our proposed framework can offer good performance on synthetic data and improves over the reference methods on real data. In the subjective listening test of the Interspeech 2021 DNS Challenge, our proposed framework shows a 0.09 points higher overall MOS score compared to the challenge baseline. Furthermore, our framework also improves over the baseline by 0.45 background noise MOS points. Even more importantly, however, we are the first to show how real noisy data can be used to train a noise suppression DNN without the use of a generative adversarial network (GAN).