{'fraction_non_alphanumeric': 0.0259198381590593, 'fraction_numerical': 0.009925401441395879, 'mean_word_length': 5.01711677443895, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Perceptual Speech Enhancement via Generative Adversarial Networks

Automatic speech recognition (ASR) systems are of vital importance nowadays in commonplace tasks such as speech-to-text processing and language translation. This created the need of an ASR system that can operate in realistic crowded environments. Thus, speech enhancement is now considered as a fundamental building block in newly developed ASR systems. In this paper, a generative adversarial network (GAN) based framework is investigated for the task of speech enhancement of audio tracks. A new architecture based on CasNet generator and additional perceptual loss is incorporated to get realistically denoised speech phonetics. Finally, the proposed framework is shown to quantitatively outperform other GAN-based speech enhancement approaches.


INTRODUCTION
In a typical noisy environment, an audio signal is perceived as a mixture between a desired audio (speech) and an intrusive background noise. Accordingly, speech enhancement or denoising is interpreted as a source separation problem, where the goal is to separate the desired audio signal from the intrusive noise. The background noise type and the signal-to-noise ratio (SNR) have a direct influence on the quality of the estimated denoised speech. For instance, some common background noise types can be very similar to the desired speech such as cafe or food court noise. In these cases, estimating the desired speech from the mixed is challenging and sometimes impossible in low SNR situations as the components of the mixed signal are sharing the same distribution and frequency bands. This process of eliminating background noise from speech tracks is a vital building block for automatic speech recognition (ASR) systems.
Previously, traditional approaches were adopted for speech enhancement applications such as spectral subtraction [1,2] and binary masking techniques [3,4]. Moreover, statistical approaches based on variants of Wiener filter were also applied in speech enhancement [5,6,7]. However, all of these approaches require a prior knowledge of the SNR and can operate on limited noise types in high SNR situations.
These authors contributed to this work equally.
To overcome such SNR assumptions and noise type limitations, data driven approaches based on deep neural networks (DNNs) are widely used in literature to learn deep underlying features of either the required speech or the intrusive background noise from the given data. For instance, denoising autoencoders (AE) were used in [8,9] to estimate a clean track from a noisy input based on the L1-loss. Long shortterm memory (LSTM) networks have also been utilized to incorporate temporal speech structure in the denoising process [10,11]. Also, an adaptation of the autoregressive generative WavNet was used in [12] where a denoised sample is generated based on the previous input and output denoised samples.
In 2014, generative adversarial networks (GANs) were introduced as the state-of-the-art for deep generative models [13]. In GANs, a generator is trained adversarially with a discriminator to generate images belonging to the same joint distribution of the training data. Afterwards, variants of GANs such as conditional generative adversarial networks (cGANs) were introduced for image-to-image translation tasks [14]. The pix2pix model introduced in [15] is one of the first attempts to map natural images from an input source domain to a certain target domain. Henceforth, cGANs were used for speech enhancement either utilizing raw 1D audio tracks or the 2D log-Mel time-frequency (TF) representation. For instance, speech enhancement GAN (SEGAN) is a 1D adaptation of the pix2pix model operating on 1D raw audio tracks [16]. This model was further adapted to operate on 2D TF representations, via the frequency SEGAN (FSEGAN) framework [17]. Due to the TF representations being interpreted as an implicit feature extraction, an improved speech enhancement was reported. However, both models suffer from multiple limitations. They rely mainly on pixel-wise losses, which have been reported to produce inconsistencies in the resultant local structures [15]. Additionally, both models were utilized to enhance speech tracks of fixed durations and under relatively mild noise conditions (average SNR of 10 dB).
In this work, a new adversarial approach is proposed for enhancing speech tracks based on MedGAN framework, introduced in our previous work [18] to operate on the 2D TF representation of noisy speech input. MedGAN incorporates a non-adversarial perceptual loss which penalizes the discrepancies in the feature space between the outputs and the targets. This enhances the robustness of the speech enhancement under stringent SNR conditions and challenging noise types.  Additionally, we propose a new dynamic time resolution technique to embed variable track lengths in a fixed TF representation by adapting the time overlap based on the track length. To illustrate the performance of the proposed approach, quantitative comparison is carried out against the SEGAN, FSEGAN and a classical Wiener filter approach under different noise types and SNR levels. Furthermore, the word error rate (WER) of a pre-trained automatic speech recognition (ASR) system is also evaluated.

DYNAMIC TIME RESOLUTION
Previously proposed architectures are designed to work on a fixed input size with respect to the pixel-dimensionality or the number of samples for FSEGAN and SEGAN, respectively. In order to accommodate this dimensionality constraint, the input track length is fixed to 1 s intervals. Accordingly, the network inference is not efficient as any arbitrary length track should be first divided into 1 s windows and then the denoising is applied sequentially on each window.
In our proposed framework, the pixel-dimensions of the input 2D TF representation is fixed to 256 × 256 pixels. However, the time resolution per pixel is dynamically changing based on the length of the 1D track as shown in Fig. 1. The used TF representation is computed based on short time Fourier transform (STFT) where a window function followed by FFT is applied to overlapping segments of the 1D track. To fix the frequency dimension N F , the size S of the overlapping segments and thus the corresponding window function should be fixed for all the input tracks. In our implementation, a hamming window of S = 512 samples is used to get a one-sided spectrum of N F = 256 bins. To fix the time dimension N T = 256 bins, the overlapping parameter O of the 1D segments should be adjusted based on the input track length L according to the following relation: Finally, the track length L is modified either by omitting or appending extra samples based on the following constraint: General block diagram of the proposed system. The log-magnitude is passed to the network and the output of the network is used with the input noisy phase for reconstruction.
After applying the denoising framework to the resultant TF representation, getting back to an audible 1D track is mandatory. For this we choose to use the least square inverse short time Fourier transform (LS-ISTFT) proposed in [19]. Based on this implementation an acceptable signal-todistortion ratio (SDR) reconstruction can be achieved with an overlap of 25%. By substituting this overlap in Eq. 1, the longest track length that can be embedded in a 256 × 256 TF representation should not exceed 6.1 s. The LS-ISTFT requires both magnitude and phase of the TF representation for reconstruction. However, the speech phonetic patterns are mostly available in the magnitude, therefore only this component (y m ) is passed as input to the network. For reconstruction, the noisy phase (y p ) is used together with the enhanced magnitude as shown in Fig. 2.

METHOD
In this section, the proposed adversarial approach for speech enhancement will be described. First, a brief explanation of traditional cGANs will be outlined followed by the utilized MedGAN framework. An overview of the proposed approach is presented in Fig. 3.

Conditional Generative Adversarial Networks
In general, adversarial frameworks are a game-theoretical approach which pits multiple networks in direct competition with each other. More specifically, a cGAN framework consists of two deep convolutional neural networks (DCNNs), a generator G and a discriminator D [15]. The generator receives as input the 2D TF representation of the corrupted speech data y m . It attempts to eliminate the intrusive background noise by outputting the synthetically enhanced TF representationx = G(y m ). The main goal of the generator is to renderx to be indistinguishable from the target ground-truth clean noise signal x. Parallel to this process, the discriminator network is trained to directly oppose the generator. D acts as a binary classifier receiving y m and either x orx as inputs and classifying which of the input-pairs are synthetically generated and which are real. In other words, G attempts to produce a realistically enhanced TF representation to fool D, while conversely D constantly improves it's performance to better detect the generator's output as fake. This adversarial training setting drives both network to improve their respec-tive performance until Nash's equilibrium is reached. This training procedure is expressed via the following min-max optimization task over the adversarial loss function L adv : To further improve the output of the generator and avoid visual artifacts, an additional L1 is utilized to enforce pixel-wise consistency between the generator output and the ground-truth targets [15]. The L1 loss is given by

Perceptual Loss
The magnitude component of the speech TF representation is full of rich patterns directly reflecting human speech phonetics. A straightforward minimization of the pixel-wise discrepancy, via L1 loss, will not enable the effective elimination of challenging speech-based noise types, e.g. cafe or food court.
To overcome this issue, we propose the utilization of the perceptual loss, based on the MedGAN framework, to regularize the generator network to produce more globally consistent results by focusing on more wider feature representations rather than individual pixels. This is achieved by utilizing the discriminator network as a trainable feature extractor to extract intermediate feature representations. The perceptual loss is then calculated as the weighted average of the mean absolute error (MAE) of the extracted feature maps: where D n is the feature map extracted from the n th layer of the discriminator. N and λ n are the total number of layers and the individual weight given to each layer, respectively.

Architectural Details
Based on the MedGAN framework, a CasNet generator and a patch discriminator architecture are utilized for the proposed framework [18]. CasNet concatenates three U-blocks in an end-to-end manner, whereas each U-block consists of a encoder-decoder architecture joint together via skip connections. These connections avoid the excessive loss of information due to the bottleneck layer. The output TF representations are progressively refined as they propagate through the multiple encoder-decoder pairs. The architecture of each U-block is identical to that proposed in [15]. Regarding the patch discriminator, it divides the input TF representations into smaller patches before proceeding with classifying each patch as real or fake. For the final classification score, all patch scores are averaged out. However, unlike the 70 × 70 pixel patches recommended in [15], a patch size of 16 × 16 was found to be more beneficial in this use-case.

EXPERIMENTS
The proposed speech enhancement framework is evaluated on the TIMIT dataset [20]. This dataset consists of 10 phonetically rich sentences spoken by 630 speaker with 8 different American English dialects. All the tracks are sampled at 16 kHz and the tracks durations are between 0.9 s to 7 s. The majority of the tracks are satisfying the aforementioned track length constraint in Sec. 2 with only 15 tracks exceeding the 6.1 s limit and these tracks are excluded from the dataset. For the training procedure, three different noise types were utilized (crowded cafe, food court and moving car) from the QUT-TIMIT noise proposed in [21]. The background noise was added to the clean speech tracks in order to create a paired training set. Additionally, different SNR levels were used for each noise type (0, 5 and 10 dB). Thus, the total training dataset consists of 36,000 tracks. For validation, two different experiments were conducted. In the first experiment, the trained network was validated on a dataset of 5000 tracks utilizing the same training noise types albeit from different individuals of 2 new dialects. In the second experiment, the generalization capability of the network was investigated by validating on a dataset of 500 tracks corrupted by a new noise type, the city street noise. Both experiments were conducted using the same SNRs used in training (average SNR 5 dB).
To compare the performance of the proposed adversarial approach, quantitative comparisons were conduced against the FSEGAN, SEGAN and a traditional Weiner filter approach [17,16,7]. All trainable models were trained using the same hyperparameters for 50 epochs to ensure a fair comparison. Multiple metrics were used for the comparison in order to give a wider scope of interpretation for the results. The utilized metrics are the perceptual evaluation of speech quality (PSEQ) [22], the mean opion score (MOS) prediction of the signal distortion (CSIG), the MOS prediction of background noise (CBAK) and the overall MOS prediction score (COVL) [23]. Additionally, the WER was evaluated using Google's open-source pre-trained ASR model [24].

RESULTS
First a qualitative comparison on a TF representation example is illustrated to get a clear interpretation about the advantage of the proposed model. As shown in Fig. 4, the MedGAN is superior in canceling the low power components of the background noise in comparison to FSEGAN as annotated by ( ). In contrast to the MedGAN, the SEGAN model shows a clear elimination of required speech bands as annotated by ( ). This is also reflected in the quantitative comparison shown in Tables I and II. The proposed MedGAN achieves better scores over the other approaches in all the used quantitative metrics. It must also be pointed that in literature the FSEGAN authors claim a better performance in WER over the SEGAN model. However, this have not been observed in the above results. We hypothesize that this is the result of FSEGAN now having to deal with variable time resolution input TF representations, due to the utilized dynamic time resolution, which posses a challenge compared to the SEGAN.

CONCLUSION
In this work, an adversarial speech enhancement technique is introduced to operate on audio TF representations. The proposed approach involves an additional perceptual loss and a CasNet generator architecture to enhance the detailed local features of the output audio tracks. Moreover, to improve the inference efficiency, time-domain tracks with variable durations are embedded in a fixed TF representation by changing the corresponding time resolution. Challenging speech-based noise types, e.g. cafe and food court noise, were involved in training with low average SNR (5 dB). To evaluate the generalization capability of our model, two experiments were conducted on different speakers and noise types. The proposed approach exhibits an enhanced performance in comparison to the previously introduced GANbased and traditional model-based approaches. In future, the proposed framework will be extended to involve general audio data and enhancement tasks such as dereverberation.