{'fraction_non_alphanumeric': 0.03576524741081703, 'fraction_numerical': 0.010080552359033372, 'mean_word_length': 4.792055451879499, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Online Monaural Speech Enhancement Using Delayed Subband LSTM

This paper proposes a delayed subband LSTM network for online monaural (single-channel) speech enhancement. The proposed method is developed in the short time Fourier transform (STFT) domain. Online processing requires frame-by-frame signal reception and processing. A paramount feature of the proposed method is that the same LSTM is used across frequencies, which drastically reduces the number of network parameters, the amount of training data and the computational burden. Training is performed in a subband manner: the input consists of one frequency, together with a few context frequencies. The network learns a speech-to-noise discriminative function relying on the signal stationarity and on the local spectral pattern, based on which it predicts a clean-speech mask at each frequency. To exploit future information, i.e. look-ahead, we propose an output-delayed subband architecture, which allows the unidirectional forward network to process a few future frames in addition to the current frame. We leverage the proposed method to participate to the DNS real-time speech enhancement challenge. Experiments with the DNS dataset show that the proposed method achieves better performance-measuring scores than the DNS baseline method, which learns the full-band spectra using a gated recurrent unit network.


Introduction
This paper addresses the problem of online single-channel speech enhancement/denoising, where 'online' means that the signal is received and processed frame by frame. Deeplearning-based speech enhancement has widely been studied and has largely surpassed the traditional noise estimation and Bayesian-filtering-based methods [1,2], An overview of deeplearning-based speech enhancement methods can be found in [3]. These methods are often conducted in the time-frequency (TF) domain, and use a neural network to map noisy speech spectral feature onto clean speech targets. The input features, e.g. (logarithm) signal spectra, cepstral coefficients and linear prediction based features, generally represent the frame-wise full-band spectral structure of noisy speech. The output target consists of either the clean speech (logarithm) spectral vector or an TF mask vector to be applied to the corresponding noisy speech frame. Various training targets are summarized in [4]. Recurrent neural network (RNN), especially memory enhanced RNNs, such as long short-term memory (LSTM) [5] and gated recurrent units (GRUs) [6], are widely used for speech enhancement to model the temporal dynamics of the signal spectra. [7], one of the first works using LSTM to perform speech enhancement, takes the logarithmic Mel-scale spectrograms of noisy signal and clean speech as input and output, respectively. It was shown in [8] that LSTM outperforms the generic multilayer perceptron network. In [9], multiple-target learning was proposed for LSTM based speech enhancement. A recent work [10] used GRU to improve the computational efficiency compared to LSTM. Instead of directly using signal features as input to RNNs, convolutional-recurrent neural networks [11,12] employ several convolutional layers prior to the RNN layer to extract more abstract spectral representations. All these RNNbased methods process the frame-wise full-band spectra. When only forward RNN is used, these methods are naturally suitable for online processing.
In this work, we propose a speech enhancement method based on an output-delayed subband LSTM network. For each frequency, a sequence of STFT (short-time Fourier transform) magnitude of this frequency (together with some context frequencies) of a noisy signal input is fed into the LSTM network, which outputs the corresponding sequence of clean speech target at this frequency. A unique LSTM network is trained across all frequencies, and hence is shared by all frequencies during inference. With the aim of discriminating between speech and noise, this subband LSTM network is designed and expected to perform on the following grounds. First, it learns the frequencywise signal stationarity to discriminate between speech and stationary noise. It is known that speech is non-stationary, while many types of noise are relatively stationary. The temporal evolution of frequency-wise STFT magnitude reflects the stationarity, which is the foundation for the conventional noise power estimators [13,14] and speech enhancement methods [1,2]. In our previous work [15], it was demonstrated that the subband LSTM network is able to be trained as a good noise power estimator for relatively stationary noise. In this work, we train the subband LSTM network to directly estimate clean speech. Second, it learns the local spectral pattern presented in the current and context frequencies to discriminate between speech and non-stationary/instantaneous noise. There was one attempt to use subband feature in [16] to perform TF-wise speech/noise classification, which testifies that local spectral pattern is informative for discriminating between speech and other signals.
Exploiting a look-ahead is able to improve the performance. This causes a processing latency, but a small latency is normally tolerable for many applications. Bidirectional RNN is usually used for exploiting future information, which however is difficult to use for online processing. In this work, we adopt a simple yet effective network, i.e. output-delayed LSTM network [17], to exploit the look-ahead with forward LSTM. When training the forward LSTM network, the output sequence is set to be delayed relative to the input sequence. To do this, the network learns to store the future information used for inferring one frame in the memory cell and hidden units of the forward LSTM. During inference, at one frame, the network receives the input vector of this frame, and predicts the speech target for one previous frame. Experiments show that this network achieves a speech enhancement performance close to the one of bidirectional LSTM.
Unlike the subband method of [16] and the aforementioned full-band methods, the proposed one shares the same network parameters across subbands, i.e. frequencies. This results in a drastic reduction of the number of network parameters as well as of the size of the training dataset. The proposed method focuses on the subband information and has a low-dimensional input vector, which makes the proposed network less subject to the curse of dimensionality relative to the full-band networks. The network training procedure converges normally in a few epochs, and overfitting rarely happens, which leads to a good generalization performance. In our previous work [18], we proposed to preform multichannel speech enhancement based on subband LSTM. In this work, we study the online singlechannel (monaural) speech enhancement problem based on subband LSTM. We experimentally testify the effectiveness of subband LSTM for monaural speech enhancement. We exploit the subband spectral patterns to discriminate between speech and instantaneous noise, as [18] only adopts the frequency-wise information. We develop the output-delayed LSTM network with look-ahead, for online speech enhancement.

The Proposed Method
We consider single-channel signal in the STFT domain: where k = 0, . . . , K − 1 and t = 1, . . . , T denote the frequency and frame indices, respectively, x(k, t), s(k, t) and u(k, t) are the (complex-valued) STFT coefficients of the microphone, speech and noise signals, respectively. The noise-free speech s(k, t) represents the reverberant image signal received at the microphone. This work focuses only on the denoising task, and the target is to suppress noise u(k, t) and recover the reverberant speech signal s(k, t).

Target and Input
To recover the speech STFT coefficients s(k, t), one popular way is to first predict a magnitude-based mask, such as ideal ratio mask or magnitude ratio mask [4], and then reconstruct the complex-valued coefficients using the phase of noisy signal x(k, t). In order to also estimate the phase of clean speech, [19] proposed a complex ideal ratio mask (cIRM), defined as the ratio of the STFT coefficients between the speech and noisy signals. The cIRM was shown in [19] in the full-band framework, and is verified by our preliminary experiments in our subband framework, to outperform the magnitude-based mask. In this work, we directly adopt cIRM as the training target, and compute cIRM exactly following the equations presented in [19]. For one TF bin, we denote cIRM as y(k, t) ∈ R 2 . For each frequency bin, we aim to predict the cIRM sequencẽ Different from the full-band framework, e.g. [3], (the network is trained to learn a regression from a full-band noisy spectrum to a full-band clean-speech target), in this work we devise a network to learn an speech-to-noise discrimination function based on signal stationarity and on the local spectral pattern, and then to predict the frequency-wise clean-speech target. For one TF bin, the STFT magnitude of the current frequency and the neighbor frequencies are concatenated to form the input, where | · | and T denote the absolute value and transpose operators, respectively, and N is the number of neighbour frequencies considered on each side. For boundary frequencies, with k − N < 0 or k + N > K − 1, circular Fourier frequencies are used. For frequency k, the input sequence is theñ In this sequence, the temporal evolution of |x(k, t)| in (3) reflects the signal stationarity, which is an efficient cue to discriminate between speech and relatively stationary noise. To some extend, the local spectral pattern encoded in x(k, t) and its temporal dynamics is able to discriminate between speech from other non-stationary or instantaneous noise signals.
As for training, input-target sequence pairs are generated with a constant-length sequence. To facilitate network training, the input sequence has to be normalized to equalize the input levels. We empirically normalize the input sequence with the mean STFT magnitude of the present frequency, i.e.

Output-Delayed LSTM Network
RNN transmits the hidden units along the time steps. To avoid the problem of exponential weight decay (or explosion) along time, LSTM introduces an extra memory cell allowing to learn long-term dependencies [5].
For online processing, the network receives and processes data one time step at a time. A look-ahead causes a processing latency. Many of the speech applications can tolerate tens to hundreds of milliseconds of latency. In this work, we adopt a simple yet effective network, i.e. an output delayed LSTM network, to exploit the look-ahead advantages. Fig. 1 shows the network diagram used in the present work. Two layers of unidirectional forward LSTM networks are stacked, followed by a dense layer to output the prediction. The network input is the noisy signal sequence defined in (4), while the network output is the cIRM sequence defined in (2), but with a delay τ relative to the input sequence. Note that the frequency index k is omitted, since all the frequencies share the same network with the same parameters, and is equivalently taken as training/inference samples by the network. The LSTM layers are trained to learn a speech/noise discriminative function based on the signal stationarity and on the local spectral pattern, and then to predict the target. To infer y(t − τ ), the input of the future time steps, i.e. x(t − τ + 1), . . . , x(t), are provided. To exploit the future time steps, instead of adopting extra backward LSTMs, this network stores future information, used for inferring y(t−τ ) in the memory cell and hidden units of the forward LSTM. At inference time, the network behaves the same as the standard LSTM, namely unrolling the forward LSTM once per time step, and automatically exploiting the future time steps τ .
The proposed output-delayed subband network is of relatively small in size, namely approximatively 1.3 millions learnable parameters, due to the fact that we focus on learning the Input x(t) size: 2F + 1 t · · · T Forward LSTM output size: 384 Forward LSTM output size: 256 Forward LSTM output size: 384 subband information with a relatively low-dimensional input vector. The mean squared error (MSE), i.e. |y(k, t) −ŷ(k, t)| 2 , is used as the training loss, whereŷ(k, t) represents the prediction.

Online Inference
During inference, the signal is received and processed frame by frame, and each frequency is processed independently and sequentially. First, the raw input signal is normalized using the mean of the STFT magnitude, which is recursively updated: where α = L−1 L+1 is a smoothing parameter, with L being the desired number of frames to be smoothed. Then the network predicts the cIRM for bin (k, t − τ ), using the normalized input vector together with the memory cell and hidden units from t − 1, i.e. c(k, t − 1) and h(k, t − 1). Let LSTM() denote one LSTM inference, formally we have: Finally, the STFT coefficients of the enhanced speech are computed using the predicted cIRM following the equations presented in [19].

Experimental Setup
We took part to the INTERSPEECH 2020 Deep Noise Suppression (DNS) challenge [20] whose objective is real-time singlechannel speech enhancement. The clean speech dataset for training is extracted from the audio book dataset Librivox by selecting good quality recordings, with more than 500 hours of speech from 2,150 speakers. The noise dataset consists of about 150 audio classes and 60,000 clips extracted from Audioset, and 10,000 noise clips from Freesound and DEMAND databases.
Training data generation. To generate reverberant speech, two real-recorded multichannel room impulse response (RIR) datasets are used: (i) the Multichannel Impulse Response Database [21], with three reverberation times (T60) 0.16 s, 0.36 s and 0.61 s, and (ii) the Reverb challenge [22] dataset with three reverberation times 0.3 s, 0.6 s and 0.7 s. The two datasets are recorded with two different eight-channel microphone arrays, with various microphone-to-speaker distances and speaker directions. Single-channel RIRs are randomly selected from these multichannel RIRs and convoluted with the clean speech clips to generate reverberant speech. Then the reverberant speech and randomly selected noise clips are added to generate noisy speech, with a signal-to-noise-ratio (SNR) randomly selected from the set of {−5, 0, 5, 10, 15} dB. Using the properly modified DNS signal generation script, a total of 20 hours of noisy speech signals are generated for training, from which 5 hours of data are reverberation-free speech. The signals are transformed to the STFT domain using a 512-sample (32 ms) Hanning window with a frame step of 256 samples. The sequence length for training is set to T = 192 frames (about 3 s). The training sequences are picked out from the clip-level signals with 50% overlap for two adjacent sequences. In total, about 11.1 million training sequences are generated. It is obvious that the amount of training data, i.e. 20 hours, is way less than the amount provided in the DNS dataset, which means that only a small part of the DNS data are used. Such a small amount of training data is already able to provide a rich set of subband spectral patterns.
Test data. The DNS challenge provides a publicly available test set including two categories of synthetic clips, i.e. without and with reverberations. Each category has 150 noisy clips with SNR levels distributed between 0 dB to 20 dB. In addition to these data, for each category, we generate one new group of noisier speech by remixing the test clean speech and noise, all with SNR of 0 dB. Implementation details. We use Keras [23] to implement the proposed method. The Adam optimizer [24] is used with a learning rate of 0.001. The batch size is set to 512. The training sequences were shuffled. The number of train epochs is empirically set to 8. For speech enhancement, one clip of test data is processed frame by frame. The online inference is implemented using the stateful LSTM to transmit the memory cell and hidden units. The function 'predict on batch' is used to process all frequencies. The smoothing factor α is set with L = 192.

Experimental Results
Three performance metrics are used, i) perceptual evaluation of speech quality (PESQ) [25]; ii) short-time objective intelligibility (STOI) [26]; and iii) signal-to-distortion ratio (SDR) [27] in dB. For all metrics, the larger the better. 1 Setting the number of context frequencies.
To predict the mask for one single frequency, N neighbour frequencies for each side are adopted as the network input to exploit the subband spectral pattern that is able to discriminate between speech and noise. Experiments with output delay. The network output is set as a delayed sequence relative to the input sequence. This allows the unidirectional forward LSTM to exploit future information, and to facilitate the online processing with a look-ahead. Table 1 (bottom) shows the performance as a function of τ . It is seen that the performance measures noticeably increase with increasing τ , until τ = 8. When a larger τ is set, e.g. τ = 10, the performance measures decrease, which is possibly due to the fact that more look-ahead frames do not provide more useful information, but increase the training difficulty. The last column in this table presents the results of bidirectional LSTM which is only suitable for offline applications. It is unsurprising that the performance measures of the output delayed LSTM are lower the ones of bidirectional LSTM, but the performance gap is small, especially for PESQ and STOI. This indicates that even if the output-delayed LSTM can not fully exploit the future information, it is still very efficient.
Comparison with the DNS baseline method. The DNS challenge baseline method, i.e. NSNet [10], is similar with the proposed method in the sense that both methods use RNN to perform the so-called single-frame-in and single-frame-out online speech enhancement. The main difference is that NSNet takes as input the full-band noisy spectra and outputs the full-band speech mask, while the proposed network takes as input the subband noisy spectra and outputs the frequency-wise speech mask. Thence NSNet provides a perfect comparison method that can demonstrate the difference between the full-band and subband frameworks. We use the dns challenge recipe provided in the asteroid toolkit 2 to implement NSNet. The configurations provided in the recipe are used, except that the training data are generated with reverberations and SNRs following the principles used for the proposed method. A complex network is trained, which takes as input the complex and magnitude spectra and output the complex mask. Table 2 briefly summarizes NSNet and the proposed network. Since NSNet learns the fullband spectra, it requires a larger network and more training resources than the proposed subband network.
According to the rule of the DNS challenge that a maximum of 40 ms look-ahead can be used, we submitted enhanced signals yielded by the proposed method with τ = 2, which exploits two future frames to enhance the current frame, and uses a 16 × 2 = 32 ms look-ahead. Table 3 shows the performance of NSNet and of our submissions. The comparison between NSNet and the proposed method is quite consistent across the four different reverberant and noisy conditions. NSNet slightly outperforms the proposed method in terms of SDR and STOI, and the superiority is more noticeable for the low SNR cases (with SNR=0 dB). This indicates that the full-band spectra is indeed more discriminative than the subband spectra in the aspect of suppressing the noise level. The proposed method yields higher PESQ scores than NSNet by a large margin. The reasons for this could be that (i) the proposed method discriminates between speech and noise highly relying on the signal stationarity, and thus is especially superior to suppress relatively stationary/continuous noise, and that (ii) NSNet predicts the speech mask of all frequencies together, for which the prediction errors are correlated and structured along frequencies, which leads to noticeable speech or noise distortion. By contrast, the proposed method processes frequencies independently, hence the prediction errors are likely to be independent as well. By some informal listening test, the enhanced signals of the proposed method sound smoother and more natural than NSNet. Moreover, the suppressed noise is rarely distorted by the proposed method. By the listening test, the proposed method suppresses the noise level and does not introduce other unpleasant sound effects.
Inference was run on an Intel i5-1035G1 quad core CPU with a base frequency of 1.0 GHz. The computation of per STFT frame takes 7.0 ms, which is in real-time. The DNS Challenge has announced the P.808 subjective evaluation results on a blind noisy test set (consists of both synthetic and real data). The Mean Opinion Scores (MOS) of the noisy set is 3.01, and the one of the proposed method is 3.32, which ranks the 4th place out of the 16 Real-Time track submissions.

Conclusion
We proposed an online monaural speech enhancement method based on a delayed subband LSTM network. Focusing on one frequency at a time, the proposed network requires a small amount of training data and of resources. Promising speech denoising results were achieved, which testifies that subband information, i.e. signal stationarity and local spectral patterns, is indeed discriminative for the task at hand. The output-delayed scheme provides a computationally efficient yet performant way to exploit the look-ahead potential.