{'fraction_non_alphanumeric': 0.03933424760004987, 'fraction_numerical': 0.015210073556913103, 'mean_word_length': 4.666902154715649, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
DPT-FSNet: Dual-path Transformer Based Full-band and Sub-band Fusion Network for Speech Enhancement

Sub-band models have achieved promising results due to their ability to model local patterns in the spectrogram. Some studies further improve the performance by fusing sub-band and full-band information. However, the structure for the full-band and sub-band fusion model was not fully explored. This paper proposes a dual-path transformer-based full-band and sub-band fusion network (DPT-FSNet) for speech enhancement in the frequency domain. The intra and inter parts of the dual-path transformer model sub-band and full-band information, respectively. The features utilized by our proposed method are more interpretable than those utilized by the time-domain dual-path transformer. We conducted experiments on the Voice Bank + DEMAND and Interspeech 2020 Deep Noise Suppression (DNS) datasets to evaluate the proposed method. Experimental results show that the proposed method outperforms the current state-of-the-art.


INTRODUCTION
Speech enhancement (SE) is a speech processing method that aims to improve the quality and intelligibility of noisy speech by removing noise [1]. It is commonly used as a front-end task for automatic speech recognition, hearing aids, and telecommunications. In recent years, the application of deep neural networks (DNNs) in SE research has received increasing interest.
In general, DNN-based methods can be divided into two major categories: time-domain methods [2,3,4] and time-frequency domain (T-F) methods [5,6,7,8]. Time-domain methods estimate clean waveforms directly from the noisy raw data in the time domain.
Traditional T-F domain methods usually transform the noisy input waveform into a Fourier magnitude spectrum by short-time Fourier transform, modify the spectrum by T-F mask, and reconstruct the enhanced spectrum into enhanced waveform by inverse short-time Fourier transform. They usually use the phase of the noisy mixture, which limits the upper bound of the denoising performance. Recent T-F domain methods using complex spectra as features can preserve phase information and have achieved promising performance [8].
Sub-band processing is a common method in audio processing [9,10,11], which takes sub-band spectral features as input and output. Previous work [12] pointed out that the local patterns in the spectrum tend to be different in each frequency band. The sub-band model handles each frequency independently, which allows the subband model to focus on the local patterns in the spectrum and therefore achieve good results in SE tasks. In [7] further improves the performance by fusing sub-band and full-band information.
Recently, dual-path networks [13,14,15,16] have achieved exceptional performance due to their ability to model local and global features of the input sequence. Some studies [14,15] have introduced transformer structures [17] into dual-path networks, where input elements can interact directly based on self-attention mechanism, to further improve the performance of dual-path networks. However, these studies were based on simple time-domain features and did not further investigate the effect of the input of the dual-path network on the enhancement performance. [16] did try to change the structure of encoder and decoder to extract more effective inputs for the dual-path network, but still limited to time-domain features.
Inspired by the above problems, we propose a dual-path transformer based full-band and sub-band fusion network (DPT-FSNet) for speech enhancement. Specifically, our proposed model consists of an encoder, decoder, and dual-path transformer. We utilize a convolutional encoder-decoder (CED) structure to extract an efficient latent feature space for the dual-path transformer. Both the encoder and decoder consist of a 1x1 convolutional layer and a dense block [18], where dilated convolutions are utilized inside the dense block for context aggregation. The dual-path transformer is composed of two parts, intra-transformers and inter-transformers. The intratransformer models sub-band information and the inter-transformer merges the sub-band information from the intra-transformer to model the full-band information. We evaluated our model on the VoiceBank+DEMAND (VCTK+DEMAND) dataset [19] and Interspeech 2020 Deep Noise Suppression (DNS) dataset [20]. The experimental results show that the proposed model achieves better results than other speech enhancement models.

IMPROVED TRANSFORMER
Generally speaking, a transformer consists of an encoder and a decoder [17]. In this paper, we choose the transformer encoder as our basic block. To avoid confusion, the reference to the transformer in this paper refers to the encoder part of the transformer. The original transformer encoder usually contains three modules: positional encoding, multi-head self-attention, and position-wise feed-forward network. In this paper, our transformer consists of two modules as in [14]: multi-head self-attention and modified position-wise feedforward network.

Multi-head self-attention
We used the multi-headed self-attention from [17]. The multiheaded self-attention module can be formulated as: where Z ∈ R l×d is the input sequences with length l and dimension d, and Qi, Ki, Vi ∈ R l×d/h are the mapped queries, keys and values, respectively. W Q i , W K i , W V i ∈ R d×d/h and W O ∈ R d×d are linear transformation matrices.

Modified position-wise feed-forward network
A key issue for the transformer is how to exploit the order information in the speech sequence. Previous studies [14,21] have found that the positional encoding utilized in the original transformer is not suitable for dual-path networks. Inspired by the effectiveness of recurrent neural networks in tracking order information, a GRU layer is used as the replacement of the first fully connected layer in the feed-forward network to learn the location information [21]. The output of the multi-head self-attention is passed through the feedforward network followed by residual and normalization layers to obtain the final output of the transformer.
where F F N (·) denotes the output of the position-wise feed-forward

PROPOSED DPT-FSNET
In this section, we propose a frequency-domain dual-path transformer network for the SE task. As shown in Fig.1, our proposed model consists of an encoder, a dual-path transformer processing module (DPTPM), and a decoder.

Encoder
The encoder consists of a 1x1 convolutional layer and a dilateddense block, where the dilated-dense block consists of four dilated convolutional layers. The input to the encoder is the complex spectrum X ∈ R 2×T ×F resulted from short-time Fourier transform (STFT), and the output is a high-dimensional representation U ∈ R C×T ×F with C T-F spectral feature maps.

Dual-path transformer processing module
The DPTPM consists of two 1x1 convolutional layers, B dual-path transformers (DPTs), and a gated 1x1 convolutional layer. Before the DPTs, we use a 1x1 convolutional layer to halve the channel dimension of the encoder output features to form a new 3-D tensor D ∈ R C ×T ×F (C = C/2), and use D as the input to the DPTs, as presented in Fig.1. Each DPT consists of an intra-transformer and an inter-transformer, where the intra-transformer models sub-band information and the inter-transformer models full-band information. Different from [22], the DPT handles time and frequency paths alternatively instead of parallelly. The intra-transformer processing block models the sub-band of the input features, which acts on the second dimension of D where D intra is passed through a 1x1 convolutional layer to double the channel dimension of the output feature and then through a gated convolutional layer to smooth the output value of the DPTPM.

Decoder
The decoder consists of a 1x1 convolutional layer and a dilateddense block, where the dilated-dense block is the same as in the encoder. The feature from the DPTPM output is passed through the decoder to obtain the estimated complex ratio mask [23]. The enhanced complex spectrum is obtained by the element-wise multiplication between encoder's input and the mask, which is passed through the ISTFT to obtain the enhanced speech waveform.

Loss fuction
In order to make full use of the time-domain waveform-level features and the T-F domain spectrum features, our loss function combines both time-domain and T-F domain losses. The loss function is as follows: L audio is mean square error (MSE) loss: where y andỹ are the sample of the clean speech and the enhanced speech, respectively. and N denotes the number of samples in the waveform. L spectral is L1 loss, which is defined as: where Y andỸ denote the spectrum of the clean speech and the spectrum of the enhanced speech, respectively. r and i are the real and imaginary parts of the complex spectrogram. T and F are the number of frames and the number of frequency bins, respectively 4. EXPERIMENTS

Dataset
We use a small-scale and a large-scale dataset to evaluate the proposed model. For the small-scale dataset, we use the VCTK+ DE-MAND dataset, which is widely used in SE research. This dataset contains pre-mixed noisy speech and its paired clean speech. The clean sets are selected from the VoiceBank corpus [25], where the training set contains 11,572 utterances from 28 speakers, and the test set contains 872 utterances from 2 speakers. For the noise set, the training set contains 40 different noise conditions with 10 types of noises (8 from DEMAND [26] and 2 artificially generated) at SNRs of 0, 5, 10, and 15 dB. The test set contains 20 different noise conditions with 5 types of unseen noise from the DEMAND database at SNRs of 2.5, 7.5, 12.5, and 17.5 dB. All the utterances are downsampled to 16kHz. We use 4-second long segments. If an utterance is longer than 4 seconds, a random 4-second slice will be selected from that utterance. For the large-scale dataset, we use the DNS dataset. The DNS dataset contains over 500 hours of clean clips from 2150 speakers and over 180 hours of noise clips from 150 classes. We simulate the noisy-clean pairs with dynamic mixing during training stage. Specifically, before the start of each training epoch, 75% of the clean speeches are mixed with randomly selected room impulse responses (RIR) provided by [27]. By mixing the clean speech (75% of them are reverberant) and noise with a random SNR in between -5 and 20 dB, we generate the speech-noise mixtures. For evaluation, the DNS dataset has two non-blind test sets named with reverb and no reverb, both of which contain 150 noisy-clean pairs.

Experimental setup
The window length and frame shift of STFT and ISTFT are 25ms and 6.25ms, respectively, and the FFT length is 512. The number of feature maps C of the T-F spectrum is set to 64. All convolutional layers in the encoder and the decode are followed by layer normalization and parametric ReLU nonlinearity. Convolutional layers in the DPTPM are followed by parametric ReLU nonlinearity. The dense block consists of four dilated convolutional layers with dilated rate d = 2. The number of input channels in the successive layers of the dense block increases linearly as C, 2C, 3C, 4C, and the output after each convolution has C channels. We use 4 stacked dual-path transformers, i.e., B = 4 and h = 4 parallel attention layers are employed. The hyperparameters α1, α2 in the Eq.(9) are set to 0.4 and 0.6, respectively. In the training stage, we train the proposed model for 100 epochs. We use Adam [28] as the optimizer and a gradient clipping with maximum L2-norm of 5 to avoid gradient explosion. A dynamic strategy [17] is used to adjust the learning rate during the training stage.
n > warmup. (12) where n is the number of steps, d model denotes the feature size of the input of the transformer, and k1, k2 are tunable scalars. In this paper, k1, k2, d model , and warmup are set to 0.2, 4e −4 , 32, 4000, respectively.

Evaluation metrics
On both datasets, we use wide-band PESQ (dubbed WB-PESQ) [29] and STOI [30] as evaluation metrics. WB-PESQ and STOI quantify the perceptual quality and the intelligibility of a speech signal, respectively. For the VCTK+DEMAND dataset, we also employ the three most commonly used metrics in the VCTK+DEMAND dataset, which are CSIG for signal distortion, CBAK for noise distortion evaluation, and COVL for overall quality evaluation [31]. CSIG, CBAK, and COVL are mean opinion score (MOS) predictors, with a score range from 1 to 5. For the DNS dataset, we also employ SI-SDR as evaluation metrics. Higher scores indicate better performance for all metrics.

Results on the VCTK+DEMAND dataset
The proposed method is compared with other methods which also employ the same VCTK dataset. As shown in Table 1, our proposed model outperforms other transformer-based models such as TSTNN, T-GSA, SE-Conformer, and achieves state-of-the-art performance in terms of WB-PESQ, STOI, CSIG, CBAK, COVL with the least parameters.

Ablation analysis
The experimental results in the previous subsection demonstrate that our method improves the SE performance. To further validate the effectiveness of our method, we performed an ablation analysis. We designed four experiments labeled CED+Dualpath former, STFT+CED+BLSTM, STFT+CED+Sub-band former, STFT+CED+Full-sub former, which are abbreviated as exp.1, exp.2, exp.3, exp.4 in the following. Exp.4 is our proposed method. The difference between exp.1 and exp.4 is that exp.1 take time-domain features as input, which replaces STFT/ISTFT with segmentation and overlap-add stage as in [16]. The difference between exp.3 and exp.4 is that the intra part and inter part of the dual-path transformer in exp.3 both model sub-band information as in Eq.7 while the inter part of the dual-path transformer in exp.4 model full-band information as in Eq.8. Same as exp.3, the BLSTM in exp.2 only models sub-band information. For a fair comparison, the number of parameters and computational complexity of the models in the four experiments was essentially the same, and all use a window length of 25 ms and a frame shift of 6.25 ms to extract frames. Therefore, the system latency is also essentially the same for all four experiments. By comparing exp.2 and exp.3, we can see that the improved transformer performs better than BLTSM, which demonstrates the effectiveness of the improved transformer. Furthermore, exp.4 outperforms exp.3, proving the advantages of fusing sub-band information and full-band information. In both exp.1 and exp.4, the intra transformer and inter transformer in the dual-path transformer model local and global information, respectively. However, the evaluation results of exp.4 is much better than those of exp.1, which proves that the frequency domain feature is more effective than the time domain feature for the dual-path transformer. Table 3 compares the metric scores of the proposed model with those of other architectures on the DNS dataset. We can see that our method outperforms the baseline. We noticed that compared with the full-band model, the proposed model has a more significant performance improvement on the reverberation data. We also see a similar trend in FullSubNet. The possible reason is that the full-band and sub-band fusion models include a sub-band model, and the subband model helps to model reverberation effects by focusing on the temporal evolution of the narrow-band spectrum.

CONCLUSIONS
In this paper, we propose a dual-path transformer-based full-band and sub-band fusion network for speech enhancement in the frequency domain. Inspired by the full-band and sub-band fusion models, we explore features that are more efficient for dual-path structures with the intra part in the dual-path transformer models the subband information, and the inter part models the full-band information. Experimental results on the Voice Bank + DEMAND dataset and DNS dataset show that the proposed method outperforms the current state of the art at a relatively small model size.