{'fraction_non_alphanumeric': 0.026246558580605688, 'fraction_numerical': 0.008504129703273171, 'mean_word_length': 5.04959289415248, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Orthonormal Embedding-based Deep Clustering for Single-channel Speech Separation

Deep clustering is a deep neural network-based speech separation algorithm that first trains the mixed component of signals with high-dimensional embeddings, and then uses a clustering algorithm to separate each mixture of sources. In this paper, we extend the baseline criterion of deep clustering with an additional regularization term to further improve the overall performance. This term plays a role in assigning a condition to the embeddings such that it gives less correlation to each embedding dimension, leading to better decomposition of the spectral bins. The regularization term helps to mitigate the unavoidable permutation problem in the conventional deep clustering method, which enables to bring better clustering through the formation of optimal embeddings. We evaluate the results by varying embedding dimension, signal-to-interference ratio (SIR), and gender dependency. The performance comparison with the source separation measurement metric, i.e. signal-to-distortion ratio (SDR), confirms that the proposed method outperforms the conventional deep clustering method.


INTRODUCTION
Speech signals in real environment often involve distortions such as noise, interfering speakers, and reverberation [1]. It is still a challenging task to enhance speech in the interfering speaker conditions although the conventional speech enhancement algorithms obtain reasonable performance when the background noise is stationary [2]. Therefore, the need for separating mixed signals into number of sources through speech separation arises [3]. Speech separation technique can be used to solve the cocktail party problem [4,5] in automatic speech recognition (ASR) systems by enhancing the speech of the specific speaker among multiple speakers along with various background noises.
One of the speech separation techniques is computational auditory scene analysis (CASA) [6,7], which studies how humans separate speech and learn from them. However, this technique needs to manually design ad-hoc rules due to limited number of observations [8]. Another technique is nonnegative matrix factorization (NMF) [9,10], which uses handdesigned rules from human observations with the assumption that audio spectrogram has a low rank structure that can be represented with a small number of bases [11]. Even though this technique seems successful, the speaker dependency issue of the basis led to ad-hoc constraints, and there are also limitations in real-time applications due to the complexity of the decomposition process caused by large number of basis [11].
With the increasing popularity of deep learning applications in various fields, how to make good use of deep learning technique in speech separation has become an essential factor nowadays. Hershey et al. [12] proposed a deep clusteringbased speech separation framework with embedding matrix used as a key factor to cluster the mixed signals, and the neural network is trained to obtain embedding vectors corresponding to each element of high-dimensional signal. Since embeddings are represented in time-frequency (T-F) domain and each input signal can be reconstructed by clustering the embedding vectors, it is possible to separate all the sources independently. Even though deep clustering is somehow helpful to overcome the permutation problem, each separated output still contains undesired segments if there is an error during the clustering process. Therefore, further deep learning-based speech separation frameworks, i.e. deep attractor network (DANet) [13] and permutation invariant training (PIT) [14,15], were released to solve this issue. However, the permutation problem is still inevitable if the performance of constructed embedding or clustering is not high enough.
Our proposed work aims to develop an effective deep clustering framework by designing the training strategy to assign an additional regularization condition. By training the network to find the minimum loss between the embeddings and the identity matrix, the network automatically assigns an orthonormality constraint to the embeddings. Since each embedding can be considered as a basis of each input speaker, the proposed training method leads to have more embedding independency so that embeddings are useful for better decomposition. This effective formation of embeddings helps to improve the remaining permutation problem of the conven-tional method as the decomposed spectral bins help increase the performance of clustering. We demonstrate improvements of the proposed algorithm over the conventional deep clustering algorithm, and also analyze the results in various experimental setups.

DEEP CLUSTERING
Deep clustering is a deep network that uses learned feature transformations known as embeddings, to separate speech on open set of speakers [12]. In audio signals, raw input signal x can be defined as a feature vector X t, f = g t, f (x), where t and f represent frame and frequency index of the signal. Deep clustering starts with the assumption that a reasonable partition of elements exists for each region. Its objective is to find that partition using embedding matrix and K-means clustering in order to estimate masks to be applied to each input mixture of X. With audio signals, these regions are sets of time-frequency bins where each source dominates the other source. Deep clustering seeks K-dimensional embeddings of the mixed signal to apply simple clustering in embedding space.
In training stage, deep neural network is used to transform input x to K-dimensional embeddings V ∈ R {N×K} , with the embedding considered as unit-norm, |v i | 2 = 1, where v i is the embedding for element i. Embeddings V actually represent N by N estimated affinity matrix, which analyzes the content of each mixed signal in terms of a set of channels. The affinity matrix VV T is learned to match YY T by minimizing the cost using the squared Frobenius norm as follows: YY T is the target affinity matrix which is generated with the label indicator, Y , i.e. the concept of comparing each T-F bin and assigning value '1' to the dominant bin, formed as an ideal binary mask (IBM). Note that low-rank formulation leads to efficient implementation of the cost function as shown in the equation below: In inference stage, K-means clustering is used to cluster the K-dimensional embeddings into the number of sources. By assigning the value '1' to the spectral bin of each cluster and '0' to the other, ideal binary masks are formed for each source, and applying these masks to the mixed signal separates the signal into individual sources. Therefore, training to obtain appropriate embedding matrix is essential to cluster the embeddings sufficiently.
This proposed work has focused on enabling better clustering through the formation of optimal embeddings. The work of Hershey et al. [12] considers learning the embedding to follow the label indicator, however, it contains lack of regularity. Therefore, the present study trains with more regularization to generate orthonormal embedding.

ORTHONORMALITY OF EMBEDDING MATRIX
Even though the deep clustering method obtains high performance in single-channel speech separation tasks, there are still more improvements to be made. Knowing that the formation of optimal embeddings leads to better decomposition of spectral bins, focusing on regularizing the embeddings would be effective to the performance.

Penalization term
Penalization term is one of the regularization terms to encourage the diversity in annotation vectors as indicated below: where V, I, and · F are embeddings, identity matrix, and Frobenius norm, respectively. Computing V T V , sets the output dimension to become embedding dimension by embedding dimension (R K×K ), so that it is easier to compute the loss between the identity matrix, R K×K , and V T V . This term forces the diagonal elements of the embedding matrix to 1 and the off-diagonal to 0, making the embedding matrix to be orthonormal. Applying this term on deep clustering embeddings will lead to assign more independence to each other, resulting in efficient clustering. Like the efficient implementation of the cost function of deep clustering in Equation (2), Equation (3) can be shown as the equation below: Therefore, the overall training cost function of the proposed system is as follows: which helps learn the embeddings to match the target affinity matrix and be orthonormal, and the whole system architecture is shown in Fig. 1. A similar idea of the penalization term has been introduced in a self-attentive word embedding task [16] without being used for any other types of signals.

Training procedure
In order to train the proposed algorithm with a deep learning network, the ideal binary mask (IBM) was used as the target, i.e. power dominant bin was set to '1', and '0' for others in each time-frequency bin. We used two bi-directional long short-term memory (BLSTM) layers followed by one fully connected layer. Each BLSTM layer consisted of 512 hidden cells with the size of the embedding dimension K. Adam optimizer with the learning rate of 10 −4 was used for training, and the hyperbolic tangent function was used for both activation and output function of the network. The training criteria were L2 loss between affinity matrices and the additional penalization term, and the dropout rate was set to 0.5 for regularization. Also, we investigated several network models by varying embedding dimensions to 20, 30, and 40.

Visualization of embedding covariance matrix
Existing speech separation techniques, i.e. deep clustering [12], DAN [13], PIT [14,15], simply decompose the label indicator through the projection without any established standards. Therefore, it is difficult to figure out the degree of decomposition. The penalization term in our proposed method is a criterion that enables to sparsely project the label indicator to the embedding matrix, and it can be verified with Fig. 2. This figure indicates the covariance matrix among embedding dimensions of 40, and it compares that of both conventional deep clustering and our proposed method. Fig. 2(a) demonstrates that the conventional deep clustering method has some focused data on specific embedding dimensions; whereas, our proposed method equally spreads the data throughout all dimensions as shown in Fig. 2(b).

Experimental setup
We used Wall Street Journal (WSJ) corpus, both WSJ0 and WSJ1 [17] to make speech mixtures. We randomly select 10000 utterances for training and 5000 utterances for both validation and evaluation sets, where the same proportion of females and males is chosen. Especially for the evaluation set, a set of 1000 mixed utterances of 2 gender cases, i.e. same gender and different gender mixtures, is used with interfering ratio of 3, 6, 9, 12, and 15 dB. Since SIR is evenly distributed in the evaluation set, it becomes easier to analyze the mixtures among gender and SIR. Sampling frequency of all data is set to 16 kHz, and the input feature X is log spectral magnitude using STFT with 512 samples and 256 samples window shift of hanning window. Different from the conventional deep clustering network, our network uses whole utterance for the input instead of 100 frames, and the deep neural network is implemented with the Tensorflow libraries [18].

Speech separation procedure
At inference stage, speech separation is performed by Kmeans clustering. The output mask is generated by clustering the embeddings V , i.e. the output from the proposed model for each utterance. The number of clusters is set to 2 since we assume to have 2 speaker mixtures in this experiment 1 . Among various types of masks, we generated binary masks with the clustered outputs to apply them to the mixture and obtain separated sources. We evaluated two types of deep clustering models, i.e. the conventional deep clustering model and our proposed model, and compared the results. For choosing the target speech from the clustered output, we chose the speech with dominant power level as the target speech and the other as the interfering speech. For all experiments, averaged signal-to-distortion ratio (SDR) was used as the evaluation metric with mir eval library [19].

RESULTS AND DISCUSSION
We tried to analyze the proposed results in three cases: SDR vs. dimension, SDR vs. SIR, and SDR vs. gender. With the first case, we wanted to see the effect of the penalization term among different embedding dimensions. As shown in Table 1, adding penalization criterion output better result with 20 embedding dimension than the baseline with 40 embedding dimension. Also, as the embedding dimension increased, the improvement rate between proposed method and With the second case, we wanted to see the effect of penalization term among different SIRs. Table 1 also shows the absolute SDR in embedding dimension 20 and 40 for various SIRs. As SIR increased, the effect of penalization term increased; however, there were some points that we needed to focus on. In embedding dimension of 20, the penalization term gave negative effect with SIR of 3 dB, meaning that the embedding dimension below 20 may be not enough to handle low SIR mixtures. Forcing to give orthonormality to embeddings can break the correlation when it is actually needed; therefore, it could give more distortion to the signal instead.
For the last case, we wanted to analyze the result between genders, and we found that the penalization term works better on mixtures with different genders than same genders as shown in Table 2. With embedding dimension of 20, the same gender mixtures showed degradation in performance with penalization term. This confirmed that the embedding dimension below 20 is not enough to increase the effect of penalization term. Also, we observed that the penalization term works better with embeddings that already have some independence, such as the mixture signals with mixed gender.
Also, with the analysis of improved normalized projection alignment (NPA) metric [20] and relative error rate, we have verified that the permutation problem of the conventional deep clustering method has been improved with our proposed system, as shown in Fig. 3. Improved NPA shows the improvement in mask formation between deep clustering and Signal-to-interference (dB) Improved NPA (dB) Relative error rate (%) Improved NPA Relative error rate Fig. 3. Quality measurements of estimated IBMs our proposed method, and the relative error rate compares the improved bin prediction error between the target and estimated masks. As the undesired segments in the separated outputs indicate the permutation problem, the quality of the estimated masks is associated with the effective separation of mixed speech. With high SIR mixtures, conventional deep clustering method already obtains high performance, meaning it has less permutation problems. On the other hand, low SIR mixtures has difficulty in forming an optimal mask in speech separation and need more improvements. Therefore, Fig. 3 shows that both improved NPA and relative error rate generally increase as the SIR decreases.

CONCLUSION
In this paper, we have observed the efficacy of our proposed method through the comparison with the conventional deep clustering method. Analyzing the covariance matrix of the embedding output of both conventional and prosed methods, we have confirmed that penalization term sparsely projects the label indicator to all the dimensions in the embedding matrix. Through diverse experiments, we have verified that the performance of our proposed method is maximized with high embedding dimension, high SIR, and different gender, which already contains some independence assumption between speakers in the mixture. Also we have improved the permutation problem in the conventional deep clustering method, and it is verified with the evaluation metric for the estimated masks. Through the verification of the efficacy of penalization term on embeddings, we are to further extend our studies by implementing this term on other embedding-based speech separation methods, e.g. deep attractor network [13]. Also, we will analyze the potentials and limitations of our method and stabilize the penalization term to optimize its effect on the embeddings.