{'fraction_non_alphanumeric': 0.04764698978993076, 'fraction_numerical': 0.014200211242811877, 'mean_word_length': 4.232729505680074, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Independent Deeply Learned Matrix Analysis for Multichannel Audio Source Separation

In this paper, we address a multichannel audio source separation task and propose a new efficient method called independent deeply learned matrix analysis (IDLMA). IDLMA estimates the demixing matrix in a blind manner and updates the time-frequency structures of each source using a pretrained deep neural network (DNN). Also, we introduce a complex Student's t-distribution as a generalized source generative model including both complex Gaussian and Cauchy distributions. Experiments are conducted using music signals with a training dataset, and the results show the validity of the proposed method in terms of separation accuracy and computational cost.


I. INTRODUCTION
Blind source separation (BSS) is a technique for extracting specific sources from an observed multichannel mixture signal without knowing a priori information about the mixing system. The most commonly used algorithm for BSS in the (over)determined case (number of microphones ≥ number of sources) is independent component analysis (ICA) [1]. Recently, independent low-rank matrix analysis (ILRMA) [2], [3], which is a unification of independent vector analysis (IVA) [4] and nonnegative matrix factorization (NMF) [5], was proposed as a state-of-the-art BSS method. ILRMA assumes both statistical independence between sources and a low-rank time-frequency structure for each source, and the frequency-wise demixing matrices are estimated without encountering the permutation problem. The source generative model assumed in ILRMA was generalized from a complex Gaussian distribution [2] to complex Student's t-distribution (t-ILRMA) [6] for more robust BSS. As a more general framework, in [7], demixing matrix optimization based on a given spectrogram estimate for the source was proposed, showing that the precise source spectrogram model enables accurate spatial model estimation.
In the underdetermined case (number of microphones < number of sources), the Duong model [8] is a commonly used framework. In the Duong model, frequency-wise spatial covariances, which encode source locations and their spatial spreads, are estimated by an expectation-maximization (EM) algorithm, where the permutation problem must be solved after the optimization. Similarly to ILRMA, an NMFbased low-rank assumption is employed in the Duong model to automatically solve the permutation problem, resulting in multichannel NMF (MNMF) [9], [10]. Note that these algorithms formulate a mixing model, whereas ICA-based methods including ILRMA estimate a demixing model for the separation by focusing only on the determined case. It has been experimentally confirmed that the optimization of a demixing model is more efficient and numerically stable than that of a mixing model [2].
In supervised (informed) source separation, deep neural network (DNN) has shown promising performance in both single-channel [11] and multichannel source separation [12]. In fact, when sufficient data of the audio sources are available, DNN can effectively model their time-frequency structures. However, it is almost impossible to compose an appropriate and generalized spatial model with DNN from training data observed in a multichannel format. This is because the spatial model depends on many factors, including source and microphone locations, the recording room, and reverberation. Therefore, it is reasonable to combine a pretrained DNN source model and a blind estimation of the spatial model. Nugraha et al. proposed a DNN-based multichannel source separation framework [13] using the Duong model (hereafter referred to as Duong+DNN). Although this is a convincing approach, a large computational cost is required to estimate the spatial covariance (the EM algorithm in the Duong model) and the performance is not satisfactory owing to the difficulty of parameter optimization.
In this paper, we unify the ICA-based blind estimation of the demixing matrix and the DNN-based supervised update of the source spectrogram model. In the proposed method, we introduce a complex Student's t-distribution as a generalized source generative model, and the demixing matrix (spatial model) is efficiently optimized using a majorization-minimization (MM) algorithm [14]. Since the proposed method utilizes a timefrequency spectrogram matrix estimated by DNN to optimize the spatial model, we call this method independent deeply learned matrix analysis (IDLMA). Table I shows the relationship between the existing and proposed methods. The spatial model is blindly estimated in all the methods, while the source spectrogram model is estimated by DNN in Duong+DNN and the proposed IDLMA.

II. CONVENTIONAL METHOD A. Formulation
Let N and M be the numbers of sources and channels, respectively. The short-time Fourier transform (STFT) of the  [2], [6] Proposed IDLMA multichannel source, observed, and estimated signals are defined as s ij = (s ij1 , . . . , s ijN ) , x ij = (x ij1 , . . . , x ijM ) , and y ij = (y ij1 , . . . , y ijN ) , where i = 1, . . . , I; j = 1, . . . , J; n = 1, . . . , N ; and m = 1, . . . , M are the integral indexes of the frequency bins, time frames, sources, and channels, respectively, and denotes the transpose. We also denote these spectrograms as S n ∈ C I×J , X m ∈ C I×J , and Y n ∈ C I×J , whose elements are s ijn , x ijn , and y ijn , respectively. In ILRMA, the following mixing system is assumed: where A i = (a i1 , . . . , a iN ) ∈ C M ×N is a frequency-wise mixing matrix and a in is the steering vector for the nth source.
The assumption of the mixing system (1) corresponds to restricting the spatial covariance in the Duong model to a rank-1 matrix [8]. When M = N and A i is not a singular matrix, the estimated signal y ij can be represented as where . . , w iN ) H is the demixing matrix, w in is the demixing filter for the nth source, and H denotes the Hermitian transpose. ILRMA estimates both W i and y ij from only the observation x ij assuming statistical independence between s ijn and s ijn , where n = n .

B. ILRMA and Its Generalization with Student's t-distribution
In [2], [3], the following time-frequency-varying complex Gaussian source generative model is assumed (hereafter referred to as Gauss-ILRMA): where σ ijn is the variance (source spectrogram model), k = 1, . . . , K is the index of the bases, and t ikn and v kjn are the parameters in the NMF-based low-rank model. We also denote the variance matrix as Σ n ∈ R I×J ≥0 , whose elements are σ ijn . In t-ILRMA [6], (3) is generalized to a complex Student's t-distribution as follows: where ν is the degree-of-freedom parameter and p is the domain parameter. When ν → ∞ and p = 2, (5) and (6) become (3) and (4), respectively. Also, (5) with ν = 1 represents the Cauchy-distribution likelihood. The demixing matrix W i and NMF source model t ikn v kjn can be optimized in the maximum-likelihood (ML) sense on the basis of (3) or (5). Since the low-rank structure of |Yn| .2 is ensured by the NMF source model, the permutation problem can be avoided, where |·| .p for matrices denotes the element-wise absolute and pth-power operations.

III. PROPOSED METHOD A. Motivation
The NMF source model in ILRMA is effective for some sources that have a low-rank time-frequency structure. However, this source spectrogram model is not always valid. For example, speech signals have continuously varying spectra, which cannot be efficiently modeled by NMF, and the separation performance of ILRMA is degraded for such sources. If sufficient training data for each source can be prepared in advance, it is possible to construct a suitable source spectrogram model by employing DNN [11]. On the other hand, since the spatial parameters depend on many factors, it is simply impractical to train a general spatial model with DNN even if huge amounts of multichannel observation data are available; therefore, the spatial parameters should be estimated blindly.
In this paper, we propose a new framework, IDLMA, which combines the ICA-based blind estimation of demixing matrix W i and the supervised learning of variance matrix Σ n based on DNN, where the loss function in DNN is designed to maximize the likelihood of the source generative model. In addition, similarly to t-ILRMA, we use a generalized model based on a complex Student's t-distribution including both Gaussian and Cauchy distributions. Duong+DNN also employs DNN that maximizes the likelihood of the Gaussian or Cauchy distribution. However, since the mixing model (spatial covariance) in Duong+DNN is defined by only the Gaussian model, the estimations of the spectral and spatial parameters are inconsistent. In the proposed method, this conflict is resolved by modeling the spatial parameters with the Student's t-distribution model and deriving its optimization algorithm fully consistently in the ML sense.

B. Cost Function in IDLMA
Let DNN n be the DNN source model that enhances the nth source component from a mixture signal, namely, the variance matrix Σ n is estimated by DNN n , and these DNN source models are trained in advance. Fig. 1 shows the principle of the separation mechanism in the proposed IDLMA.
On the basis of (3), the cost function (negative loglikelihood of x ij = W −1 i y ij ) in IDLMA with the complex Gaussian distribution (Gauss-IDLMA) is obtained as and (7) can be generalized with (5) (t-IDLMA) as where y ijn = w H in x ij . Note that L t converges to L Gauss when ν → ∞.

C. Update Rule of Source Spectrogram Model Based on DNN
DNN n is trained so that the source spectrogram |S n | .1 is predicted from an input mixture spectrogram |X| .1 , wherẽ S n ∈ C I×J andX ∈ C I×J are source and mixture spectrograms in the training data, respectively. When we define the output spectrogram as D n = DNN n (|X| .1 ) ∈ R I×J ≥0 , the loss function of DNN n for Gauss-IDLMA is defined as wheres ijn and d ijn are the elements ofS n and D n , respectively, and δ 1 is a small value to avoid division by zero [13]. Also, the loss function of DNN n for t-IDLMA is defined as Since minimizing (9) or (10) is equivalent to the ML estimation of σ ijn in (7) where ε is a small value to increase the numerical stability of the spatial update described in Sect. III-D. The DNN architectures used in this paper are described in detail in Sect. IV-B.

D. Update Rule of Demixing Matrix
The demixing matrix W i can be optimized while taking the statistical independence between sources and the variance matrix Σ n into account on the basis of (3) or (5). In Gauss-IDLMA, W i can be updated by applying iterative projection (IP) [16] to (7), where IP is a fast and stable optimization algorithm that can be applied to the sum of w H in x ij 2 and − log |det W i |. In t-IDLMA, IP cannot be applied to (8) because w H in x ij 2 is intrinsic in the logarithm function.
Therefore, we apply an MM algorithm [14] to derive the update rule of w in .
To design a majorization function for (8), we apply the tangent line inequality to the logarithm term in (8), where z > 0 is the original variable and α > 0 is an auxiliary variable. The majorization function can be designed as where α ijn is the auxiliary variable, and L t and L + t become equal only when We can apply IP in analogy with the derivation in Gauss-ILRMA. The majorization function (14) is reformulated as By applying IP and substituting (15), the demixing filter w in can be updated as follows: where and e n is an N -dimensional vector whose nth element is one and whose other elements are zero. After calculating (18) and (19), we update the separated signal by y ijn ← w H in x ij . In particular, when ν → ∞, the majorization function (14) converges to the original cost function (7), and (20) converges The update rule (18)-(21) is equal to that in t-ILRMA.
To fix the scales of y ijn among the frequency bins, the following back-projection technique is applied before updating Σ n by (11) and (12): where y ijn is an element of Y n , • is the Hadamard product, [·] n is the nth value of the vector, and m ref is the index of the reference channel.

E. Relation between Parameter ν and Numerical Stability
In Gauss-IDLMA, U in defined by (22) can be interpreted as the spatial covariance matrix x ij x H ij weighted by σ ijn −2 . In general, σ ijn is estimated by DNN n , whose output likely fluctuates, resulting in many spectral chasms in the timefrequency plane. Therefore, the weight coefficient σ ijn −2 may be an excessively large value, reducing the numerical stability of Gauss-IDLMA in IP. In t-IDLMA, on the other hand, c ijn in (20) is the point internally dividing σ ijn 2 and |y ijn | 2 with a ratio of ν : 2. Since y ijn is the output of a linear filter, |y ijn | 2 contains fewer chasms than σ ijn 2 ; this yields a beneficial spectral smoothing and numerical stability in optimization.
A prospective drawback of t-IDLMA is slower convergence, especially in the case of small ν close to unity, because the strong inference of DNN is discounted. Thus, there is a tradeoff when setting ν. The appropriate selection of ν will be discussed in the next section.

IV. EXPERIMENTAL EVALUATION A. Task, Dataset, and Conditions
We confirmed the validity of the proposed method by conducting a music source separation task. We compared four methods: ILRMA (blind, K = 20), DNN+WF, Duong+DNN, and proposed IDLMA, where DNN+WF applies a Wiener filter constructed using all the outputs of the DNN source models to the observed monaural signal [17]. Note that MNMF was not included in this experiment because its performance is almost always inferior to that of ILRMA [18]. For Duong+DNN and IDLMA, the variance matrix Σ n was updated by DNN n after every 10 iterations of the spatial optimization.
We used the DSD100 dataset of SiSEC2016 [19] as the dry sources and the training datasets of DNN, where only bass (Ba.), drums (Dr.), and vocals (Vo.) were used in this experiment. The 50 songs in the dev data were used to train DNN n and the top 25 songs in alphabetical order in the test data were used for performance evaluation. The test songs were trimmed only in the interval of 30 to 60 s. To simulate a reverberant mixture, we produced the two-channel observed signals by convoluting the impulse response E2A (T 60 = 300 ms) obtained from the RWCP database [20] with each source, and the mixture of Ba. and Vo. (Ba./Vo.) or Dr. and Vo. (Dr./Vo.) was separated. The recording condition of E2A is given in [6]. All the signals were downsampled to 8 kHz. STFT was performed using a 512-ms-long Hamming window with a 256-ms-long shift in the Ba./Vo. case and a 256-ms-long Hamming window with a 128-ms-long shift in the Dr./Vo. case. We used the signal-to-distortion ratio (SDR) [21] as the total separation performance.

B. Architecture and Training of DNN Source Model
We constructed a fully connected DNN with four hidden layers. Each layer had 1024 units, and a rectified linear unit was used for the output of each layer. To prepare the training data of mixture signals, we defined the following vectors: x j = ( n α jn s jn ) n α jn s jn 2 + δ 2 wheres jn ∈ C I is the STFT of the nth source at j (the column vector ofS n ), x j ands jn are the mixture and source vectors, respectively, α jn is a random variable in the range [0.05, 1], which controls the signal-to-noise ratio in x j , and δ 2 is a small value to avoid division by zero. The input and output vectors of DNN n are | x j | .1 and |s jn | .1 , respectively. To optimize DNN, we added the term (λ/2) q g q 2 to (9) or (10) for regularization, where g q is the weight coefficient in DNN, and ADADELTA [22] with a 128-size mini-batch was performed for 200 epochs. The parameter ε was experimentally optimized and set to 0.1 × (IJ) −1 i,jr ijn . The other parameters were set to δ 1 = δ 2 = 10 −5 , c = 3, and λ = 10 −5 . that the proposed IDLMA outperforms the other methods for both mixtures of instruments. In particular, t-IDLMA with ν = 1000 achieves the highest separation accuracy.

D. Computational Times
To show the efficiency of the proposed approach, we compared the computational times of ILRMA, Duong+DNN, and IDLMA for 100 iterations of spatial optimization. We used Python 3.5.2 (64-bit) and Chainer 2.1.0 with an Intel Core i7-6850K (3.60 GHz, 6 Cores) CPU. To calculate the DNN outputs, a GeForce GTX 1080Ti GPU was utilized. Examples of computational times were 23.3 s for ILRMA, 287.1 s for Duong+DNN, and 26.6 s for IDLMA. These results confirm that the proposed method is as fast as conventional ILRMA and more than 10 times faster than Duong+DNN.

V. CONCLUSION
In this paper, we proposed a new determined source separation method that unifies ICA-based blind spatial optimization and the DNN-based supervised source spectrogram model. The proposed method employs a complex Student's t-distribution as the source generative model. An experimental comparison showed the efficacy of the proposed method in terms of both the separation accuracy and the computational cost.