{'fraction_non_alphanumeric': 0.03128652586370926, 'fraction_numerical': 0.013183288482970453, 'mean_word_length': 4.716283405842138, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Singing Voice Separation and Vocal F0 Estimation based on Mutual Combination of Robust Principal Component Analysis and Subharmonic Summation

This paper presents a new method of singing voice analysis that performs mutually-dependent singing voice separation and vocal fundamental frequency (F0) estimation. Vocal F0 estimation is considered to become easier if singing voices can be separated from a music audio signal, and vocal F0 contours are useful for singing voice separation. This calls for an approach that improves the performance of each of these tasks by using the results of the other. The proposed method first performs robust principal component analysis (RPCA) for roughly extracting singing voices from a target music audio signal. The F0 contour of the main melody is then estimated from the separated singing voices by finding the optimal temporal path over an F0 saliency spectrogram. Finally, the singing voices are separated again more accurately by combining a conventional time-frequency mask given by RPCA with another mask that passes only the harmonic structures of the estimated F0s. Experimental results showed that the proposed method significantly improved the performances of both singing voice separation and vocal F0 estimation. The proposed method also outperformed all the other methods of singing voice separation submitted to an international music analysis competition called MIREX 2014.


I. INTRODUCTION
S INGING voice analysis is important for active music listening interfaces [1] that enable a user to customize the contents of existing music recordings in ways not limited to frequency equalization and tempo adjustment. Since singing voices tend to form main melodies and strongly affect the moods of musical pieces, several methods have been proposed for editing the three major kinds of acoustic characteristics of singing voices: fundamental frequencies (F0s), timbres, and volumes. A system of speech analysis and synthesis called TANDEM-STRAIGHT [2], for example, decomposes human voices into F0s, spectral envelopes (timbres), and non-periodic components. High-quality F0-and/or timbre-changed singing voices can then be resynthesized by manipulating F0s and spectral envelopes. Ohishi et al. [3] represents F0 or volume dynamics of singing voices by using a probabilistic model and transfers those dynamics to other singing voices. Note that these methods deal only with isolated singing voices. Fujihara and Goto [4] model the spectral envelopes of singing voices in polyphonic audio signals to directly modify the vocal timbres without affecting accompaniment parts.
The authors are with the Department of Intelligence Science and Technology, Graduate School of Informatics, Kyoto University, Kyoto, Japan (email:{ikemiya, itoyama, yoshii}@kuis.kyoto-u.ac.jp).
To develop a system that enables a user to edit the acoustic characteristics of singing voices included in a polyphonic audio signal, we need to accurately perform both singing voice separation and vocal F0 estimation. The performance of each task could be improved by using the results of the other because there is a complementary relationship between them. If singing voices were extracted from a polyphonic audio signal, it would be easy to estimate a vocal F0 contour from them. Vocal F0 contours are useful for improving singing voice separation. In most studies, however, only the one-way dependency between the two tasks has been considered. Singing voice separation has often been used as preprocessing for vocal F0 estimation, and vice versa.
In this paper we propose a novel singing voice analysis method that performs singing voice separation and vocal F0 estimation in an interdependent manner. The core component of the proposed method is preliminary singing voice separation based on robust principal component analysis (RPCA) [5]. Given the amplitude spectrogram (matrix) of a music signal, RPCA decomposes it into the sum of a low-rank matrix and a sparse matrix. Since accompaniments such as drums and rhythm guitars tend to play similar phrases repeatedly, the resulting spectrogram generally has a low-rank structure. Since singing voices vary significantly and continuously over time and the power of singing voices concentrates on harmonic partials, on the other hand, the resulting spectrogram has a not low-rank but sparse structure. Although RPCA is considered to be one of the most prominent ways of singing voice separation, non-repetitive instrument sounds are inevitably assigned to a sparse spectrogram. To filter out such non-vocal sounds, we estimate the F0 contour of singing voices from the sparse spectrogram based on a saliency-based F0 estimation method called subharmonic summation (SHS) [6] and extract only a series of harmonic structures corresponding to the estimated F0s. Here we propose a novel F0 saliency spectrogram in the time-frequency domain by leveraging the results of RPCA. This can avoid the negative effect of accompaniment sounds in vocal F0 estimation.
Our method is similar in spirit to a recent method of singing voice separation that combines rhythm-based and pitch-based methods of singing voice separation [7]. It first estimates two types of soft time-frequency masks passing only singing voices by using a singing voice separation method called REPET-SIM [8] and a vocal F0 estimation method (originally proposed for multiple-F0 estimation [9] integrated into a unified mask in a weighted manner. On the other hand, our method is deeply linked to human perception of a main melody in polyphonic music [10], [11]. Fig. 1 shows an instrumental composition of popular music. It is thought that humans easily recognize the sounds of rhythm instruments such as drums and rhythm guitars [10] and that in the residual sounds of non-rhythm instruments, spectral components that have predominant harmonic structures are identified as main melodies [11]. The proposed method first separates the sounds of rhythm instruments by using a time-frequency (TF) mask estimated by RPCA. Main melodies are extracted as singing voices from the residual sounds by using another mask that passes only predominant harmonic structures. Although the main melodies do not always correspond to singing voices, we do not deal with vocal activity detection (VAD) in this paper because many promising VAD methods [12]- [14] can be applied as pre-or post-processing of our method. The rest of this paper is organized as follows. Section II introduces related works. Section III explains the proposed method. Section IV describes the evaluation experiments and the MIREX 2014 singing-voice-separation task results. Section V describes the experiments determining robust parameters for the proposed method. Section VI concludes this paper.

II. RELATED WORK
This section introduces related works on vocal F0 estimation and singing voice separation. It also reviews some studies on the combination of those two tasks.

A. Vocal F0 Estimation
A typical approach to vocal F0 estimation is to identify F0s that have predominant harmonic structures by using an F0 saliency spectrogram that represents how likely the F0 is to exist in each time-frequency bin. A core of this approach is how to estimate a saliency spectrogram [15]- [19]. Goto [15] proposed a statistical multiple-F0 analyzer called PreFEst that approximates an observed spectrum as a superimposition of harmonic structures. Each harmonic structure is represented as a Gaussian mixture model (GMM) and the mixing weights of GMMs corresponding to different F0s can be regarded as a saliency spectrum. Rao et al. [16] tracked multiple candidates of vocal F0s including the F0s of locally predominant non-vocal sounds and then identified vocal F0s by focusing on the temporal instability of vocal components. Dressler [17] attempted to reduce the number of possible overtones by identifying which overtones are derived from a vocal harmonic structure. Salamon et al. [19] proposed a heuristicsbased method called MELODIA that focuses on the characteristics of vocal F0 contours. The contours of F0 candidates are obtained by using a saliency spectrogram based on subharmonic summation. This method achieved the state-of-the-art results in vocal F0 estimation.

B. Singing Voice Separation
A typical approach to singing voice separation is to make a TF mask that separates a target music spectrogram into a vocal spectrogram and an accompaniment spectrogram. There are two types of TF masks: soft masks and binary masks. An ideal binary mask assigns 1 to a TF unit if the power of singing voices in the unit is larger than that of the other concurrent sounds, and 0 otherwise. Although vocal and accompaniment sounds overlap with various ratios at many TF units, excellent separation can be achieved using binary masking. This is related to a phenomenon called auditory masking: a louder sound tends to mask a weaker sound within a particular frequency band [20].
Nonnegative matrix factorization (NMF) has often been used for separating a polyphonic spectrogram into nonnegative components and clustering those components into vocal components and accompaniment components [21]- [23]. Another approach is to exploit the temporal and spectral continuity of accompaniment sounds and the sparsity of singing voices in the TF domain [24]- [26]. Tachibana et al. [24], for example, proposed harmonic/percussive source separation (HPSS) based on the isotropic natures of harmonic and percussive sounds. Both components were estimated jointly via maximum a posteriori (MAP) estimation. Fitzgerald et al. [25] proposed an HPSS method applying different median filters to polyphonic spectra along the time and frequency directions. Jeong et al. [26] statistically modeled the continuities of accompaniment sounds and the sparsity of singing voices. Yen et al. [27] separated vocal, harmonic, and percussive components by clustering frequency modulation features in an unsupervised manner. Huang et al. [28] have recently used a deep recurrent neural network for supervised singing voice separation.
Some state-of-the-art methods of singing voice separation focus on the repeating characteristics of accompaniment sounds [5], [8], [29]. Accompaniment sounds are often played by musical instruments that repeat similar phrases throughout the music, such as drums and rhythm guitars. To identify repetitive patterns in a polyphonic audio signal, Rafii et al. [29] took the median of repeated spectral segments detected by an autocorrelation method, and improved the separation by using a similarity matrix [8]. Huang et al. [5] used RPCA to identify repetitive structures of accompaniment sounds. Liutkus et al. [30] proposed kernel additive modeling that combines many conventional methods and accounts for various features like continuity, smoothness, and stability over time or frequency. These methods tend to work robustly in several situations or genres because they make few assumptions about the target signal. Driedger et al. [31] proposed a cascading method that first decomposes a music spectrogram into harmonic, percussive, and residual spectrograms, each of which is further decomposed into partial components of singing voices and those of accompaniment sounds by using conventional methods [28], [32]. Finally, the estimated components are reassembled to form singing voices and accompaniment sounds.

C. One-way or Mutual Combination
Since singing voice separation and vocal F0 estimation have complementary relationships, the performance of each task can be improved by using the results of the other. Some vocal F0 estimation methods use singing voice separation techniques as preprocessing for reducing the negative effect of accompaniment sounds in polyphonic music [24], [29], [33], [34]. This approach results in comparatively better performance when the volume of singing voices is relatively low [35]. Some methods of singing voice separation use vocal F0 estimation techniques because the energy of a singing voice is concentrated on an F0 and its harmonic partials [32], [36], [37]. Virtanen et al. [32] proposed a method that first separates harmonic components using a predominant F0 contour. The residual components are then modeled by NMF and accompaniment sounds are extracted. Singing voices and accompaniment sounds are separated by using the learned parameters again.
Some methods perform both vocal F0 estimation and singing voice separation. Hsu et al. [38] proposed a tandem algorithm that iterates these two tasks. Durrieu et al. [39] used source-filter NMF for directly modeling the F0s and timbres of singing voices and accompaniment sounds. Rafii et al. [7] proposed a framework that combines repetition-based source separation with F0-based source separation. A unified TF mask for singing voice separation is obtained by combining the TF masks estimated by the two types of source separation in a weighted manner. Cabañas-Molero et al. [40] proposed a method that roughly separates singing voices from stereo recordings by focusing on the spatial diversity (called center extraction) and then estimates a vocal F0 contour for the separated voices. The separation of singing voices is further improved by using the F0 contour.

III. PROPOSED METHOD
The proposed method jointly executes singing voice separation and vocal F0 estimation (Fig. 2). Our method uses robust principal component analysis (RPCA) to estimate a mask (called an RPCA mask) that separates a target music spectrogram into low-rank components and sparse components. The vocal F0 contour is then estimated from the separated sparse components via Viterbi search on an F0 saliency spectrogram, resulting in another mask (called a harmonic mask) that separates harmonic components of the estimated F0 contour. These masks are integrated via element-wise multiplication, and finally singing voices and accompaniment sounds are obtained by separating the music spectrogram according to the integrated mask. The proposed method can work well for complicated music audio signals. Even if the volume of singing voices is relatively low and music audio signals contain various kinds of musical instruments, the harmonic structures (F0s) of singing voices can be discovered by calculating an F0 saliency spectrogram from an RPCA mask.
Overview of the proposed method. First an RPCA mask that separates low-rank components in a polyphonic spectrogram is computed. From this mask and the original spectrogram, a vocal F0 contour is estimated. The RPCA mask and the harmonic mask calculated from the F0 contour are combined by multiplication, and finally the singing voice and the accompaniment sounds are separated using the integrated mask.

A. Singing Voice Separation
Vocal and accompaniment sounds are separated by combining TF masks based on RPCA and vocal F0s.
1) Calculating an RPCA Mask: A singing voice separation method based on RPCA [5] assumes that accompaniment and vocal components tend to have low-rank and sparse structures, respectively, in the TF domain. Since spectra of harmonic instruments (e.g., pianos and guitars) are consistent for each F0 and the F0s are basically discretized at a semitone level, harmonic spectra having the same shape appear repeatedly in the same musical piece. Spectra of non-harmonic instruments (e.g., drums) also tend to appear repeatedly. Vocal spectra, in contrast, rarely have the same shape because the vocal timbres and F0s vary continuously and significantly over time.
RPCA decomposes an input matrix X into the sum of a low-rank matrix X L and a sparse matrix X S by solving the following convex optimization problem: where X, X L , and X S ∈ R T ×F , · * and · 1 represent the nuclear norm (also known as the trace norm) and the L1-norm, respectively. λ is a positive parameter that controls the balance between the low-rankness of X L and the sparsity of X S . To find optimal X L and X S , we use an efficient inexact version of the augmented Lagrange multiplier (ALM) algorithm [41]. When X is the amplitude spectrogram given by the shorttime Fourier transform (STFT) of a target music audio signal (T is the number of frames and F is the number of frequency bins), the spectral components having repetitive structures are assigned to X L and the other varying components are assigned to X S . Let t and f be a time frame and a frequency bin, respectively (1 ≤ t ≤ T and 1 ≤ f ≤ F ). We obtain a TF soft mask M (s) RPCA ∈ R T ×F by using Wiener filtering: RPCA ∈ R T ×F is also obtained by comparing X L with X S in an element-wise manner as follows: The gain γ adjusts the energy between the low-rank and sparse matrices. In this paper the gain parameter is set to 1.0, which was reported to achieve good separation performance [5]. Note that M

(b)
RPCA is used only for estimating a vocal F0 contour in Section III-B.
Using M (s) VOCAL ∈ R T ×F is roughly estimated as follows: where ⊙ indicates the element-wise product. If the value of λ for singing voice separation is different from that for F0 estimation, we execute two versions of RPCA with different values of λ (Fig. 2). If we were to use the same value of λ for both processes, RPCA would be executed only once. In section V we discuss the optimal values of λ in detail.
2) Calculating a Harmonic Mask: Using a vocal F0 con-tourŶ = {ŷ 1 ,ŷ 2 , · · · ,ŷ T } (see details in Section III-B), we make a harmonic mask M H ∈ R T ×F . Assuming that the energy of vocal spectra is localized on the harmonic partials of vocal F0s, we defined M H ∈ R T ×F as: where w(n; W ) denotes the n-th value of a window function of length W , f(h) denotes the index of the nearest time frame corresponding to a frequency h [Hz], n is the index of a harmonic partial, w is a frequency width [Hz] for extracting the energy around the partial, hŷ t is the estimated vocal F0 [Hz] of frame t. We chose the Tukey window whose a shape parameter is set to 0.5 as a window function.
3. An F0-saliency spectrogram is obtained by integrating an SHS spectrogram derived from a separated vocal spectrogram with an F0 enhancement spectrogram derived from an RPCA mask.
Furthermore, an integrated binary mask M RPCA+H is also defined as: Although the integrated masks have fewer spectral units assigned to singing voices than the RPCA mask and the harmonic mask do, they provide better separation quality (see the comparative results reported in Section V).
Using the integrated masks M ( * ) RPCA+H , the vocal and accompaniment spectrogramsX Finally, time signals (waveforms) of singing voices and accompaniment sounds are resynthesized by computing the inverse STFT with the phases of the original music spectrogram.

B. Vocal F0 Estimation
We propose a new method that estimates a vocal F0 contour Y = {ŷ 1 , · · · ,ŷ T } from the vocal spectrogram X RPCA and a vocal F0 contour is estimated by finding an optimal path in the saliency spectrogram with the Viterbi search algorithm.
1) Calculating a Log-frequency Spectrogram: We convert the vocal spectrogram X (b) VOCAL ∈ R T ×F to the log-frequency spectrogram X ′ VOCAL ∈ R T ×C by using spline interpolation on the dB scale. A frequency h f [Hz] is translated to the index of a log-frequency bin c (1 ≤ c ≤ C) as follows: where h low is a predefined lowest frequency [Hz] and p a frequency resolution [cents] per bin. The frequency h low must be sufficiently low to include the low end of a singing voice spectrum (i.e., 30 Hz).
To take into account the non-linearity of human auditory perception, we multiply the A-weighting function R A (f ) to the vocal spectrogram X This function is a rough approximation of the inverse of the 40-phon equal-loudness curve 1 and is used for amplifying the frequency bands that we are perceptually sensitive to, and attenuating the frequency bands that we are less sensitive to [19].
2) Calculating an F0-Saliency Spectrogram: Fig. 3 shows the procedure of calculating an F0-Saliency spectrogram. We calculate a subharmonic summation (SHS) spectrogram S SHS ∈ R T ×C from the tentative vocal spectrogram X ′ VOCAL ∈ R T ×C in the log-frequency domain. SHS [6] is the most basic and light-weight algorithm that underlies many vocal F0 estimation methods [19], [42]. S SHS is given by where c is the index of a log-frequency bin (1 ≤ c ≤ C), N is the number of harmonic partials considered, and β n is a decay factor (0.86 n−1 in this paper). We then calculate an F0 enhancement spectrogram S RPCA ∈ R T ×C from the RPCA mask M RPCA . To improve the performance of vocal F0 estimation, we propose to focus on the regularity (periodicity) of harmonic partials over the linear frequency axis. The RPCA binary mask M RPCA can be used for reducing half or double pitch errors because the harmonic structure of the singing voice strongly appears in it.
We first take the discrete Fourier transform (DFT) of each time frame of the binary mask as follows: This idea is similar to the cepstral analysis that extracts the periodicity of harmonic partials from log-power spectra. We do not need to compute the log of the RPCA binary mask because M RPCA ∈ {0, 1} T ×F . The F0 enhancement spectrogram S RPCA is obtained by picking the value corresponding to a frequency index c:  Finally, the reliable F0-saliency spectrogram S ∈ R T ×C is given by integrating S SHS and S RPCA as follows: where α is a weighting factor for adjusting the balance between S SHS and S RPCA . When α is 0, S RPCA is ignored, resulting in the standard SHS method. While each bin of S SHS reflects the total volume of harmonic partials, each bin of S RPCA reflects the number of harmonic partials.

3) Executing Viterbi Search:
Given the F0-saliency spectrogram S, we estimate the optimal F0 contourŶ = {ŷ 1 , · · · ,ŷ T } by solving the following problem: where c l and c h are the lowest and highest log-frequency bins of an F0 search range. G(y t , y t+1 ) is the transition cost function from the current F0 y t to the next F0 y t+1 . G(y t , y t+1 ) is defined as where b = 150 2 2 and c y indicates the log-frequency [cents] corresponding to log-frequency bin c. This function is equivalent to the Laplace distribution whose standard deviation is 150 [cents]. Note that the shifting interval of time frames is 10 [ms]. This optimization problem can be efficiently solved using the Viterbi search algorithm.

IV. EXPERIMENTAL EVALUATION
This section reports experiments conducted for evaluating singing voice separation and vocal F0 estimation. The results of the Singing Voice Separation task of MIREX 2014, which is a world-wide competition between algorithms for music analysis, are also shown.

A. Singing Voice Separation
Singing voice separation using different binary masks was evaluated to verify the effectiveness of the proposed method.

1) Datasets and Parameters:
The MIR-1K dataset 2 (MIR-1K) and the MedleyDB dataset (MedleyDB) [43] were used for evaluating singing voice separation. Note that we used the 110 "Undivided" song clips of MIR-1K and the 45 clips of MedleyDB listed in Table II. The clips in MIR-1K were recorded at a 16 kHz sampling rate with 16 bit resolution and the clips in MedleyDB were recorded at a 44.1 kHz sampling rate with 16 bit resolution. For each clip in both datasets, singing voices and accompaniment sounds were mixed at three signal-to-noise ratios (SNR) conditions: −5, 0, and 5 dB.
The datasets and the parameters used for evaluation are summarized in Table I, where the parameters for computing the STFT (window size and hopsize), SHS (the number N of harmonic partials), RPCA (a sparsity factor λ), a harmonic mask (frequency width w), and a saliency spectrogram (a weighting factor α) are listed. We empirically determined the parameters w and λ according to the results of grid search (see details in Section V). The same value of λ (0.8) was used for both RPCA computations in Fig.2. The frequency range for the vocal F0 search was restricted to 80-720 Hz.
2) Compared Methods: The following binary masks were compared.

RPCA-H-GT:
Using an integrated soft mask made by using a ground-truth F0 contour ISM: Using an ideal soft mask "RPCA" is a conventional RPCA-based method [5]. "H" used only a harmonic mask created from an estimated F0 contour. "RPCA-H-S" and "RPCA-H-B" represent the proposed methods using soft masks and binary masks, respectively, and "RPCA-H-GT" means a condition that the ground-truth vocal F0s were given (the upper bound of separation quality for the proposed framework). "ISM" represents a condition that oracle TF masks were estimated such that the ground-truth vocal and 2 https://sites.google.com/site/unvoicedsoundseparation/mir-1k accompaniment spectrograms were obtained (the upper bound of separation quality of TF masking methods). For H, RPCA-H-S and RPCA-H-B, the accuracies of vocal F0 estimation are described in Section IV-B.
3) Evaluation Measures: The BSS EVAL toolbox 3 [44] was used for measuring the separation performance. The principle of BSS EVAL is to decompose an estimateŝ of a true source signal s as follows: (17) where s target is an allowed distortion of the target source s and e interf , e noise and e artif are respectively the interference of the unwanted sources, perturbing noise, and artifacts in the separated signals (such as musical noise). Since we assume that an original signal consists of only vocal and accompaniment sounds, the perturbing noise e noise was ignored. Given the decomposition, three performance measures are defined: the Source-to-Distortion Ratio (SDR), the Source-to-Interference Ratio (SIR) and the Source-to-Artifacts Ratio (SAR): where · denotes a Euclidean norm. We then calculated the Normalized SDR (NSDR) that measures the improvement of the SDR between the estimateŝ of a target source signal s and the original mixture x. To measure the overall separation performance we calculated the Global NSDR (GNSDR), which is a weighted mean of the NSDRs over all the mixtures x k (weighted by their length l k ): In the same way, the Global SIR (GSIR) and the Global SAR (GSAR) were calculated from the SIRs and the SARs. For all these ratios, higher values represent better separation quality. Since this paper does not deal with the VAD and we intended to examine the effect of the harmonic mask for vocal separation, we used only the voiced sections for evaluation; that is to say, the amplitude of the signals in unvoiced sections was set to 0 when calculating the evaluation scores.

4) Experimental
Results: Fig. 4 shows the evaluation results. In spite of F0 estimation errors, the proposed methods using soft masks (RPCA-H-S) and those using binary masks (RPCA-H-B) outperformed both RPCA and H in GNSDR for all datasets. This indicates that combining an RPCA mask and a harmonic mask is effective for improving the separation quality of singing voices and accompaniment sounds. The removal of the spectra of non-repeating instruments (e.g., bass guitar) significantly improved the separation quality.   5 shows an example of an output of singing voice separation by the proposed method. We can see that vocal and accompaniment sounds were sufficiently separated from a mixed signal even though the volume level of vocal sounds was lower than that of accompaniment sounds.

B. Vocal F0 Estimation
We compared the vocal F0 estimation of the proposed method with conventional methods.  clips of popular music which were recorded at a 44.1 kHz sampling rate with 16 bit resolution. The dataset contains 20 songs with English lyrics performed in the style of American popular music in the 1980s and 80 songs with Japanese lyrics performed in the style of Japanese popular music in the 1990s.

2) Compared Methods:
The following four methods were compared. PreFEst [15] is a statistical multi-F0 analyzer that is still considered to be competitive for vocal F0 estimation. Although PreFEst contains three processes -the PreFEst-front-end for frequency analysis, the PreFEst-core computing a saliency spectrogram, and the PreFEst-back-end that tracks F0 contours using multiple agents -we used only the PreFEst-core and estimated F0 contours by using the Viterbi search described in Section III-B3 ("PreFEst-V"). MELODIA is a stateof-the-art algorithm for vocal F0 estimation that focuses on the characteristics of vocal F0 contours. We applied the Viterbi search to a saliency spectrogram derived from MELODIA ("MELODIA-V") and also tested the original MELODIA algorithm ("MELODIA"). In this experiment we used the MELO-DIA implementation provided as a vamp plug-in 4 .

PreFEst
Singing voice separation based on RPCA [5] was applied before computing conventional methods as preprocessing ("w/ RPCA" in Table III). We investigated the effectiveness of the proposed method in conjunction with preprocessing of singing voice separation.
3) Evaluation Measures: We measured the raw pitch accuracy (RPA) defined as the ratio of the number of frames in which correct vocal F0s were detected to the total number of voiced frames. An estimated value was considered correct if the difference between it and the ground-truth F0 was 50 cents (half a semitone) or less.  These results are originated from the characteristics of the proposed method. In vocal F0 estimation, the spectral periodicity of an RPCA binary mask is used to enhance vocal spectra. The harmonic structures of singing voices appear clearly in the RPCA mask when music audio signals contain various kinds of repetitive musical instrument sounds. The proposed method therefore works well especially for songs of particular genres such as rock and pops.

C. MIREX2014
We submitted our algorithm to the Singing Voice Separation task of the Music Information Retrieval Evaluation eXchange (MIREX) 2014, which is a community-based framework for the formal evaluation of analysis algorithms. Since the datasets are not freely distributed to the participants, MIREX provides meaningful and fair scientific evaluations.
There is some difference between our submission for MIREX and the algorithm described in this paper. The major difference is that only an SHS spectrogram (with the exception of an F0 enhancement spectrogram in Section III-B2) was used as a saliency spectrogram in the submission. Instead a simple vocal activity detection (VAD) method based on an energy threshold was used after singing voice separation.
1) Dataset: 100 monaural clips of pop music recorded at 44.1-kHz sampling rate with 16-bit resolution were used for evaluation. The duration of each clip was 30 seconds.
2) Compared Methods: 11 submissions participated in the task 5  gorithms using deep recurrent neural networks [28]. YC1 separates singing voices by clustering modulation features [27]. RP1 is the REPET-SIM algorithm that identifies repetitive structures in polyphonic music by using a similarity matrix [8]. GW1 uses Bayesian NMF to model a polyphonic spectrogram, and clusters the learned bases based on acoustic features [23]. JL1 uses the temporal and spectral discontinuity of singing voices [26], and LFR1 uses light kernel additive modeling based on the algorithm in [30]. RNA1 first estimates predominant F0s and then reconstructs an isolated vocal signal based on harmonic sinusoidal modeling using estimated F0s. IIY1 and IIY2 are our submissions. The only difference between IIY1 and IIY2 is their parameters. The parameters for both submissions are listed in Table IV.
3) Evaluation Results: Fig. 6 shows the evaluation results for all submissions. Our submissions (IIY1 and IIY2) provided the best mean NSDR for both vocal and accompaniment sounds. Even though the submissions using the proposed method outperformed the state-of-the-art methods in MIREX 2014, there is still room for improving their performances. As described in Section V-A, the robust range for the parameter w is from 40 to 60. We set the parameter to 100 in the submissions, however, and that must have considerably reduced the sound quality of both separated vocal and accompaniment sounds.
V. PARAMETER TUNING In this section we discuss the effects of parameters that determine the performances of singing voice separation and vocal F0 estimation.

A. Singing Voice Separation
The parameters λ and w affect the quality of singing voice separation. λ is the sparsity factor of RPCA described in Section III-A1 and w is the frequency width of the harmonic mask described in Section III-A2. The parameter λ can be used to trade off the rank of a low-rank matrix with the sparsity of a sparse matrix. The sparse matrix is sparser when λ is larger and is less sparse when λ is smaller. When w is smaller, fewer spectral bins around an F0 and its harmonic partials are assigned as singing voices. This is the recall-precision tradeoff of singing voice separation. To examine the relationship between λ and w, we evaluated the performance of singing voice separation for combinations of λ from 0.6 to 1.2 in steps of 0.1 and w from 20 to 90 in steps of 10.
1) Experimental Conditions: MIR-1K was used for evaluation at three mixing conditions with SNRs of −5, 0, and 5 dB. In this experiment, a harmonic mask was created using a ground-truth F0 contour to examine only the effects of λ and w. GNSDRs were calculated for each parameter combination.
2) Experimental Results: Fig. 7 shows the overall performance for all parameter combinations. Each unit on a grid represents the GNSDR value. It was shown that λ from 0.6 to 1.0 and w from 40 to 60 provided robust performance in all mixing conditions. In the −5 dB mixing condition, an integrated mask performed better for both of the singing voice and the accompaniment when w was smaller. This was because most singing voice spectra were covered by accompaniment spectra and only few singing voice spectra were dominant around an F0 and harmonic partials in the condition.

B. Vocal F0 Estimation
The parameters λ and α affect the accuracy of vocal F0 estimation. λ is the sparsity factor of RPCA and α is the weight parameter for computing the F0-saliency spectrogram described in Section III-B2. α determines the balance between an SHS spectrogram and an F0 enhancement spectrogram in a F0-saliency spectrogram, and there must be range of its value that provides robust performance. We evaluated the accuracy of singing voice separation for combinations of λ from 0.6 to 1.1 in steps of 0.1 and α from 0 to 2.0 in steps of 0.2. RWC-MDB-P-2001 was used for evaluation, and RPA was measured for each parameter combination. Fig. 8 shows the overall performance for all parameter combinations of grid search. Each unit on a grid represents RPA for each parameter combination. It was shown that λ from 0.7 to 0.9 and α from 0.6 to 0.8 provided comparatively better performance than any other parameter combinations. RPCA with λ within the range separates vocal sounds to a moderate degree for vocal F0 estimation. The value of α was also crucial to estimation accuracy. The combinations with α = 0.0 yielded especially low RPAs. This indicates that an F0 enhancement spectrogram was effective for vocal F0 estimation. VI. CONCLUSION This paper described a method that performs singing voice separation and vocal F0 estimation in a mutually-dependent manner. The experimental results showed that the proposed method achieves better singing voice separation and vocal F0 estimation than conventional methods do. The singing voice separation of the proposed method was also better than that of several state-of-the-art methods in MIREX 2014, which is an international competition in music analysis. In the experiments on vocal F0 estimation, the proposed method outperformed two conventional methods that are considered to achieve the state-of-the-art performance. Some parameters of the proposed method significantly affect the performances of singing voice separation and vocal F0 estimation, and we found that a particular range of those parameters results in relatively good performance in various situations.
We plan to integrate singing voice separation and vocal F0 estimation in a unified framework. Since the proposed method performs these tasks in a cascading manner, separation and estimation errors are accumulated. One promising way to solve this problem is to formulate a unified likelihood function to be maximized by interpreting the proposed method from a viewpoint of probabilistic modeling. To discriminate singing voices from musical instrument sounds that have sparse and non-repetitive structures in the TF domain like singing voices, we attempt to focus on both the structural and timbral characteristics of singing voices as in [35]. It is also important to conduct subjective evaluation to investigate the relationships between the conventional measures (SDR, SIR, and SAR) and the perceptual quality.