{'fraction_non_alphanumeric': 0.033271127699280194, 'fraction_numerical': 0.007571314316182351, 'mean_word_length': 5.003841229193342, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
MedleyVox: An Evaluation Dataset for Multiple Singing Voices Separation

Separation of multiple singing voices into each voice is a rarely studied area in music source separation research. The absence of a benchmark dataset has hindered its progress. In this paper, we present an evaluation dataset and provide baseline studies for multiple singing voices separation. First, we introduce MedleyVox, an evaluation dataset for multiple singing voices separation that corresponds to such categories. We specify the problem definition in this dataset by categorizing the problem into i) duet, ii) unison, iii)main vs. rest, and iv) N-singing separation. Second, we present a strategy for construction of multiple singing mixtures using various single-singing datasets. This can be used to obtain training data. Third, we propose the improved super-resolution network (iSRNet). Jointly trained with the Conv-TasNet and the multi-singing mixture construction strategy, the proposed iSRNet achieved comparable performance to ideal time-frequency masks on duet and unison subsets of MedleyVox. Audio samples, the dataset, and codes are available on our GitHub page (https://github.com/jeonchangbin49/MedleyVox).


INTRODUCTION
Singing voice separation (SVS) research has grown remarkably with the progress of deep learning [1] and high-quality benchmark data [2]. The most popular SVS task has been separating singing voices from their accompaniments. Recently, choral music separation [3][4][5][6][7] gained attention, which is the SVS task of extracting individual voices from a choral singing mixture consists of soprano, alto, tenor, and bass parts. However, multiple singing voices separation applicable to popular music remains unstudied.
The two main reasons for the slow advances of multiple singing voices separation are the absence of a benchmark dataset and baseline studies. A benchmark dataset is essential for problem understanding. For example, in speech separation research, it was reported in [8] that recent deep learning-based models have almost achieved the performance of the upper bound. This kind of study can only be performed when there exist high-quality benchmark data [9,10].
In this paper, we introduce MedleyVox, an evaluation benchmark data for multi-singing separation. While building the dataset, we categorize the problem of multiple singing voices separation into four categories. It is done based on the various musical properties of singing mixtures, which affects the separation difficulty. This cate- gorization is applied to the items of MedleyVox for detailed performance analysis.
Another challenge for multi-singing separation is that there does not exist any baseline studies including publicly available train data made for such task. To handle the issues, we present a multi-singing mixture construction strategy that leverages various single-singing and speech data for training multi-singing separation models. We also investigate the appropriate analysis/synthesis basis and objective functions for multi-singing separation, and propose the improved super-resolution network (iSRNet) inspired by the recent state-of-the-art speech separation model [11]. Jointly trained with the Conv-TasNet [12], the proposed iSRNet achieves comparable performance to ideal time-frequency masks on duet and unison separation of MedleyVox.
Our work bridges the gap between speech and singing voice separation research. Challenging aspects of multi-singing separation come from the input mixture source consisting of highly correlated vocals. They often have the same pitch with simultaneous on-offset and even consist of segments sung by identical singers, which makes the task more challenging compared to speech separation where speech mixtures rarely consist of one speaker's different speeches. We give the baseline studies and carefully analyze these challenges for future research on multi-singing separation.

MEDLEYVOX
MedleyVox is a benchmark dataset for multiple singing voices separation. To extend the singing voice separation research towards multi-singing separation, we propose the following four categories that constitute the separation of multiple singing voices. We assume the ideal circumstance where the vocals are cleanly separated from accompaniments by other singing voice separation systems.
i) Duet is the separation where a mixture of two singing voices is given that have different melodies or on/offset patterns, or lyrics.
ii) Unison is the separation where a mixture of two singing voices is given that have identical or octave-shifted melodies, same on/offset patterns, and lyrics. Unison is frequently used in both the popular and choral music for making sound perceptually richer. Although both unison and duet consist of two singing voices, we separated them into different categories due to the expected difficulty of unison separation, where two singing voices have much higher correlation than duet.
iii) Main vs. rest is the separation of a main vocal and the remaining vocals where more than three singing voices exist. Each singing voice is mostly composed with different melodies or onoffset patterns, however, identical melodies and on-offset patterns are possible 2 . We expect main vs. rest perhaps can be a useful category for commercial usage, e.g., a karaoke system; imagine one sings a main vocal part of a song with the separated accompaniment that the rest chorus vocals still exist so that enrich the one's singing, unlike previous singing voice separation, which removes every singing to make the accompaniments. iv) N-singing is the separation of a multi-singing mixture consists of an unknown number of singings into N each singing. This includes all the three categories introduced earlier. The naming of N-singing is inspired by N-speaker separation in speech separation field. Notice that we used N-singing instead of N-singer because a mixture can consist of the same singer's different singings. Also, note that we leave this task as a future work due to its difficulty.
For all the four categories of multi-singing voices separation, we introduce MedleyVox dataset for an evaluation of such task. We collected the vocal tracks of 23 songs in MedleyDB v1 and v2 [13,14], then manually categorized the voiced segments of each track into i) duet, ii) unison, and iii) main vs. rest. We also provide annotations of both the number of singing voices and the number of total singers. For example, if duet or main vs. rest consist of singing voices from one singer, we annotated them as one singer. Also, if unison is performed by two different singers, we annotated them as two singers. Data specifications are depicted in Table 1. Note that N-singing can be easily obtained using the metadata of each duet, unison, and main vs. rest.

Construction of Multi-singing Mixture
We propose a strategy of using various single-singing voice data to construct each training item, which is a multi-singing mixture. We randomly sample different singing sources and mixed them to make the mixture like a dynamic mixing strategy in speech separation [15]. Specifically, we consider input training mixture on i) duet as a combination of two single-singing examples, ii) unison as a mixture of an example and its pitch (±1 octave), formant-shifted example, and iii) main vs. rest as a summation of one loudest single-singing and a mixture of an arbitrary number of singing voices. Also, considering the high correlation of each vocal in multi-singing mixtures frequently exist in popular music, we construct a mixture using identical singer's different singings and the same song sung by different singers with certain probabilities.

STFT/iSTFT-based Analysis and Synthesis
In speech separation, the best-performing model [11] currently is based on the learnable analysis encoder and synthesis decoder given time-domain waveform in-output [12]. However, short-time Fourier transform (STFT) analysis and inverse STFT (iSTFT) synthesis were found effective in universal sound separation [16], where the inoutput distributions are expected to be broader than speech. Similarly, we assume that STFT/iSTFT-based analysis-synthesis framework may outperform the learnable encoder-decoder model in our task because singing shows a broader distribution than speech, e.g., wider fundamental frequency ranges, richer harmonic structures, and longer vocalization. Specifically, we use complex spectral mapping strategy following recent state-of-the-art speech separation [17] and singing voice separation [18,19] models, which directly estimates real and imaginary coefficients of STFT given an input mixture.

Objective Functions
Scale-invariant signal-to-distortion ratio (SI-SDR) [20] is a popular training objective in recent speech separation [11,12]. However, we found that this objective leads to a huge scale of output waveform, which results in a significant clipping error when saving the output in digital format. To handle this, we use a mixture consistency training [21]. Furthermore, we replace SI-SDR with a signal-to-noise ratio (SNR) objective and add a multi-resolution STFT loss [22] to enhance the perceptual quality of network outputs. Since we use a complex spectral mapping strategy for training, we also calculated STFT loss on real-imaginary coefficients of STFT.

Permutation Invariant Training
In previous singing voice separation or choral music separation, target output source channels are fixed, e.g., vocals and accompaniments [19], or one of four-part harmony [3,4]. However, in multisinging separation, permutation problem [23] exists as it does in speech separation. For example, in duet separation, neural networks have no prior about what output channel should be assigned to a specific voice.
To this end, we used utterance-level permutation invariant training (uPIT) [24] for duet and unison separation where only two vocals exist in input mixture and one-and-rest PIT (OR-PIT) [25] for main vs. rest separation. Especially when using OR-PIT, one major difference between main vs. rest and speech separation [25] is that we can use prior knowledge that the main vocal is louder than the rest vocals in general for popular music. Hence, we intentionally make the main vocal louder than rest vocals during training mixture construction process, thereby no need to calculate objectives for all possible ones and rest assignment; if there are N single-singing sources in a mixture, the original OR-PIT calculates objectives for N possible one-and-rest splits and uses the assignment that has the lowest loss for training. Fig. 2. Overview of (a) the proposed iSRNet and (b) Heuristic method. The iSRNet enhances the initial estimates of a primary network (e.g., Conv-TasNet) given concatenated inputs consist of mixture, initial estimates, and heuristic magnitude STFT.

Improved Super-resolution Network
Inspired by the recent state-of-the-art super-resolution network (SR-Net) -based framework on speech separation [11], we propose the improved SR network (iSRNet) for multi-singing separation. As depicted in Fig 2, iSRNet enhances the initial estimates of a backbone separation network, e.g., Conv-TasNet. We replace the original convolutional layers of SRNet with ConvNeXt blocks [26] for efficiency and performance. Also, unlike the original heuristic method [11] of dividing upper and lower frequency ranges roughly by half the Nyquist frequency, we explore the frequency boundaries that separate the frequency regions since we found that Conv-TasNet for multi-singing separation suffers from leakage above 1-5kHz frequency ranges.

Overview
In Exp 1, we first take an experiment on the appropriate analysissynthesis frameworks and multi-singing mixture construction strategy (Table 2, 16 kHz sample rate). In Exp 2, we then investigate the effect of mixture consistency loss for preventing output scale exploding (Table 3, 24 kHz sample rate). In Exp 3, we first train the Conv-TasNet-L baseline model, which has an increased channel size from the Conv-TasNet model trained with mixture consistency loss in Exp 2. Then, we improve it by adding SNR, multi-resolution STFT objectives, and using the proposed iSRNet (Table 4, 24 kHz sample rate). In Exp 4, we train main vs. rest separation network (Table 5) with the best-performed setting on duet separation. In Exp 5, ablation studies on the proposed iSRNet are conducted.

Result and Discussion
Exp 1 The results in Table 2 show that on duet separation, STFTbasis slightly outperforms the Conv-TasNet-learn in multi-singing  Table 3. Evaluation on the mixture consistency training.
separation. Although the evaluation scores are similar, we observe that Conv-TasNet-learn consistently generates crunchy noise unlike Conv-TasNet-STFT. It is also reported in [19] that the waveformbased model for singing voice separation is prone to such artifacts. For subjectively high-quality outputs, we use STFT basis for the following experiments. We also observe that our mixture construction strategy -which uses additional speech data and constructing the mixture of the identical singer's different singings or the same song from different singers -is helpful. Nevertheless, it is noticeable that the model only trained with different singers' different singings (w/ Only diff singings in Table 2) achieves fair quality contrary to [5], where the random singing mixing for training data resulted in a huge performance decrease. This is perhaps because we used 400 hours of singing data, significantly larger than 104 minutes of [5].
Exp 2 As in Fig 3, without mixture consistency training, the model suffers from output scale exploding due to the scale-invariant nature of SI-SDR objective. This results in significant performance degradation in Table 3 Table 4. Evaluation results on investigating optimal objectives, the proposed iSRNet, and ideal time-frequency masks on duet and unison separation.  Table 5. Evaluation results of the baseline, the proposed iSRNet, and ideal time-frequency masks on main vs. rest separation.
not a solution since keeping the scale of in-outputs is relevant for the application usage of singing voice separation, e.g., a main vocal removal from a multi-singing mixture for a karaoke system. To handle the issue, the mixture consistency loss is used by default for the following experiments. Exp 3 We improve our Conv-TasNet-L baseline model with using new objectives and networks. As in Table 4, multi-resolution STFT loss with SNR objective outperforms the baseline models. With additional iSRNet, our model achieves 1 dB of performance gain compared to the baseline on SI-SDRi.
On unison separation also in Table 4, our model shows comparable performance to ideal masks and even outperforms ideal ratio mask (IRM) on SDRi. Comparing the performance of the ideal binary mask (IBM) and IRM on duet and unison, we notice that the scores on unison is more than 10 dB lower than the scores on duet. This implies that phase information is critical for unison separation. Unlike our model which is trained to work on both duet and unison separation, a model that is trained solely for unison with cIRM objective (e.g., [40]) can be an option to maximize the performance in unison, but it is out of the scope of this paper.
Exp 4 We validate main vs. rest separation model with the bestperformed setting in duet separation, which is, SNR loss combined with multi-resolution STFT loss (Conv-TasNet-L in Table 5) and the additional iSRNet (w/ iSRNet in Table 5). Similar to duet separation, the proposed iSRNet brings performance gain compared to the baseline model. However, there is a significant performance gap between the oracle methods (e.g., SI-SDRi 13.2 dB on IRM) and the proposed iSRNet (6.3 dB). We will discuss this failure in a later section.
Exp 5 In ablation studies on the proposed iSRNet as in Table 6, by simply replacing the original convolutional layers of SRNet into ConvNeXt blocks, the performance rises 0.5dB with 40 times smaller number of trainable parameters and computational costs. Also, the model achieves the best performance when using a heuristic frequency upper boundary at 3 kHz.
Analysis On main vs. rest separation, the model frequently suffers from the circumstance that the section originally corresponds to silence regions in a main vocal part wrongly filled with one of the rest vocals. This is mainly because we train the model using OR-PIT with the input mixture in which the main vocal is louder than the rest, whereas silence regions are quieter than the rest vocals. Also, the model suffers from extracting an unintended vocal for the main. This is also because of our OR-PIT training scheme; the main vocal is louder for most popular music, however, not always. We consider that the approach of separating every source in a mixture at once [41] could be the solution for these issues, which is also directly applicable to N-singing separation.
Future works Since we performed the experiments on segmentwise, long input sequences such as 3-4 minutes songs have to be considered in future works for application usage. Unfortunately, plain chunk-wise processing that has been frequently used in continuous speech separation [42,43], which assigns the output speakers based on the similarity of overlapped regions is not feasible to music because the multi-singing mixtures possibly contain much longer silences unlike speeches; appropriate chunk window and hop sizes would be a solution but it seems impossible to find a universal setting that perfectly fits every song.
Chunk-wise processing that is robust on long silences proposed in [44] can be another option to handle this. However, the question remains because such method used speaker information for processing and assignments. Since multi-singing mixtures are often composed with identical singer's singings, assignments based on singer representations can lead to sub-optimal results. To overcome this, it is necessary to study the robust representations for singer assignments that reflect not only speaker-related information such as timbre, but also context-related information such as melodic sequences.

CONCLUSIONS
In this paper, we introduced MedleyVox, an evaluation dataset for separation of multiple singing voices into each constituting singings. We presented a multi-singing mixture construction strategy that uses various single-singing and speech data to overcome the absence of publicly available training data for multi-singing separation networks. Also, the proposed iSRNet achieved superior performance on duet and unison separation when jointly trained it with the Conv-TasNet. Lastly, we thoroughly analyzed the remaining challenges for multi-singing separation.