{'fraction_non_alphanumeric': 0.03241093801067654, 'fraction_numerical': 0.008170824708573918, 'mean_word_length': 5.233955857385399, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Multi-input Architecture and Disentangled Representation Learning for Multi-dimensional Modeling of Music Similarity

In the context of music information retrieval, similarity-based approaches are useful for a variety of tasks that benefit from a query-by-example scenario. Music however, naturally decomposes into a set of semantically meaningful factors of variation. Current representation learning strategies pursue the disentanglement of such factors from deep representations, resulting in highly interpretable models. This allows the modeling of music similarity perception, which is highly subjective and multi-dimensional. While the focus of prior work is on metadata driven notions of similarity, we suggest to directly model the human notion of multi-dimensional music similarity. To achieve this, we propose a multi-input deep neural network architecture, which simultaneously processes mel-spectrogram, CENS-chromagram and tempogram in order to extract informative features for the different disentangled musical dimensions: genre, mood, instrument, era, tempo, and key. We evaluated the proposed music similarity approach using a triplet prediction task and found that the proposed multi-input architecture outperforms a state of the art method. Furthermore, we present a novel multi-dimensional analysis in order to evaluate the influence of each disentangled dimension on the perception of music similarity.


Introduction
Music information retrieval (MIR) approaches traditionally aim to automatically annotate music recordings with textual annotations that describe their musical attributes. Such annotations form semantic spaces within data collections, which structure the music library and allow its efficient browsing.
However, the retrieval of songs based on these semantic spaces requires a relatively high musical understanding by part of the user in its query. In most scenarios, arguably, the user is not really sure how to textually describe the desired musical traits, but has an example in another song of the similar characteristics or "feel" searched for. In such cases of higher subjectivity, a query-by-example approach is convenient, as it simplifies the retrieval task for the user. Another application scenario is music replacement, where a seed musical track needs to be replaced during the production process, for instance due to the licensing restrictions, but still a similar musical track is wanted. For this purposes, music similarity-based retrieval algorithms are developed.
Moreover, music similarity is a highly subjective concept, which is considered inherently multi-dimensional [1]. Different musical pieces can be considered as similar (or dissimilar) over one or more dimensions of the different facets they present, such as their specific music characteristics (e.g. rhythm, tempo, key, melody, instrumentation), semantic conceptualizations (e.g. genre, style, theme), perceived moods (e.g. happy, aggressive, calm), listening situations (e.g. to work-out, to study, to party), or musicological factors (e.g. composer, influences), among others. One of the greatest * Submitted to ICASSP 2022 arXiv:2111.01710v1 [eess.AS] 2 Nov 2021 challenges for music similarity algorithms is to identify which combination of these dimensions are most relevant to a particular user when rating two pieces as similar.
MIR approaches are currently based on learning efficient hierarchical representations from input data by deep neural networks. Recent representation learning approaches for music similarity [2,1] focus on the disentanglement of the multi-dimensionality inherent in deep representations. This step will allow to perform a wide range of music retrieval applications with a higher efficiency [2] and to configure the extent to which the different disentangled musical dimensions are considered in the latent space of the network. Thus, a single framework can potentially be used to generate adjustable similarity spaces, targeting tasks with different degrees of specificity, as shown in Fig. 1. Furthermore, disentangled representations allow the modeling of inter-dimensional relations in user queries. This is of particular relevance for mid/low specificity tasks, in which the notion of similarity is increasingly subjective.  Figure 1: Specificity spectrum for music similarity-based retrieval tasks (inspired by [3]). Examples are allocated along the horizontal axis according to the degree of specificity they convey. The elliptical shapes illustrate the similarity spaces generated by the configurable musical dimensions.

Specificity
In this paper, we expand on recent work on disentanglement representation learning [2] and employ a triplet-based metric learning approach to disentangle six musical concepts: genre, mood, instrumentation, era, tempo and key. We introduce a novel multi-input deep architecture in order to disentangle these concepts. In this way, we incorporate both low-level and mid-level musical features in the representation space of the network, instead of relying solely on high-level conceptualizations. We employ a perceptual similarity approach, in which triplets containing human annotations of music similarity are used to evaluate different configurations of our proposed model. Additionally, we conduct a novel multi-dimensional analysis to study the influence of the disentangled musical concepts on the human perception of music similarity.
2 Related Work

Music Similarity & Deep Metric Learning
Many tasks in MIR such as music replacement, music recommendation, and music playlist generation fall naturally into a query-by-example setting, where a user queries the system by providing a song or a fragment of it and the system responds by retrieving a list of relevant or similar songs [4]. Such applications do not require textual annotations, but ultimately depend on the notion of music similarity between songs to produce high-quality results.
Deep metric learning is a representation learning approach, which allows to quantify such abstract notions of similarity. Through a distance metric, it aims to map data to an embedding space, in which similar data-points are close together and dissimilar data-points are further away from each other. One common deep metric learning paradigm includes triplets of samples as input in the form of an anchor, a positive (similar to the anchor), and a negative (dissimilar to the anchor) sample. This relationship between samples can be exploited to train a deep architecture [5] using an embedding loss, namely the triplet margin loss [6].
Although training may be slower compared to other metric learning paradigms [7], the triplet-based learning approach offers a higher flexibility, since its form of supervision comprises a relative comparison. Recently, sampling strategies to generate triplets that lead to a faster convergence have been proposed, such as semi-hard negative mining [8] or distance weighted sampling [9].

Disentangled Representation Learning
Apart from deep metric learning, the idea of learning disentangled representations has also attracted a lot of attention in the machine learning research community in recent years. In general, it is considered that the disentanglement of representations is an important step towards an improved representation learning [10,11]. In the music domain, disentanglement learning has mostly been developed in the field of music generation for tasks, such as genre transfer [12], timbre and pitch synthesis [13] [14], instrument rearrangement [15], musical attributes manipulation [16] [17], and rhythm transfer [18] [19]. Recently however, a disentangled metric learning approach for music similarity has been proposed by Lee et al. [1]. The authors adapt the conditional similarity network (CSN) [20] model from the computer vision domain (image similarity) to the music domain. It essentially consists in the expansion of the basic triplet metric learning network [5] to include dimensional masks, which activate certain portions of the embeddings generated by a backbone architecture during training. In this way, equally-sized portions of the embedding space of the network are assigned for metric learning of the different concepts to disentangle.

Proposed Method
We propose to expand the CSN architecture for music similarity to disentangle six musical dimensions: genre, mood, instrument, era, tempo, and key.

Audio Pre-Processing & Input Features
We train the network using three second long audio segments of samples from the Million Song Dataset (MSD) [21]. Following previous work [22,2], we group the Last.FM tags associated with the MSD files into four dimensions: genre, mood, instrument, and era. We use the Madmom Python library [23] to extract key [24] and tempo [25] information from the dataset files.
Using the librosa library [26], we compute three different input representations of the dataset files: the mel-spectrogram, the cyclic tempogram [27], and the chroma energy normalized statistics (CENS-chromagram) [28]. For the melspectrograms, a window size of 23 ms with 50 % overlap is used and 128 mel-bands per frame are computed with logarithmic compression as log 10 (1 + 10 * S), where S corresponds to the energy-magnitude spectrogram. Additionally, the resulting mel-spectrograms are z-score standardized [1].
For the tempograms and CENS-chromagrams, the same window size and hop size (23 ms and 50 % overlap, respectively) is used to compute the short-time Fourier transform (STFT). In this way, all input representations have a concurring temporal axis, which simplifies the sampling process. However, the length of the onset auto-correlation window in the tempogram is set to the default value of 384 bins; while the number of chroma for the CENS-chromagram is set to 12 (one for each note of the chromatic scale), computed from a CQT with a resolution of 36 bins per octave. . This architecture consists of a basic block of convolutional and maxpooling operations processing the single input of the network (mel-spectrogram), followed by six Inception blocks, each consisting of two Inception modules [23]: a "naïve" module, followed by a "dimension reduction" module. The final layer is a dense layer with the same number of neurons as the resulting embedding size. Layer normalization [29] is applied to the generated embeddings (i.e., embedding-wise normalization) in order to increase efficiency during training [30].

Neural Network Architecture
Furthermore, we propose a variant of the baseline embedding architecture (Fig. 2b). It consists on the parallel processing of the mel-spectrogram, the tempogram and CENS-chromagram of the input samples. Thus, a processing branch is created for each input, conformed by a convolutional layer, and followed by two inception blocks. For the CENSchromagram branch an additional zero-padding layer is employed to upsample the input along the frequency axis to 16 bins. In this way, the outputs of all branches are 16x16 feature maps, which are concatenated and further processed in the same way as for the baseline model.

Conditional Triplet Sampling & Learning Strategy
The employed learning method requires sampling triplets of the form t s = (x a , x p , x n ; s)  where x a , x p , and x n are the anchor, positive and negative samples, respectively, within a considered embedding dimension s. We employ a conditional triplet loss L s (t s ) [1] for each dimension individually, and average them to produce the total multi-dimensional loss of the network L MD for N D number of dimensions: For the triplet sampling, we consider certain notions of similarity. For the dimensions inferred from the dataset labels (genre, mood, instrument, and era), we form anchor-positive pairs from samples that simply present the same label. For tempo, we consider two samples as similar if the extracted global tempo of their respective songs are within a 5 bpm margin (to compensate for possible noise in the tempo estimation algorithm) or if they are any number of tempo-octaves higher or lower, following previous work done on tempo similarity [31]. Finally, for key, samples are considered similar if the extracted key of their respective songs are the same, or if they are parallel keys. Additionally, we employ semi-hard negative mining [8] to find negative samples that are close to the anchor, but not as close as the positive sample, within each musical dimension individually. Negatives are mined within a margin value of 0.1 of the anchor-positive distance to avoid easy negatives.

Training Parameters
All model-variations were trained using "Adam" optimizer [32] with an initial learning rate of 1e −5 , which is reduced by a factor of 5 each time the validation loss does not decrease after 10 epochs, up to a minimum learning rate of 1e −10 . Cosine distance was selected as the distance metric for training.

Dataset & Evaluation Metric
In an evaluation phase, we test the performance of the disentangled embeddings obtained from the implemented method (CSNs) for perceptual music similarity. To do so, we employ the recently published Dim-sim dataset [1]. It consists of 4,000 triplets sampled from the test split of the MSD, each with human similarity annotations. However, as all similarity annotations are subjective, we make use of a refined split of 426 high-agreement triplets with an agreement rate equal or above 90%. Embeddings are extracted from three second long non-overlapping sections and averaged over the full song duration in order to obtain the track embeddings used for evaluation.
We use the triplet prediction score as evaluation metric, calculating the fraction of correct predictions given by the evaluated model for the ground-truth triplets in the dataset. Here, the predictions are regarded as correct if the computed euclidean distance in the embedding space between the anchor and positive sample is smaller than that of the anchor and negative. In this way, a measure of concurrence between the trained model predictions of similarity within a triplet, and the human perception of triplet similarity is obtained.

Experiment 1 -Model Performance
To test model performance of different configurations, the entirety of the embedding space is activated (no masks) while processing the evaluation triplets. Tab. 1 displays the results obtained. Models with four dimensions include exclusively the disentanglement of the concepts present in the MSD-labels, while six-dimensional models include, additionally, the tempo and key dimensions. The "baseline" configuration refers to the CSN model with the standard backbone network and all parameters set mirroring the original study [2]. As an additional benchmark, we evaluate the OpenL3-embeddings [33], also averaged over the full song duration in the same manner as the disentangled embedding.
It is observed that solely increasing the number of dimensions with respect to the baseline does not provide an improvement in performance. However, by employing the proposed multi-input architecture and keeping the same number of dimensions, already a slight improvement is observed. The best result is obtained for the six-dimensional multi-input architecture. It further improves the triplet prediction performance compared to all other configurations including the baseline. These observations are indicative that higher-level dimensions (genre, mood, instrument, and era) also make use of the information present in the two additional input representations; and that solely the mel-spectrogram does not provide sufficient informative cues for the tempo and key dimensions.

Experiment 2 -Multi-Dimensional Analysis of Similarity Perception
To carry out the multi-dimensional analysis, the best performing model (see Tab. 1) is evaluated by activating only particular sub-spaces of the produced embeddings, which correspond to the different disentangled concepts. The top half of Tab. 2 shows the top six combinations of disentangled concepts, which yielded the best triplet prediction results; while the bottom half displays the results when activating each individual concept. Figure 3 depicts these last results in a Kiviat diagram, in order to obtain an improved visualization of the influence each individual concept has over the human perception of music similarity, according to the implemented deep model.  Figure 3: Influence of each musical dimensions individually.
We observe that the combination of all musical dimensions produces the best result for perceptual triplet prediction. This was expected, since having all dimensions activated provides the most informative representation. It can be observed that genre and era are very good indicatives for music similarity, since they are present in all the best combinations. This is further confirmed when evaluating each dimension separately. Somewhat surprisingly, the era dimension presents the higher rating, followed closely by the genre and mood dimensions. These results are a clear indicative that higher-level conceptualizations in music (genre, mood, and era) are individually more informative for music similarity, in comparison to mid/low-level notions (instrument, key, and tempo). This can be attributed to the fact that higher-level dimensions may already contain some information about them in their encoded representation. Still, the disentanglement of these low-level concepts provide a higher degree of informative cues about them, which benefits the entire combination.

Conclusion
In this paper, we introduce a multi-input deep architecture tailored for the disentanglement of both high-level semantic conceptualizations (genre, mood, and era), as well as mid/low-level descriptors (instrument, key, and tempo) of music from deep representations. We present a perceptual evaluation of music similarity using a disentanglement metric learning approach. The proposed architecture outperforms the baseline when evaluated with human-annotated similarity triplets from the Dim-sim dataset. Through the produced disentangled representations, we present a multi-dimensional analysis of perceptual music similarity, showing the influence of each musical dimensions individually. In this way, it is observed for the first time how a deep architecture models the multi-dimensionality of the human notion of music similarity.