{'fraction_non_alphanumeric': 0.03597612958226769, 'fraction_numerical': 0.011423699914748509, 'mean_word_length': 4.82842000662471, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Self-Supervised Hierarchical Metrical Structure Modeling

We propose a novel method to model hierarchical metrical structures for both symbolic music and audio signals in a self-supervised manner with minimal domain knowledge. The model trains and inferences on beat-aligned music signals and predicts an 8-layer hierarchical metrical tree from beat, measure to the section level. The training procedural does not require any hierarchical metrical labeling except for beats, purely relying on the nature of metrical regularity and inter-voice consistency as inductive biases. We show in experiments that the method achieves comparable performance with supervised baselines on multiple metrical structure analysis tasks on both symbolic music and audio signals. All demos, source code and pre-trained models are publicly available on GitHub.


INTRODUCTION
Metrical structure analysis is a useful task for music information retrieval (MIR) [1,2], computer musicology [3] and music generation [4]. With the development of deep learning, we have witnessed improved performance on beat and downbeat tracking [5,6,7], part segmentation [8] and hierarchical metrical structure analysis [6,7,9]. However, most of these methods are based on supervised training, which requires manual ground-truth labeling of a large music collection. This poses two potential risks: (1) the labeling procedure is time-consuming and requires domain experts; (2) the ambiguous nature of music and the subjectivity of human annotators commonly cause biases in MIR datasets [10].
Self-supervised learning has been a promising way to model signals without labeled datasets [11,12,13]. Self-supervised learning is recently introduced to MIR tasks including pitch estimation [14], music classification [15], cover song identification [16] and tempo estimation [17]. These works inspire us to utilize selfsupervised learning in hierarchical metrical structure analysis, where the scarcity of labeled datasets limits the model's performance.
Hierarchical metrical structure modeling aims to describe temporal rhythmic patterns of music at different time scales. Different levels of metrical structures are interrelated (e.g., multiple beats form a measure), building up the metrical hierarchy. The concept can be further extended to larger time scales, forming concepts like hypermeasures and hypermeters [18,19], which are useful in analyzing other high-level features like melodic phrase structure, harmonic structure and form [20]. The notation of hierarchical structure is formally described in the Generative Theory of Tonal Music (GTTM) [19] by using different numbers of dots to represent different metrical boundary levels (as in fig. 1  In this paper, we describe a self-supervised approach to hierarchical metrical structure modeling (see fig. 1). A Temporal Convolutional Network (TCN) accepts a beat-aligned spectrogram or piano roll as input and predicts a metrical boundary level for each time step. For simplicity, we ignore non-binary metrical structures (e.g., 3/4 or 6/8 meters) and focus on binary structures in the paper (temporary binary irregularity is still allowed). We encourage the model to predict a sequence that follows binary regularity (i.e., a level-l metrical unit contains exactly 2 level-(l − 1) metrical units). We implement this inductive bias using an unsupervised Conditional Random Fields (CRF) loss. We show that by minimizing this loss term, the model learns a semantically meaningful hierarchical metrical structure up to some global offset. Similar to other self-supervised approaches [14,17], the global offset is shared across all songs and can be calibrated with a few annotated samples (1 song), which is the only supervised data required for the model. For multi-track data like MIDI files, we also propose a consistency-based loss that further boosts the model's performance.
The paper's main contributions are: • We provide a novel self-supervised approach for hierarchical metrical structure analysis. The model performance is comparable with supervised methods on several tasks. • We propose an unsupervised training target based on metrical regularity, and show its effectiveness on metrical structure modeling on multiple scales. • As far as we know, our model is the first hypermetrical structure analyzer for audio signals, opening up new possibilities for downstream MIR applications.
(a) (b) Fig. 2. The intuition of self-supervision for metrical analysis. For simplicity, assuming the level-(l − 1) metrical boundaries are known and we want to build the level-l metrical structure. Under the assumption of binary regularity, we strongly prefer one of the choices in (a) and (b). In either case, the labeling for consecutive beats is different, forming negative sample pairs in the sense of contrastive methods.

Design Intuition
Before discussing the details of the model, we first describe the intuition behind the self-supervision setting. One common way to train self-supervised models is to introduce pseudo-labels. In contrastive learning, several views of a same data sample form positive pairs, and views of different samples form negative pairs [13,21]. An equivalent-based data augmentation also creates pseudo-labels with relative distance constraints, preventing the model from trivial solutions like predicting everything as a single label [14,17].
In self-supervised metrical analysis, the main source of supervision comes from the preference of metrical regularity. For binary metrical regularity (with a periodicity of 2), a level-l metrical always contains 2 level-(l − 1) metrical units, forming an alternating pattern as shown in fig. 2. To relate with contrastive methods, we can regard two consecutive level-(l − 1) measure boundaries as negative pairs for level-l boundary prediction and two level-(l−1) measure boundaries with distance 2 as positive pairs. Instead of using a constrastive loss which regards metrical regularity as hard constraints, we design an unsupervised CRF loss to incorporate it as a soft constraint with controllable degree of penalty to metrical irregularity. Also, we design our CRF loss to enable joint estimation of multiple metrical levels instead of just a single level (see section 2.3).

Model Architecture
The backbone of our model is 6 stacked temporal convolutional layers with the same configuration as [9]. For symbolic input (MIDI file) with T tracks, we regard the 16th-note-quantized (tatumquantized) piano roll of each track t as input mt. The TCN then predicts a metrical level prediction h (t) i and a confidence score α (t) i for each time step i ∈ {1, ..., N }. Similar to [9], the final prediction is acquired by a weighted pooling over all tracks: where p il denotes the predicted probability that time step i has a metrical boundary of level l ∈ {0, ..., L}. For convenience, we use p i(≥l) = L l =l p il to denote the probability that the time step i has a metrical boundary of at least level l.
For audio input, some changes are made compared to symbolic input. We use a Short-Time Fourier Transform (STFT) spectrogram instead of the piano roll as input, and the spectrogram is re-quantized to a 128th-note level using time stretching. We prepend 3 extra convolutional-pooling layers before the TCN. Each pooling layer has a kernel size of 2, bringing down the resolution to the 16th-note level to align with the time step of symbolic input. The extra convolutional layers are identical to the ones in TCN but with a constant dilation 1. Also, since we cannot acquire cleanly separated tracks of audio, the TCN directly predicts hi without the weighted pooling operation, and the probability is calculated as pi = Softmax(hi).

Unsupervised CRF Loss
The aim of the CRF loss is to encourage the TCN's output to be close to a sequence that follows binary regularity on all levels (see some examples in fig. 3). The design of CRF's state space and transition potential is similar to [9], and we recommend the reader refer to it for detailed explanation and some concrete examples. We here adopt a simplified version to reduce computational cost. For a hierarchical metical structure with L layers, the joint state space is defined as represents the the number of complete level-(l − 1) metrical units in this level-l unit up to the current time step. If 2 (or more) complete level-(l − 1) metrical units are spanned, a new level-l unit must start, otherwise a binary irregularity penalty is posed and we keep z The transition potential is defined the same way as in [9]: where li = arg max l=0...L [z is the single-layer transition potential matrix for level-l metrical boundaries: where w ins > 0 are hyperparameters that controls the penalty of a level-l metrical unit deletion and insertion respectively.
ins to +∞ will result in a hard binary regularity constraint (i.e., the states on level-l must alternate between 0 and 1). Different from [9], the insertion transition is now a self-transition, thus can be applied several times in a larger metrical unit. The emission potential function is defined as ψ(zi, pi) = p il i .
The unsupervised CRF loss is defined as the negative log likelihood of the observation sequence p1...N . This can be calculated by summing up all possible latent state sequences z over the unnormalized CRF likelihood: which can be computed efficiently by the well-known dynamicprogramming-based forward algorithm.

Consistency Loss
For metrical analysis of multi-track data, we also include the consistency loss L2. For two time-aligned track-wise prediction p (1) , p (2) , we want them to indicate the same metrical structures. We achieve this by requiring the same CRF hidden states to emit both observations by defining the joint emission potential ψ (zi, p il i . The consistency loss is defined by:

Post-Training Calibration
Since the model solely relies on self-supervision, the model's output might mismatch the true metrical boundaries by a global offset. For a well-trained model, imagine that if all model predictions shift left or right some constant time steps, both loss terms remain low. The global offset always lies in the receptive field of TCN, usually within ±32 time steps. Therefore, we need to look into a few annotated samples to calibrate the global offset (in practice, 1 song is enough). The calibration only needs to be done once per trained model and no more annotated data is required before or after the procedure.

Datasets
We use the Lakh MIDI dataset [22] to train the model for symbolic music analysis. We first acquire a subset of 3,739 multi-track MIDI files according to the criteria described in [9] to exclude some lowquality files. We use the pretty midi package [23] to acquire the beats of each song and quantize each song into 16-note units. No hierarchical metrical information (e.g., downbeat annotations) is used in training. For evaluation, we use the RWC-POP MIDI dataset. We only use the test split in [9] to make a fair comparison with the supervised model. A single song (RWC-POP-001, not in the test split) is used to calibrate the model.  To train the audio model, we collect a training set of 7,500 songs from an online music collection from a rhythm game osu! 2 . In this collection, every song comes with highly precise beat annotation by crowd annotators as beat labels are essential for rhythm game mapping. Downbeat annotations are also available, but they are not used in training. The collection has a main genre of J-pop but also contains a variety of other genres including western pop and rock. All audio files and annotations are publicly accessible, making it easier for reproduction and further research. The reader may refer to the paper's GitHub repository for more information about the dataset. For evaluation, we use the audio recordings of the RWC-POP dataset. We use the same data split as the symbolic model evaluation and the recording version of RWC-POP-001 to calibrate the model.

Model Hyperparameters
We let the model predict a hierarchy of L = 8 metrical layers from the 16th-note level. Since each level increases the metrical unit length by 2, the 2nd layer corresponds to beats and the 4th layer corresponds to measures for 4/4 songs. For both symbolic and audio music structure analysis, we set w The format of the piano roll input aligns with [9]. For audio input, we resample the audio to 22050Hz. Then, we use a 1024-point STFT with 75% overlapping hann windows before time-stretching the spectrogram to a 128th-note level. Each model is trained for 10 epochs using Adam optimizer [24] with a fixed 10 −4 learning rate.

Reference
Self-supervised Reference Self-supervised Reference Self-supervised Fig. 6. A case study of the symbolic hypermetrical structure modeling. The 10-measure piece (from RWC-POP-005) contains binary irregularity on a 4-measure scale (4+2+4). The model successfully identifies a temporary hypermeter change but deletes the wrong hypermeasure (2+4+4), resulting in wrong boundaries (marked in red).

Main Results: Hypermetrical Structure Prediction
We first evaluate our model on metrical structure prediction beyond measures (l = 5...8). For symbolic music analysis, we include 2 baseline models, the supervised model from [9] trained on 50 annotated songs (726 MIDI tracks) and a rule-based model based on homogeneity metrics on different scales [25]. To make a fair comparison, the prediction of the self-supervised model will go through the same CRF decoder with the same hyperparameters as the supervised model. In this experiment, the downbeat labels are assumed to be known at test time; only predictions on ground-truth downbeats are used for decoding hypermetrical structures for all models.
For symbolic hypermetrical structure evaluation, we also include an ablation test that removes the track-wise consistency loss. The results are shown in fig. 4. The self-supervised model shows impressive performance on 2-measure and 4-measure hypermetrical structures, and the consistency loss further boosts the performance. When only the melody track is used (melody only), the model even outperforms the supervised baseline, showing that a large unlabeled dataset is helpful for the model to understand metrical structures better from limited information.
For audio hypermetrical structure evaluation, the results are shown in fig. 5. Since we are unable to train a supervised baseline without overfitting an extremely limited training set (50 songs), the only viable way is to use self-supervised learning to gain reasonable performance for the task. Compared to MIDI files, audio hypermetrical structure analysis is more challenging since we cannot impose track-wise consistency loss, and the input feature is often noisier.
Notice that the prediction of the highest layer (the 16-measure level, or l = 8) is poor for self-supervised models. The main reason is that a stable binary rhythmic structure on a 16-measure level is much less common compared to lower levels, invalidating our model assumption. Therefore, the model fails to learn meaningful representation with self-supervision.
We also observe that the proposed method does not perform well near hypermetrical changes, with an example shown in fig. 6. The model is capable of detecting most hypermetrical changes, but the labeling of the transition part is sometimes not musically meaningful. The main reason is that the preferences of irregular hypermeasure decomposition is not encoded into our CRF loss, and the model chooses the wrong decomposition under self-supervision.  Table 2. Evaluated downbeat scores on RWC-POP audio recordings.

Downbeat Detection
As another demonstration of the proposed method's utility, we evaluate our models on downbeat detection, the task to classify if a beat is a downbeat or not. For 4/4 songs, the downbeats correspond to the level-4 metrical structure (l = 4) in our proposed method. We assume correct beats are known at inference time and perform a simple peak picking over p i(≥4) to classify downbeats. We train supervised baselines using known downbeat labels from the Lakh (MIDI) or osu (audio) dataset with the same TCN architecture. We also include STOA audio downbeat tracking models (we use synthesized audio for MIDI evaluation) for reference, where predicted downbeats are attributed to the nearest ground-truth beats. The results are shown in table 1 for MIDI files and table 2 for audio. The results show that the self-supervised methods' performances are comparable with supervised counterparts, showing that self-supervision is a strong inductive bias for low-level metrical regularity. The self-supervised methods even surpass the supervised method on MIDI datasets. The main reason is that the inaccuracy of downbeat labels 5 harms the performance of supervised methods.

CONCLUSION
In this paper, we propose a novel approach to model hierarchical metrical structures for symbolic music and audio signals. We show that the unsupervised CRF loss helps the model learn a musically meaningful metrical hierarchy, and a consistency loss further boosts the performance for multi-track data. The work shows the potential to improve other rhythm-related MIR tasks with inductive biases based on metrical regularity.
One limitation is that our model requires beat labels for quantization. One future work is to apply self-supervised learning to audio (or performance MIDI) beat tracking, enabling end-to-end metrical structure analysis. Another direction is to explore semi-supervised settings to further boost the model's performance.