{'fraction_non_alphanumeric': 0.03851513999746611, 'fraction_numerical': 0.008910849275729549, 'mean_word_length': 5.0500766479305055, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 4, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Score Transformer: Generating Musical Score from Note-level Representation

In this paper, we explore the tokenized representation of musical scores using the Transformer model to automatically generate musical scores. Thus far, sequence models have yielded fruitful results with note-level (MIDI-equivalent) symbolic representations of music. Although the note-level representations can comprise sufficient information to reproduce music aurally, they cannot contain adequate information to represent music visually in terms of notation. Musical scores contain various musical symbols (e.g., clef, key signature, and notes) and attributes (e.g., stem direction, beam, and tie) that enable us to visually comprehend musical content. However, automated estimation of these elements has yet to be comprehensively addressed. In this paper, we first design score token representation corresponding to the various musical elements. We then train the Transformer model to transcribe note-level representation into appropriate music notation. Evaluations of popular piano scores show that the proposed method significantly outperforms existing methods on all 12 musical aspects that were investigated. We also explore an effective notation-level token representation to work with the model and determine that our proposed representation produces the steadiest results.

(a) (b) Figure 1: Overview of (a) MIDI-to-score process and (b) musical score generation task. We focus on (b), which generates human-readable music notation (lower) out of quantized note-level representation (upper).
instruments. Various musical symbols (e.g., time and key signature) and attributes (e.g., voice and grouping of notes) in musical scores enable us to comprehend music easily. In contrast, note-level representations (e.g., piano-rolls and MIDI files) cannot express such high-level visual musical information: they only represent music aurally. Considering the differences, it is of great importance to develop automated systems able to generate representations equivalent to musical scores. Deep learning techniques have yielded impressive results in music generation and music transcription. However, their application to the generation of comprehensive musical scores or even effective representations thereof remains unexplored. In music generation, sequence models with symbolic music representation have exhibited considerable success [12,19]. In particular, the combination of the Transformer model [25] and a token representation of music has been revealed to generate coherent and structural music [11,14]. However, the music representations used in prior studies were note-level (MIDI-equivalent); hence, their expressions were limited compared to those of musical scores. With regard to automatic music transcription (AMT), neural network models have achieved remarkable success in the audio-to-MIDI process (e.g., multi-pitch estimation and onset/offset detection) [1]. In contrast, the subsequent MIDI-to-score process in which note-level representation is transcribed into music notation [7], has not been comprehensively addressed [4]. More specifically, the MIDI-to-score process comprises two subtasks [7,17,22] as illustrated in Figure 1(a). There has been a lack of focus on the latter musical score generation (or score typesetting) subtask, in which rhythm-quantized (beat-estimated and quantized) MIDI is transcribed into a comprehensive musical score. Although notation-level symbolic representations have been partially employed in a few studies [2,15,20,21], using those of comprehensive musical scores comprising various musical symbols and attributes with sequence models has yet to be explored. Therefore, the issue of whether sequence models can generate comprehensive music notations and of how to effectively represent musical scores to feed such models remain unresolved.
In this study, we address the generation of comprehensive musical scores using the Transformer model [25], which complements music generation and music transcription research. This task is equivalent to the musical score generation subtask in music transcription and also aids music generation by transcribing its notelevel output into musical scores (Figure 1(a)). We first design token representations corresponding to musical scores (Section 3). Then, we train and evaluate Score Transformer, a Transformer model that translates quantized note-level representations into appropriate music notation (Sections 4-6). Figure 1(b) illustrates a variety of subtasks required to notate music, such as staff assignments of notes, key estimation, and voice segregation of the note stream. We address this challenging and important task while addressing the research issues presented above.
Our contributions are summarized as follows: • We propose a novel method to generate comprehensive musical scores, which jointly estimate various musical symbols and attributes required in musical scores. • We propose an effective token representation of music notation that works well with the Transformer model. To promote further research, tools designed to convert between the proposed representation and MusicXML [8], a standard open score exchanging format, are provided. 1 • We devise a simple training scheme for bridging two levels of symbolic musical representation: note-level and notation-level. This scheme requires only musical score data.

RELATED WORK 2.1 Musical Score Generation
Musical information should be encoded in a form that an employed model can interpret. For this purpose, the tokenization of musical information is a common choice for sequence models. For note-level (MIDI) tokenization, two representation styles have been proposed: MIDI-like [19] and REMI [12]. The former style represents notes with note-on, note-off, and delta-time, which are similar to MIDI events, whereas the latter style includes bar, position, and note duration tokens to improve the former representation. In REMI, a set of note-on, note-off events of the same note can be represented as a single token, which resembles note value in musical scores. Recent research has also revealed that the sequence model can generate note-level representations in an efficient form where related tokens are grouped [10]. In this paper, we design notation-level token representations that correspond to a musical score. We also expand REMI to various meters for our note-level representation.

Note-level Tokenization
In a few prior studies on music transcription, the generation of musical scores has been attempted in a limited manner. Two types of studies have been conducted: MIDI-to-score and audio-to-score (in an end-to-end manner). The former studies [4,7] addressed the MIDI-to-score task with a probabilistic model or grammar model. For example, a framework was proposed in [4] that transcribes performance MIDI into a musical score using LilyPond [18] by splitting the task and solving them using a probabilistic analysis tool [24]. The latter studies [2,15,20,21] attempted to output music notations directly from waveforms. For example, a recurrent neural network (RNN) or its variant has been utilized to generate LilyPond [2,15], Humdrum [21], and designed characters [20]. These studies have shown the possibility of automated generation of musical scores with RNNs; however, they considered only a limited set of musical symbols. In this paper, we aim to generate comprehensive musical scores in the form of token representation with the Transformer model and explore their suitable representation. We focus on piano scores, the typical musical scores with multiple staves in a part, and multiple voices in a staff.

SCORE TOKENIZATION
We address score tokenization in two ways: 1) by designing new token representations, and 2) by using existing score formats.

Proposed Representation
Inspired by the success of note-level tokenization, we design a token representation that symbolizes score elements. Our design principles are as follows: • One token per symbol or attribute: A single token corresponds to a score symbol (e.g., barline, clef, key signature, and time signature) or a note attribute (e.g., note value, stem direction, and tie). The only exception is a voice symbol, which consists of a pair of tokens (Section 3.1.3). Because following this principle results in long sequences, we also consider a shorter form (Section 3.1.5). • Compatible with music21 attributes: The attributes of the symbols are designed to be compatible with those of music21 [5] objects, thus enabling utilization of the music21 toolkit to create MusicXML scores from the token representation. • Concatenated sequences of staves: We concatenate time-ordered token sequences of staves (i.e., right-and left-hand parts of piano scores) to form a single sequence [6], which allows the model to refer to generated tokens while making inferences. This helps the model maintain the consistency of score attributes and staff assignments between staves. • Tokenize essential musical symbols only: We represent the essential musical symbols in musical scores listed in Table 1. Additional expression symbols (e.g., articulations, dynamics, and ornaments) and repeat symbols are not included in this work.

Staff and Barline.
We symbolize the two staves in piano scores as R and L. Every score token sequence starts with R and is followed by symbols in the right-hand staff; on the other hand, after L emerges, symbols in the left-hand staff follow (Figure 2(b)). Whereas, barlines are expressed as bar tokens in the same manner as in the MIDI tokenization (Figures 2(b) and (c)).

Clef and Signatures.
Clefs, key signatures, and time signatures are tokenized as clef, key, and time tokens, respectively. Attributes follow each. For example, clef_treble indicates G clef, and key_sharp_2 indicates a key with two sharps, D major or B minor. Besides the beginning of a score, these symbols also appear when the attribute changes (e.g., key change).

Voice.
A voice, or a stream of notes, is expressed as a pair of tag-like tokens: <voice> and </voice>. The <voice> token declares the start of a new stream, which is followed by the notes and other sym-bols assigned to the voice, whereas </voice> indicates the end of the stream. For example, if a measure has two concurrent voices sharing a staff, two sets of voice tokens appear in succession ( Figure  2(b)). We represent voices as tag-like tokens to visually clarify the concurrent voices.

Note and
Rest. The note representation consists of five types of tokens that appear in the following order. The note token indicates the pitch (e.g., note_C4); the len token indicates note duration measured in quarter notes (e.g., len_1/2 = 8th note); the stem token indicates stem direction (e.g., stem_up); the beam token indicates the properties of each beam (e.g., beam_start_continue, which indicates the initiation of the first beam and the continuation of the second); and the tie token indicates the property of tie (e.g., tie_stop). Of these types, stem, beam, and tie tokens appear only when the preceding note has an attribute. The note tokens can appear successively before the len token, indicating that the notes constitute a chord (cf. second line in Figure 2(b)). The rest representation is simple and always consists of two tokens: rest and len. The rest token has no attributes; the subsequent len tokens have the same attribute as that of the note duration token (e.g., len_1/4).

Concatenated Tokens.
We consider shortening the score token representation by concatenating tokens because the score token sequences can be relatively lengthy and can thus deteriorate the computational efficiency of training and inference. We concatenate some of the note-attribute tokens because they share large parts of the sequences. By concatenating three note-attribute (duration, stem direction, and beaming) tokens, for example, len_1/2, stem_up, and beam_start are represented as attr_1/2_up_start in a concatenated form, where three note-attributes are expressed as a single token.

Alternative Approach with Existing Formats
Various formats have been developed for symbolic score representations, such as tree-structured (e.g. MusicXML [8] and MEI [9]) and text-like (e.g. ABC notation, Humdrum (**kern) [13], and LilyPond [18]). Although not designed to feed sequence models, text-like formats can be handled as token sequences when properly segmented. We create token sequences from text-like formats by eliminating header and footer information, and segmenting formatted contents while inserting necessary separators, connectors and staff-voice relation tokens to help restore the formats. For each format, we tokenize note pitch (or rest), note value, and other attributes (e.g., stem direction and beaming group) separately. Meanwhile, for Humdrum, we tokenize its 2D representation in two ways: on a row-major basis and a spine-major basis, denoted as row and spine, respectively.

FROM MIDI TO SCORE 4.1 Token Representation
Input (MIDI). For note-level tokenization, we adopt REMI [12], expanding it by adding beat tokens to deal with various meters. With this modification, the position (pos) token indicates the note position based on the timing difference from the preceding bar or beat token. We only adopt bar, beat, position (pos), note pitch (note), and note duration (len) tokens for our purpose. A beat is evenly divided into 24 timesteps to represent 32nd triplet notes and rests in our setup. An example of note-level tokenization is shown in Figure 2(c).
Output (Score). We adopt the notation-level tokenization methods described in Section 3.

Model
In the research area of music generation, many studies have reported that sequence models, such as the Transformer [25], fit the task quite well [10][11][12]. Although many Transformer variants have been proposed [23], we adopt the vanilla Transformer model to utilize its original attention mechanism. Figure 3 illustrates our training scheme. The model is trained in a supervised manner to restore the original score data from downconverted MIDI data, thus minimizing the loss between estimated and original score tokens. This scheme enables training of the model using only musical score data.

EXPERIMENTAL SETUP 5.1 Dataset
We employed popular and classical piano scores for the experiments. For the popular scores, we collected sheet music of popular songs arranged for solo piano from a public website. 2 For the classical scores, we collected piano sheets from KernScores. 3 The classical data consist of piano sonatas composed by Beethoven, Clementi, Haydn, and Scarlatti, and mazurkas and preludes by Chopin. Table 2 lists the specifications for each dataset. The scores were split systemby-system (using four measures each for the classical scores) and tokenized into token sequences. We split each dataset song-wise in the ratio 8:1:1 for training, validation, and test data, respectively. Considering the dataset size, we used the popular piano dataset mainly for the experiment and the classical dataset for the crossgenre evaluation.

Training
We trained the Transformer model using the following configurations: embedding sizes model = 256 and ff = 512; number of

Baselines and Alternative Tokenization
We adopted the transcription framework proposed in [4] as a baseline because it considers musical aspects the most comprehensively to our knowledge among the prior studies. We denote this framework as "CTD. " We used the authors' implementation with a minor bug fix on a key issue. We also employed the latest version of the music notation software, Finale 26 and MuseScore 3, as baselines. MuseScore 3 has an algorithmic MIDI-to-score functionality, 4 and Finale 26 has the same functionality. We loaded down-converted single-track MIDI data (with metrical structure) into each software and saved them as MusicXML data.
We also compared the proposed tokenization method with the existing text-like score formats presented in Section 3.2. For each format, we 1) converted MusicXML scores into the desired format using a converter tool, 5 2) tokenized the output by separating the symbols (see Section 3.2 for details), 3) trained the Transformer model using the token, 4) restored the text in the format by concatenating inferred tokens, and 5) finally reconverted it to MusicXML using another converter tool. 6

Evaluation Metric
Although most metrics for transcribed musical score quality measure limited musical aspects (e.g., MV2H [16]), a metric was proposed in [3] that measures music notation quality based on the number of errors (vs. ground truth score) on 12 musical aspects of a score, utilizing the music21 toolkit [5]. That study also reported normalized numbers of errors between the transcribed and original scores correlated with subjective evaluation scores. We employed the metric with three expansions: 1) adding three musical aspects (voice, beam, and tie) to evaluate our model thoroughly and excluding two aspects that were also measured using other aspects (barline vs. time signature, and note grouping vs. voice), 2) separating insertion and deletion errors to discriminate the types of error while integrating note and rest metrics, 3) counting affected notes and rests for all aspects to facilitate aspect-wise performance comparison (for example, if a key signature is wrong, we count notes and rests under its effect). We calculated note-wise error rates based on the metric output, which is the number of errors between the original and generated scores with respect to the 12 musical aspects. Table 3: Overall error rates in % (measured based on the difference between the original and generated scores) for the popular piano dataset. Values in boldface show the lowest error rates. "w/ABC" uses ABC notation instead of our score token representation and the same for subsequent methods (see Section 3.2). For Stem Direction, models that do not specify directions are excluded from evaluation because the metric cannot measure them properly.  Table 3 shows the overall results for the popular piano dataset measured in error rates for the 12 musical aspects. According to these results, the Score Transformer (ST) performed significantly better than the baseline methods (i.e., CTD [4], Finale, and Mus-eScore) on all 12 musical aspects. The results suggest that ST not only successfully learned how to transcribe note-level representations into a musical notation, but also has a much higher capability to notate music than those of prior arts. The results also demonstrate that ST can jointly estimate various symbols and attributes in musical scores while generating non-time-ordered sequences. Figure 4 shows an example of the generated musical scores. Owing to copyright restriction, we used a public-domain song. Although minor differences can be observed, the output of ST was closest to the original score.

Evaluation of Alternative Tokenization
By comparing the error rates of Score Transformer (ST) and its variants with those of existing formats (Section 3.2) in Table 3 (see lower half), it may be observed that the methods using existing formats exhibit unstable (variably low to high) error rates among the investigated aspects. For example, Humdrum (row) achieved the lowest error rates on five aspects, but it had some of the highest error rates on the other two aspects (i.e., Key Signature and Stem Direction). The same is true for Humdrum (spine) and LilyPond. In contrast, the proposed tokenization method (ST) demonstrated stable performance over these aspects. The result implies that token representations based on existing formats are not well-adapted for leveraging the attention mechanism of the Transformer. We also determined that some methods are prone to frequent format errors. Figure 5 shows the rates of format errors and typical errors. Notably, Humdrum (spine) exhibits a high error rate, suggesting that the generation of a 2D Humdrum format is especially difficult for the model when presented on a spine-major ( Figure  5(c)). For LilyPond, length disagreements between the staves were  frequently observed (Figure 5(b)). A high error rate in the format leads to practical issues. Based on the measured performance and the format error rates, the proposed tokenization method produced the steadiest results.  Table 4 shows the statistics for token length (as average length ± standard deviation) and performance (in error rates) for the regular and concatenated forms (Section 3.1.5) of our token representation. The average token length indicates the efficiency of the concatenated form in a sequence length (approximately 2/3 of the regular form). The small differences in error rates between the two forms suggest that our score token representation can also be efficient via simple concatenation without resulting in performance degradation.  Table 5 shows the generalization capability of our method. First, the ST model trained on the popular piano dataset (denoted "pops") outperformed baselines on Average on the classical piano dataset, showing that it has a certain level of generalization capability. Second, compared with the model trained without the beat token in note-level representation (denoted "pops, w/REMI"), the "pops" model performed better, suggesting that the beat tokens in our representation facilitate generalization between genres where the model can encounter the songs with untrained meters. Table 5 also shows that fine-tuning improves the performance of ST on another genre. From a cross-genre viewpoint, the "pops" model performed worse than it did on the popular piano dataset, implying that a performance gap exists between trained and untrained genres. However, following fine-tuning on the classical piano dataset, although ST ("fine-tuned") did not outperform the baselines entirely, the Average error rates of ST decreased by nearly half. The result suggests that ST can perform well also on another genre when fine-tuned on a different genre, even if the dataset is relatively small.

Robustness on Unquantized Input
Finally, Table 6 shows the robustness of our method. We experimented with the popular piano dataset with random noise following normal distributions ( = 0, = 0.08 for note-on timings; = 0.8, = 0.24 for note durations) added to all the note-on timings and durations of note-level representation. The sampled values were multiplied by note durations before adding. As a result, the input notes were temporally deviated from the metrical structures. The results for ST were not much worse than those shown in Tables 3 and 4, suggesting that ST can generate scores robustly, even for noisy input.

CONCLUSION
In this paper, we designed score tokens to represent musical scores and trained the Transformer model using paired sequences of notelevel and notation-level data. The evaluation results show that our model significantly outperformed the baseline methods in terms of error rates for a variety of musical aspects. The results clearly show that the Transformer model can generate musical scores with a tokenized representation of scores. Additionally, we demonstrated that our token representation is among the most effective via a comparison with various tokenization methods. Our representation is extendable for various musical symbols and instruments by defining new tokens. We believe that our method opens new possibilities for future research into a variety of musical and multimedia tasks that utilize musical scores.

ACKNOWLEDGMENTS
We would like to thank Akira Maezawa and Takuya Fujishima for useful advice and comments on this work.