{'fraction_non_alphanumeric': 0.03890049050556068, 'fraction_numerical': 0.01019863046962265, 'mean_word_length': 4.908751793400287, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Visual onoma-to-wave: environmental sound synthesis from visual onomatopoeias and sound-source images

We propose a method for synthesizing environmental sounds from visually represented onomatopoeias and sound sources. An onomatopoeia is a word that imitates a sound structure, i.e., the text representation of sound. From this perspective, onoma-to-wave has been proposed to synthesize environmental sounds from the desired onomatopoeia texts. Onomatopoeias have another representation: visual-text representations of sounds in comics, advertisements, and virtual reality. A visual onomatopoeia (visual text of onomatopoeia) contains rich information that is not present in the text, such as a long-short duration of the image, so the use of this representation is expected to synthesize diverse sounds. Therefore, we propose visual onoma-to-wave for environmental sound synthesis from visual onomatopoeia. The method can transfer visual concepts of the visual text and sound-source image to the synthesized sound. We also propose a data augmentation method focusing on the repetition of onomatopoeias to enhance the performance of our method. An experimental evaluation shows that the methods can synthesize diverse environmental sounds from visual text and sound-source images.


INTRODUCTION
Environmental sound synthesis aims to synthesize natural and diverse environmental sounds for sound-content production such as games [1]. Among several deep learning methods for environmental sound synthesis, synthesis from sound event labels (i.e., discrete symbols representing sound events) is the basic approach [2], [3]. This method synthesizes a sound that appropriates the given label but cannot control the detailed structure of the sound, e.g., duration and temporal changes in tone. Another method is synthesis from a text [4]- [6]. For synthesizing sounds, onoma-to-wave [6] (Figure 1 (a)) uses an onomatopoeia, a word imitating the sound (i.e., the text representation of sounds). This method can control the detailed structure of the synthesized sound by the input onomatopoeia text. Also, the overall impression of the sound can be controlled by conditioning the auxiliary sound event label.
Onomatopoeias have another representation: visual-text representation of the sound. For example, in comics and mangas, visual onomatopoeias are drawn with arbitrary shapes, sizes, and fonts for delivering sounds to the reader [7]. A visual onomatopoeia enables dramatic expression and draws the reader in a way that descriptions of non-textual images (e.g., sounding objects or people) alone cannot. Visual concepts, e.g., size and text stretch, of a visual onomatopoeia evokes in the reader sound concepts, e.g., loudness and duration [8]. Another example is in virtual reality (VR). Presenting visual onomatopoeias along with environmental sounds in immersive VR can enhance the immersive qualities of VR [9]. These examples show the possibility of applying environmental sound synthesis from arbitrary visual onomatopoeias to audible comics [10] and immersive VR and motivate us to develop a synthesis method. Also, since visual onomatopoeias frequently pair with the sound source image to express its sound, we expect that the auxiliary use of the sound source image will guide the audio synthesis.
In this paper, we propose visual onoma-to-wave, environmental sound synthesis from visual onomatopoeias. Our method synthesizes sounds from onomatopoeia visual-text rather than onomatopoeia text. This enables us to control the detailed structure of the synthesized sound by transferring the visual concept of the visual onomatopoeia. In other words, we introduce image stretching of visual onomatopoeias to control the duration of the environmental sound. Furthermore, we introduce to condition the sound synthesis model by sound source images for controlling the overall impression of the sound. This image-based auxiliary control makes it possible to synthesize more diverse sounds than label-based control with the conventional onoma-to-wave. We also propose a data augmentation method to enhance the performance of environmental sound synthesis. We conducted experimental evaluations to investigate 1) whether our methods appropriately transfer visual concepts to the sound and 2) whether they synthesizes diverse and natural sounds. The source code and demo are open-sourced on our project pages 1 . The contributions of this study are as follows: • We propose visual onoma-to-wave, a new task and method of synthesizing environmental sounds from visual representations of onomatopoeias and sound sources. • The proposed method and data augmentation allow synthesizing a more diverse sound with the same or better naturalness than the conventional method.

Sound synthesis from discrete symbols
The basic methods for environmental sound synthesis involve synthesizing from discrete labels representing sound events, e.g., "cup" or "drum." [2], [3] Such methods use sound event labels to represent the overall impression of the synthesized sounds. The label is suitable in representing the overall impression of the synthesized sound but has no information to represent the detailed structure of the sound. In contrast, text is a good means to describe the detailed structure. There are two types of methods for textbased control: describing all attributes of the sound in a sentence and separately describing the overall impression and detailed structure. Yang's method [4] is of the former, and onoma-to-wave [6], is of the latter. Since onomatopoeias efficiently imitate temporal changes of the sound, they are considered straightforward to control the detailed structure of the sound [11], [12]. Onoma-to-wave also uses labels as auxiliary information in order to control the overall impression, following the label-based synthesis.

Sound synthesis from images
Methods for speech and sound synthesis from images have been proposed. Image-to-audio converts visual concepts in given images to synthesized sound [13], [14]. This image-based control of the overall expression of the sound is better in sound diversity than the aforementioned label-based control. In text-to-speech, visual textto-speech (vTTS) replaces text of the standard text-to-speech with visual text (i.e., text as an image) and converts visual concepts in given visual text to synthesized speech [15]. This image-based control of the detailed structure of the speech enables a variety of speech expressions that are not possible with standard text-to-speech. In light of these methods, we introduce image representation into onoma-to-wave to enable the synthesis of diverse environmental sounds. Instead of onomatopoeia text, onomatopoeia visual-text (visual onomatopoeia) is used to synthesize. Not only textual but also visual concepts determine the detailed structure of the sound. Also, instead of a sound event label, a sound event image is used as an auxiliary to enhance the diversity in overall expression. Considering the application to comics and VR described in Section 1, the combination of visual onomatopoeia and auxiliary images is appropriate. Usually, a visual onomatopoeia corresponds to the source image in comics and VR. For example, in a baseball comic, the image of a baseball bat and the visual onomatopoeia of hitting a baseball are adjacent to each other. The bat image and visual onomatopoeia give the impression of audio to the reader.

Visual onoma-to-wave
Basic architecture. Our method uses the Fastspeech2-inspired architecture [15], [16] to learn models from visual onomatopoeias and sound event images as shown in Figure 2. First, the visual onomatopoeia is sliced into individual characters. As with vTTS [15], we use visual onomatopoeias artificially generated from onomatopoeia texts. Assuming a monospaced font of the visual text, we slice the visual onomatopoeia into n images of h-by-w size, where n, h, w are the number of characters and the height and width of each visual character, respectively. These are a bit far from the realistic settings but sufficient to investigate the ideal performance of sound synthesis from visual onomatopoeias. The sliced images are fed to the visual feature extractor, followed by the FastSpeech2-inspired encoder, variance adaptor 2 , and decoder. The 2 Since environmental sounds have no pitch-related feature, our variance adaptor does not have a pitch predictor that is used in the original Fast-  sound waveform is synthesized from the generated mel-spectrogram, using a neural vocoder [17]. Conditioning by sound event image. A sound event image is used for controlling the overall expression of the sound. The pre-trained CLIP [18] image encoder encodes the input image to the sound event feature. The obtained feature is added to the output of the variance adaptor. The CLIP pre-training involves a text-image contrastive objective; the pre-trained encoder is expected to extract meaningful features from an input image 3 . For example, a feature extracted from an image of a metal baseball bat corresponds to the text "a metal baseball bat." This allows us to synthesize more diverse sounds than the sound event label, e.g., "a baseball bat."

Transferring visual effects in visual onomatopoeia to sound
We introduce duration-informed stretch of a visual onomatopoeia for transferring its visual concept to sound. As explained in Section 1, shrinking and expanding a visual onomatopoeia changes the duration of the imagined sound.
We first calculate the average sounding rate P [character/sec] in each sound event cluster (sound event category in our study). This value indicates the number of onomatopoeia characters used for describing 1-second sound. A width of a visual onomatopoeia corresponding to D-second sound is stretched at a rate of P · D. Figure 3 shows an example of two different onomatopoeias corresponding to one sound. The onomatopoeias are visualized with the same shape, regardless of their number of characters. This method is applied to the training data, and there is no change in training. During inference, sound duration can be controlled by shrinking or expanding the visual onomatopoeia.

Data augmentation
Frequent appearance of repetition is a unique characteristic of onomatopoeias [19], e.g., "ding-ding-ding-..." (bell ringing repeatedly) and "zzzzzz..." (continued snoring sound). We categorize repetition types into word-(e.g., "ding-ding...") and character-level (e.g., "zzzz..."). Our proposed data augmentation method synthesizes For word-level repetitions, we simply duplicate and concatenate a sound waveform corresponding to non-repetition onomatopoeias. For example, the "ding-ding" sound is made by concatenating the "ding" sound. For character-level repetition, we first find repeating (≥ 3 times) characters in the training data and temporally align characters and sound waveform. We chose the middle part of the continuation, which is assumed more stable than the other parts, and duplicate the waveform segment at the subsequent position as shown in Figure 4. This causes discontinuous sounding, but this may be alleviated by model training.

Experimental Setup
Dataset. We used pairs of sounds and onomatopoeia texts of 10 different sound events from RWCP-SSD [20] (environmental sound corpus) and RWCP-SSD-Onomatopoeia [21] (onomatopoeia corpus aligning to RWCP-SSD), and used 13, 170 and 664 audio samples for training and evaluation, respectively. We filtered out onomatopoeias with confidence scores [21] lower than 3 on a 5-point scale. Alignment between onomatopoeias and sound waveforms was obtained by training hidden Markov models using HTK [22]. We confirmed they were accurately aligned. Visual onomatopoeias. We used the ahaha-mojimoji 5 font and the Pillow module of python to artificially generate visual onomatopoeia from the onomatopoeia text. We used 24-pt font, so h = w = 24 when no image stretching was used. When using image stretching, the stretched image was adjusted to the same w across all data using zero-padding. Model and learning configuration. The learning rate was set to 0.001. In Sections 4.2.1-4.2.3, we used sound event labels as the auxiliary information rather than images. We used the same method as for speaker embedding [23] to obtain 256-dimensional sound event features from the sound event labels. We used images as the auxiliary information in Section 4.2.4. The sound event feature was added to the output of the variance adaptor via a linear layer. Other configurations, e.g., model configuration and training schedule, followed the open-sourced implementation of vTTS [15] 6 . Original onoma-to-wave paper [6] uses a Tacotron-inspired synthesis model [24], but in this paper, we used FastSpeech2-inspired model for both conventional onoma-to-wave and the proposed visual onoma-to-wave for fair comparison.

Evaluations
In Section 4.2.1, we investigate whether visual onoma-to-wave is comparable in the basic performance to the conventional onoma-towave. In Section 4.2.3-Section 4.2.4 we discuss our data augmentation method, image stretch-based control, and image-conditioned synthesis, respectively.
Following the onoma-to-wave paper [6], we conducted fivescale mean opinion score (MOS) evaluations on three criteria:  acceptance of synthetic sounds relative to onomatopoeias, expressiveness of the sounds relative to onomatopoeia, and naturalness of the sounds. Native Japanese speakers recruited through crowdsourcing participated in this evaluation. At least 20 listeners participated in each MOS test.

Text input vs. Visual-text input
To evaluate the basic quality of the visual-text input, we compared synthetic sound samples from text (i.e., onoma-to-wave) or visual text (i.e., visual onoma-to-wave). We used 664 samples for each synthesis method. Table 1 lists the results. Since there is no statistical significance between text input and visual-text input for all criteria, we can say that visual onoma-to-wave is comparable to the conventional onomato-wave in basic performance.

Impact of data augmentation
We then evaluated the impact of the proposed data augmentation. For word-level repetition, the number of repetitions was set from 1 through 2 e.g., /dong/ → /dongdong/ (1) or /dongdongdong/ (2), in training and 0 through 4 for evaluation. Short words (≤ 7 characters) were augmented. Namely, the evaluation data include sounds repeated more than in the training data. For character-level repetition, the number of repetitions was set from 1 through 5, e.g., /dong/ → /doong/ (1) or /doooooong/ (5), in training and 0 through 10 for evaluation. We chose 11 characters to be augmented, which covers 90% of repeated characters in the corpus. We compared synthesized sounds with and without data augmentation. Our main target is to evaluate the impact of data augmentation on visual onoma-to-wave, but we also applied data augmentation to the conventional onomato-wave. We used 1, 000 and 2, 000 audio samples for evaluation of word-level and character-level augmentation, respectively.
For objective evaluation, we calculated sound duration along with the word or character repetitions. The duration of non-repeated onomatopoeia was normalized to 1.0, and the relative changes in duration was calculated. For subjective evaluation, we conducted MOS tests as in Section 4.1. Figure 5 (a) and Figure 6 (a) show the results of the objective evaluation. Training without data augmentation did not increase the duration along with the number of repetitions. Since FastSpeech2 explicitly assigns duration to each character, the repeated words and characters had non-zero durations. However, we found that the durations of the repeated ones were very close to zero; increasing rep-   etitions no longer change the total duration. In contrast, our data augmentation method solves this problem, and the duration changes near-linearly. Figure 5 (b)-(d) and Figure 6 (b)-(d) show the results of the subjective evaluation. As shown in Figure 5 (d), our data augmentation method did not lose the MOS in naturalness compared with no data augmentation. In contrast, from Figure 5 (b)-(d), our data augmentation method had better MOSs in acceptance and expressiveness than no augmentation. As mentioned above, synthesized sounds without data augmentation do not respond to the increase in repetitions. This causes perceptual mismatch between the onomatopoeia and synthetic sound. Our data augmentation method efficiently solves this and preserves MOSs even if synthesizing from onomatopoeias with many repetitions. The same tendency was observed in character-level repetitions, as shown in Figure 6 (b)-(d), but significant improvements with our data augmentation method were observed for naturalness.

Duration control by image stretching
We evaluated duration control by image stretching. The stretch ratios were set to 0.5, 1.0, 1.5, 2.0, and 200 audio samples were used for each stretch ratio. For comparison, we trained models without pre-processing of image stretching. The objective evaluation was on the relative duration of the sound, setting the duration without stretching (i.e., the ratio of 1.0) to 1.0. The subjective evaluation was conducted in the same manner as in Section 4.2.2. Figure 7 shows the results. From Figure 7 (a), the sound duration changed linearly, so we can say that stretching-based duration control performs well. In contrast, as shown in Figure 7 (b)-(d), while the MOS on naturalness were maintained among the ratios, those on acceptance and expressiveness degraded when the ratio was 0.5. We will investigate the reason for this in the future.

Synthesis of diverse sounds from sound event image
We evaluated the quality and diversity of sound synthesized from visual onomatopoeias and auxiliary images. We prepared additional sound-onomatopoeia-image pairs for the evaluation. For sound events in RWCP-SSD, we crawled the Internet and collected 50 photos per event, which matched the sounds. Additionally, we prepared a new sound event class "baseball bat" to evaluate the synthesized sound. We crawled the Internet to collect 150 photos  of baseball bats made of metal, wood, and plastic. GameSynth 7 , a sound effect creation tool, was used to artificially generate the corresponding sounds. The corresponding onomatopoeias were made from crowdsourcing as in the onoma-to-wave paper [6], and at least eight onomatopoeias were obtained for each sound. 267 sound-onomatopoeia-image pairs were used for evaluation, and the remaining were for the training data. The RWCP-SSD sound event data was used only for training and not for evaluation. There was no overlap in sounds, onomatopoeias, and images between the training and evaluation data. Using these data, we trained the photoconditioned synthesis model and evaluated the synthetic sounds. For application to comics, we also trained the line-drawing-conditioned synthesis model. Anime2Sketch [25] 8 was used to convert the obtained photos to the line-drawing style. For comparison, we also trained a label-conditioned model, i.e., the "baseball bat" label was used for synthesis, the same as with the conventional onoma-towave. We used ViT-L/14 [26], as the CLIP [18] image encoder 9 . The objective evaluation involved sound diversity as a mean squared distance between all combinations of the synthesized sounds [27]. The subjective evaluation was conducted in the same manner as in Section 4.2.3. Figure 8 and Table 2 show the results. From Figure 8, the distributions of sounds of photo-conditioned and line-drawingconditioned synthesis were close to that of the reconstructed sounds. On the other hand, label-conditioned synthesis was less in diversity compared with the others. We confirmed that the use of sound event images enables us to synthesize more diverse sounds than using labels. From Table 2, the photo-and line-drawing-conditioned synthesis achieved significantly higher MOSs on acceptance and expressiveness than the label-conditioned synthesis. Therefore, we can say that sound event images help synthesize appropriate sounds. In contrast, we also observed that it degrades the naturalness MOSs. This may be because label-conditioned synthesis uniquely determines features, whereas photo-and line-drawing-conditioned synthesis do not, thus making the synthesis slightly unstable. We will address this for future work.

CONCLUSION
We proposed visual onoma-to-wave, which uses image representations to synthesize environmental sounds from onomatopoeias. For future work, we plan to synthesize sound from real data.