{'fraction_non_alphanumeric': 0.034634728055166694, 'fraction_numerical': 0.013739824752423913, 'mean_word_length': 4.790453317322125, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Joint Pre-Training with Speech and Bilingual Text for Direct Speech to Speech Translation

Direct speech-to-speech translation (S2ST) is an attractive research topic with many advantages compared to cascaded S2ST. However, direct S2ST suffers from the data scarcity problem because the corpora from speech of the source language to speech of the target language are very rare. To address this issue, we propose in this paper a Speech2S model, which is jointly pre-trained with unpaired speech and bilingual text data for direct speech-to-speech translation tasks. By effectively leveraging the paired text data, Speech2S is capable of modeling the cross-lingual speech conversion from source to target language. We verify the performance of the proposed Speech2S on Europarl-ST and VoxPopuli datasets. Experimental results demonstrate that Speech2S gets an improvement of about 5 BLEU scores compared to encoder-only pre-training models, and achieves a competitive or even better performance than existing state-of-the-art models1.


INTRODUCTION
Direct speech to speech translation (S2ST) has gained more and more attention from research and industry communities in recent years [1][2][3]. Traditionally, cascaded speech to speech translation consists of automatic speech recognition (ASR), machine translation (MT), and text to speech synthesis (TTS) tasks. Direct S2ST aims at integrating the above three tasks into an end-to-end model, which translates the speech of one language to the speech of another language directly. Compared to cascaded S2ST, direct S2ST has the following advantages: (1) it is able to alleviate the error propagation problem of pipeline systems; (2) it can retain the emotion, pitch, and prosody information of the speaker to the greatest extent; (3) it has faster reasoning speed and takes up fewer storage resources.
However, data scarcity is the biggest problem of direct speech to speech translation tasks [4]. At present, there is very little parallel S2ST data though lots of efforts [5][6][7]. To alleviate this problem, a line of work tries to leverage pseudo data to improve direct S2ST [3,8]. They usually convert the ASR data into speech to text translation data using an MT system, and then generate the target audio from the target text with a TTS system. Unfortunately, these methods do not guarantee the accuracy of the generated pseudo S2ST data. Another line of work aims at boosting the performance of direct S2ST through pre-training methods [3,9]. For example, the pa-per in [9] explores pre-training the encoder with mSLAM objective [10], and pre-training the decoder of Translatoron 2 [11] with MT task to generate phonemes. The authors in [3] propose to combine wav2vec 2.0 [12] encoder and mBART [13] decoder to a speech-tounit translation (S2UT) model, which also can be further boosted by data augmentation techniques.
Although the self-supervised pre-training method in [3] can initialize the direct S2ST model with the pre-trained wav2vec 2.0 encoder and mBART decoder, which are trained with discrete units extracted with HuBERT [14] model from unlabeled speech data, it still lacks effective connection between encoder and decoder, and ignores the cross-lingual modeling capacity in pre-training. In the real world, speech data, ASR data, and MT data are relative much more than direct speech to speech corpora, and MT data can be utilized to learn the transformation ability from source text to target text. How to build the cross-lingual bridge between speech encoder and unit decoder of direct S2ST with bilingual text in the pre-training stage is not well explored.
In this paper, we propose a Speech2S model, which aims at modeling cross-lingual information and alleviating data scarcity problems by jointly pre-training with unpaired speech and bilingual MT text for the direct speech to speech translation task. More specially, Speech2S consists of a speech encoder, unit encoder, and unit decoder. We propose two pre-training tasks to pre-train the three modules with unit encoder as the bridge between source speech and target units. Like HuBERT [14], the first pre-training objective is to predict the clustered units based on the output of both speech encoder and unit encoder, with unlabeled speech data. To take advantage of bilingual machine translation corpus, we first leverage two textto-unit models to convert source/target text into source/target units, with which, the cross-lingual unit encoder and decoder can be well pre-trained through cross-entropy loss.
We evaluate the proposed model on Europarl-ST [15] and Vox-Populi [5] S2ST datasets. Our contributions can be summarized as follows. (1) We propose a joint pre-trained Speech2S model, which can take advantage of bilingual text data to boost bilingual speech conversion. (2) The proposed model achieves a significant improvement of about 5 BLEU scores compared to the pre-trained model without MT data. (3) Furthermore, we conduct a detailed analysis about the effect of parallel data size, data augmentation of different domains, and subjective evaluation.

RELATED WORK
Conventional speech to speech translation is usually composed of cascaded ASR, MT and TTS modules [16,17]. On this basis, to avoid error transmission caused by cascade models, researchers ex-plore the combination of ASR and MT modules [18,19], as well as TTS modules [1,20], namely direct S2ST. This paper focuses on exploring direct S2ST with improved pre-training methods.

Direct Speech to Speech Translation
S2ST, which directly translates the source speech to the target speech, has attracted a lot of attention recently [1,2,[20][21][22]. Translatotron [1] is the first work to achieve direct speech-to-speech translation by using a sequence-to-sequence model. This system uses an encoder to model the log-mel spectrogram and predict the target spectrogram by the decoder, combined with the speaker information. Then, a vocoder is used to convert spectrogram into waveform. This work in [11] improves Translatotron system by utilzing a duration-based spectrogram synthesizer enhanced with target phoneme from decoder. Unlike Translatotron, the authors in [2] propose a novel direct speech to speech translation system, which employs discrete hidden units instead of spectrogram as model target before vocoder. They also expand it without using any text data on real-world S2ST tasks [23]. However, real speech to speech translation data is very limited due to the high cost of obtaining such data [5,6]. Our work is to leverage a pre-training approach to alleviate data dependence on direct S2ST dataset.

Pre-Training for Direct S2ST
Recent years have witnessed a great progress on pre-training techniques for direct S2ST tasks [3,9]. The work in [9] employs speechtext joint model from mSLAM as the encoder, to generate phoneme sequence with MT task and generate spectrogram with S2ST task. The most related work to our paper is [3], which enhances the speech-to-unit translation (S2UT) model by a wav2vec 2.0 [12] encoder and a decoder from pre-trained unit mBART [24]. In this S2UT model, wav2vec 2.0 is pre-trained on unlabeled audio data, and mBART leverages reduced discrete units tokenized from unlabeled audio data to train a denoised encoder-decoder model, and finally uses the mBART decoder to initialize the S2UT decoder. However, the simple combination of wav2vec 2.0 encoder and mBART decoder lacks cross-language modeling capabilities, which is particularly important for translation tasks. Motivated by this, we propose to bridge the language gap by utilizing machine translation corpus to improve model pre-training for direct speech to speech translation.

THE PROPOSED METHOD
Our goal is to leverage paired machine translation corpora to bridge the semantic gap between source speech and target speech. In this section, we will first introduce the model architecture of Speech2S, and the details of the model pre-training and fine-tuning methods.

Structure of Speech2S
As shown in Figure 1, Speech2S consists of a speech encoder Es, a unit encoder Eu and a unit decoder Du. Speech encoder and unit encoder employ standard Transformer network [25] with the same Transformer layers, except that a 5-layer CNN network in speech encoder is used to pre-process the original audio signal. Unit decoder is a multi-layer Transformer decoder layer which is composed of a multi-head self-attention mechanism, cross-attention mechanism, and a FFN network.
Formally, we denote unpaired speech as S, and denote bilingual text as (X, Y ). After applying the speech and text discretization modules (as introduced in Section 3.2.1), we obtain the speech units Su from S and bilingual units (Xu, Yu) from (X, Y ). Briefly speaking, Es is used to encode the source audio sequence S into a sequence of vector representation H m . Following the mixing mechanism proposed in [26], we also adapt it to improve alignment learning by randomly replacing part of H m with the corresponding unit embedding. Eu can transform speech representation H m into final hidden states H f , or transform source unit sequence Xu into unit hidden states U e . Besides, Du reads the encoder representations and generates a target unit sequence Yu.

Model Pre-Training
Before pre-training, we first use two discretization modules to tokenize speech and text into shared discrete tokens. Then the model can be optimized by two pre-training objectives, including speech to units task using speech encoder and unit encoder, and source units to target units task using unit encoder and unit decoder.

Speech/text discretization
We use HuBERT k-means cluster as the speech discretization module, which is learned from the HuBERT iter-1 hidden states, and can tokenize unlabeled speech into discrete hidden units. To tokenize text into the same space like speech, we introduce two text-to-unit models like [26], which are trained by using two small ASR corpus with paired speech and transcription. More specifically, we first use speech discretization to convert paired speech into hidden units, and obtain the text, unit data by combining it with paired text. Then we utilize a sequence-to-sequence model to achieve the textto-unit models trained on the paired text and unit data. Once obtaining the discrete models, we can tokenize unlabeled speech S into hidden units Su, and tokenize bilingual text (X, Y ) into bilingual units (Xu, Yu), respectively, all of which can be used to optimize the model in pre-training stage.

Pre-training objects
When the input audio S is fed into the speech encoder Es, it is partially masked and encoded into middle hidden states H m = where ut ∈ Su is the target hidden units, and the p(.) is parameterized as the same way with HuBERT [14]. Unit encoder Eu also takes Xu as input in pre-training stage, and use Eu(Xu) to output the encoded unit hidden state U e . The unit decoder Du will generate a series of hidden states U d = Du(U e ) according to the encoder representation of source units. The objective function of unit pre-training is formalized as, where yu,i ∈ Yu, Yu,<i denotes {yu,0, yu,1, ..., yu,i−1}, and p(.) is a softmax layer. Finally, we pre-train Speech2S under multi-task learning framework with L = L speech + Lunit.

Speech2S Fine-Tuning
In the fine-tuning stage, we can fine-tune Speech2S with speech encoder, unit encoder, and unit decoder to a direct speech-to-speech translation model. Leveraging the cross-entropy loss, we simply employ direct S2ST corpus as the fine-tuning dataset to optimize the model, where the target speech needs to convert into target units using speech discretization module. Finally, we utilize a unit-based HiFi-GAN [23] to generate the target waveform from target units.

Datasets
We conduct our experiments on two directions of the same language pair: Spanish-English (es-en) and English-Spanish (en-es). For pretraining, we use VoxPopuli dataset, a large-scale multilingual corpus providing 100K hours of unlabelled speech data in 23 languages, as speech pre-training data. The ASR subset of Voxpopuli (VoxPopuli-ASR) in each language is used to train the textual discretization module, namely sequence-to-sequence based text-to-unit model. We use machine translation data between English and Spanish from Europarl v10 [27] as the bilingual text data to generate paired text units for textual unit pre-training. Meanwhile, the speech-to-speech paired data VoxPopuli-S2S is used for our S2ST fine-tuning stage. We use the dev set split from VoxPopuli and the dev/test set of Europarl-ST dataset to verify the effect of speech to speech translation models. In order to avoid duplication with the corpus of the test set, we deleted the data of 2012 and earlier in the VoxPopuli training set. To avoid errors caused by audio itself, all audio is unified to the 16 kHz ogg format. In addition, we use the training sets text of CoVoST-2 and Europarl-ST datasets for additional analysis experiments on data augmentation for different domains. Data details are shown in Table 1.

Implementation Details
Discretization We use released k-means cluster model 2 from multilingual HuBERT (mHuBERT), which trained with VoxPopuli 100k subset [23], to extract units from speech data. For text discretization, we first extract the units of Voxpopuli-ASR speech using mHu-BERT cluster and normalize the units using the same 1h English or Spanish speech normalizer as [23]. Then we train the text-to-unit discretization model using the normalized units and transcripts of the corresponding speech of the units. The text-to-unit model has Pre-training Our Speech2S is composed of a 6-layer Transformer speech encoder, a 6-layer Transformer unit encoder a 6-layer Transformer decoder and an output FFN layer of 1024 units. Each Transformer layer has 768 nodes with 4 attention heads and relative positional attention bias [28]. We pre-train with the same 400k training steps for all models.
Fin-tuning The fine-tuning model structure is basically the same as the pre-training model structure. The normalized units of target language used in fine-tuning stage are extracted using the same extractor as the text-to-unit model. After generating units, we use unit based HiFi-GAN [23] to generate target speech. English and Spanish use recognition models wav2vec 3 and microsoft speech-totext tookit 4 to transcribe into text, respectively. The SacreBLEU toolkit [29] is used to calculate the final BLEU score.
Baselines For comparison, we design two strong baselines for the experiment. The first one employs HuBERT encoder to initialize the encoder of speech-to-unit translation model, and the other is existing S2UT model [3], which is initialized with HuBERT encoder plus 6-layer unit level mBART decoder. The two models use the same speech data as our model for pre-training and fine-tuning. The parameters of the S2UT base model and our Speech2S model are almost the same. Table 2 shows the BLEU scores of S2UT systems [3] and our Speech2S systems. By comparing the model fine-tuned from Hu-BERT and our proposed model, results show that our model achieves more than 4 BLEU value gains on the S2ST tasks in both directions (#5 vs. #3). Compared to S2UT base model fine-tuned from Hu-BERT encoder and mBART decoder, the proposed Speech2S model still has an improvement of more than 3 BLEU scores (#5 vs. #4). This result proves that our model can better incorporate text information into the language model through pre-training, and learn the corresponding relationship between source language speech and target language units through shared unit encoder. Furthermore, we

Effect of Parallel Data Size
An interesting question is how well does the model perform if we only have very little fine-tuning data. Here, we verify the effect of varying parallel data size for Speech2S and baselines. We evaluate the proposed Speech2S and baseline from HuBERT on 10 hour, 50 hour and 100 hour supervised data sets respectively. These training data are randomly sampled from all data of VoxPopuli-S2S.  Table 3, we can find that even if there is only 10 hours of supervised data, through our joint pre-training with speech and bilingual text, the BLEU can reach more than 10. On the 100 hour supervised data set, the fine-tuning results are close to those of hundreds of hours of supervised data fine-tuning. From the results of weak supervision, we can draw a conclusion that the Speech2S model can learn the unified mapping of speech and unit well through pre-training, thus reducing the dependence on supervised S2ST data.

Effect of Data Augmentation
In this section, we explore the effect of data augmentation for different domain datasets. As shown in Table 4, we first evaluate the performance on CoVoST-2 dev/test sets using the model trained with VoxPoluli train set. In terms of absolute performance, the BLEU scores of CoVoST-2 underperform significantly that of Europarl-ST (#3 vs. #1). A potential reason is that the pre-training and finetuning data domains are consistent for Europarl-ST test set, but it has a domain mismatch problem between VoxPopuli and CoVoST-2.
We conduct data augmentation experiments by adding the paired source speech and target unit data from Europral-ST and CoVoST-2 speech-to-text translation dataset. Based on the training data, which consists of source speech and target text, we use the text-to-unit model trained on VoxPopuli-ASR data to convert the text of the target language into units, and then enlarge the training set with the speech and generated target units, as shown in the line 2 and 4 of Table 4. With data augmentation, the Speech2S can achieve bigger improvements on CoVoST-2 than Europarl-ST, which confirms our suspicions. Experimental results also demonstrate that this data augmentation method is very effective for domain adaption.

Subjective Evaluation
To further compare the speech quality generated by different models, we select 50 samples from the Europarl-ST dev set and test the naturalness score of these samples. Table 5 lists the naturalness score of different models, including S2UT model and our Speech2S models without and with data augmentation. The results show that our proposed Speech2S achieves the naturalness score of 4.1, outperforming S2UT model fine-tuned from HuBERT and mBART. With data augmentation, the Speech2S model obtains the best naturalness score of 4.3. Experiments demonstrate that our proposed method not only significantly improves the translation quality of S2ST tasks, but also enhances the naturalness of generated speech. In addition, we can find from this experiment that more accurate units will also help to improve the quality of the final synthesized speech.

CONCLUSION
This paper proposes a novel pre-training method with unlabeled speech and paired text data for direct speech to speech translation. The core of the proposed Speech2S is to enhance the cross-lingual speech conversion capability by modeling the transformation from source units to target units, which are extracted from bilingual text data using a discrete tokenizer. Experimental results and analyses on common VoxPopuli and Europarl-ST speech-to-speech translation tasks demonstrate the effectiveness and superiority of the proposed Speech2S model.