{'fraction_non_alphanumeric': 0.025660926509822945, 'fraction_numerical': 0.009992723744845985, 'mean_word_length': 5.273889227023737, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
MaskedSpeech: Context-aware Speech Synthesis with Masking Strategy

Humans often speak in a continuous manner which leads to coherent and consistent prosody properties across neighboring utterances. However, most state-of-the-art speech synthesis systems only consider the information within each sentence and ignore the contextual semantic and acoustic features. This makes it inadequate to generate high-quality paragraph-level speech which requires high expressiveness and naturalness. To synthesize natural and expressive speech for a paragraph, a context-aware speech synthesis system named MaskedSpeech is proposed in this paper, which considers both contextual semantic and acoustic features. Inspired by the masking strategy in the speech editing research, the acoustic features of the current sentence are masked out and concatenated with those of contextual speech, and further used as additional model input. The phoneme encoder takes the concatenated phoneme sequence from neighboring sentences as input and learns fine-grained semantic information from contextual text. Furthermore, cross-utterance coarse-grained semantic features are employed to improve the prosody generation. The model is trained to reconstruct the masked acoustic features with the augmentation of both the contextual semantic and acoustic features. Experimental results demonstrate that the proposed MaskedSpeech outperformed the baseline system significantly in terms of naturalness and expressiveness.


INTRODUCTION
With the development of deep learning technology, current textto-speech (TTS) systems [1][2][3][4] could generate high-quality speech. However, most speech synthesis systems only consider the information in the current sentence to be synthesized and ignore the contextual text and speech. With the rising requirements on the naturalness and expressiveness of TTS systems, such as synthesizing audiobooks, these speech synthesis systems lack the ability to generate paragraph-level speech with adequate naturalness and expressiveness, as human speech is context coherent in a paragraph.
Thus, recent works [5][6][7][8][9][10] incorporated semantic features of the current sentence or contextual sentences to improve the prosody of synthesized speech, especially for paragraph speech generation. Xiao et al. [9] extracted semantic information from the current sentence by a pretrained BERT [11] model and showed that the incorporation of semantic features could improve the prosody of synthesized speech. Then works [5][6][7]10] further employed cross utterances semantic features in the speech synthesis systems, and all of 1 Equal contributions. This work was supported by the National Key R&D Program of China under Grant No. 2020AAA0108600. them showed significant rhythm-enhancing effectiveness. Although contextual semantic information could improve the prosodic performance to a certain extent, these works do not take into account the variation of contextual acoustic features, which is also very important for generating expressive and natural speech for a paragraph.
Some works [6,12,13] used acoustics features to improve the prosody modeling, but most of them extracted sentence-level prosody representation and lost the detailed information of contextual acoustic features. In addition, for the reference-based models [12,13], it's non-trivial to select proper reference audio. [14] and [15] introduced an acoustic encoder to learn fine-grained prosody features instead of utterance-level features and claimed that their models could improve prosody generation and relive the one-tomany mapping issue. However, the prosody features need to be predicted by the text when reference audio is unavailable during inference, this led to another one-to-many mapping issue, as the same text can potentially be used to predict acoustic features from different speech.
To incorporate both contextual semantic and acoustic features into the TTS model and improve prosody generation for a paragraph, we proposed MaskedSpeech in this paper. Inspired by the speech editing works [16][17][18], acoustic features of the current sentence are masked out and concatenated with those of neighboring sentences, and further used as augmented input for the proposed network. Then the network could learn fine-grained prosody features from the contextual speech, and no prosody prediction from text is required. Neighboring texts are also concatenated with current text and sent into the phoneme encoder of the proposed model, which helps the model learn fine-grained semantic features from the contextual text. Furthermore, a cross-utterance (CU) encoder is used to extract coarse-grained sentence-level semantic representation. The proposed MaskedSpeech is trained by reconstructing the masked acoustic features, by utilizing the contextual text, contextual speech, and sentence-level semantic representation, which enables Masked-Speech to improve prosody modeling and alleviate the one-to-many mapping issue naturally. Experimental results indicate that the proposed MaskedSpeech can significantly improve the naturalness and expressiveness of synthesized speech over the baseline Fastspeech2 model.

RELATED WORK
This section reviews the acoustic masking strategies used in speech representation learning and speech editing tasks, which is the inspiration of the proposed model. Unsupervised speech representation learning [19][20][21] has demonstrated its superiority in the field of speech recognition and related downstream tasks. To learn meaningful speech representation, these technologies usually mask several frames of acoustic features randomly, and contrastive learning is used to learn discrete quantized speech units as latent representations.
Although the success of speech representation learning in speech understanding tasks, it's not suitable for the speech synthesis task due to its discrete nature. MAM [22] and FAT-MLM [23] masked several acoustic segments randomly and used the masked acoustic features as network input, then Transformer-based [24] network is employed to reconstruct the masked acoustic features. But the quality of the reconstructed acoustic features could not meet the requirement of speech synthesis. Bai et al. [16] proposed a network that uses masked acoustic features as input and the network is trained to predict high-quality acoustic features, then their proposed network could be further applied to speech synthesis and speech edit tasks. CampNet [17] introduced a speech edit model, the acoustic features are randomly masked out by a prior ratio and then sent to the decoder for masked speech reconstruction. Each decoder layer used an attention mechanism to learn contextual features from unmasked part.
All the mentioned works for speech reconstruction try to predict high-quality and contextual consistent acoustic features by learning the context relationship of masked and unmasked speech. Inspired by these works, we incorporate the masking strategy into crossutterance speech synthesis task to improve the prosody modeling, with the assumption that both the textual and acoustic context could facilitate the prosody improvement of speech synthesis. Instead of masking on word-level speech for speech editing tasks, the acoustic features of the current sentence are concatenated with those of neighboring sentences, and then acoustic features from the current sentence are masked out. The proposed model is trained to reconstruct the masked acoustic features. In other words, we proposed a novel context-aware speech synthesis method that reconstructs speech for the current sentence with improved prosody by utilizing both the semantic and acoustic features of contextual sentences.

PROPOSED METHOD
The proposed MaskedSpeech utilizes FastSpeech2 [25] as the network backbone. The model learns to reconstruct the masked acoustic features with both contextual semantic and acoustic features. Mel-GAN [26] is used as the neural vocoder in the experiments to convert acoustic features to speech. Fig. 1 illustrates the network architecture of the proposed Masked-Speech model. The backbone network consists of a phoneme encoder, a variance adaptor, a decoder, and a PostNet module. Besides, to improve prosody for the speech in a paragraph by contextual features, a masked mel-encoder is designed to extract local prosody features from the concatenated mel-spectrograms, and then the decoder takes the output of the masked mel-encoder as augmented input and learns global prosody dependency from the contextual speech. Furthermore, a CU text encoder is employed to obtain CU semantic features from neighboring sentences.

Learning from contextual text
The prosody of the current sentence could be impacted by the relative sentence position, discourse relations (DRs) in neighboring sentences, the emotion of contextual sentences, etc. To improve the prosody modeling with contextual sentences, the concatenated phoneme sequence of neighboring sentences are sent into the The phoneme sequences of the previous sentence, current sentence, and following sentence are concatenated as the input of the phoneme encoder, rather than only using the phonemes of the current sentence in general methods. Thus, the phoneme encoder could learn fine-grained phoneme-level semantic features implicitly from the contextual sentences. Following our previous work [5], Paired BERT embedding (PBE) is used to extract sentencelevel coarse-grained semantic features between sentence pairs in the CU encoder. As shown in Fig. 1, the neighboring sentences {uN−L, ..., uN , uN+1, ..., uN+L−1} can be used to derive 2L sentence pairs, where uN means the current sentence and L is the number of sentences taken into account in the past or following context. Then PBE is extracted by a pre-trained BERT model from the [CLS] token. A multi-head attention module is used to incorporate PBEs into the proposed model, where PBEs are considered as keys and values, and the output of the phoneme encoder is considered as the query. The output of the CU encoder is concatenated with the output of the phoneme encoder, then linear projected.

Learning from contextual speech
Human speech is continuous in a paragraph coherently, the prosody and emotion of the current sentence could be affected by the contextual speech. Contextual speech usually contains very rich, expressive, and detailed prosody information, which could be utilized to improve the prosodic performance for the current sentence in speech synthesis systems, especially for the speech generation in a paragraph.
A masked mel-encoder is incorporated into the FastSpeech2 network in the proposed MaskedSpeech. The acoustic features from contextual sentences, i.e. the mel-spectrograms of the previous sentence and following sentence, and the current sentence, are concatenated and used as the input for the masked mel-encoder. The acoustic features of the current sentence are masked with a special MASK token. The masked mel-encoder consists of two 1-dimensional convolution (Conv1d) layers, it could learn to extract local prosody features from the acoustic features in the neighboring sentences. ReLU activation function is used in each Conv1d layer, followed by layer normalization [27] and dropout [28]. The output of the masked mel-encoder is concatenated with the output of the variance adaptor and then projected to the dimension of the decoder. The decoder consists of several self-attention-based Conformer [29] blocks, so the decoder could learn both local and global prosody dependency implicitly from the contextual acoustic features. The predicted acoustic features of contextual sentences are replaced with ground truth features, and only acoustic features for the current sentence are kept. Then a PostNet module with 5 Conv1d layers is used to refine the predicted acoustic features. As the Post-Net consists of Conv1d layers and learns local features, it could improve the boundary prosody transition for the current sentence.
By conditioning on contextual semantic and acoustic features, the proposed MaskedSpeech could alleviate the one-to-many mapping problem and generate speech with high naturalness and expressiveness for a paragraph.

Training and Inference Strategies
During training, the model is trained to reconstruct the masked acoustic features with the augmentation of both the semantic and acoustic features from contextual sentences. We only compute training objectives for the current sentence and ignore the loss of neighboring sentences, the losses include phoneme-level duration prediction loss, pitch and energy prediction losses, and mel-spectrogram reconstruction loss.
At the inference stage, the speech of the previous or following sentence is not always available. In general, we usually read a paragraph in order. Therefore, there usually exists intact paragraph-level sentences and speech of previous sentences, and do not always have the following speech. Thus, at inference stage, it is assumed that there is only consecutive text and previous speech as default. The speech from the previous sentence is taken as the contextual speech for the synthesis procedure of the current sentence, and the input form of the phoneme encoder and CU text encoder is the same as that at the training stage. In addition, a preliminary experiment showed that there is no noticeable deviation when not using the acoustic features of the following speech.
Actually, the inputs of MaskedSpeech are very flexible. There could be no acoustic features from neighboring sentences, and then MaskedSpeech degenerates into a PBE-based TTS model. When inputting a single sentence and masking speech for a few words, the model degenerates into a speech editing model which can achieve inserting, replacing, and deleting part of an audio file like we edit text.

Experimental Setup
A Chinese Mandarin audiobook corpus with 29 hours of recordings was used to train the proposed model in our experiments. The corpus was recorded by a professional male speaker. The text and recordings in this corpus are consecutive.200 sentences were selected for testing, and 200 sentences for validation, the remaining sentences are used for training. The test set contains several paragraphs of consecutive sentences. The audio files were down-sampled to 16kHz sample-rate. 80-dimensional mel-spectrograms were used as the acoustic features in the experiment and were extracted with a frame shift of 12 ms and frame length of 48 ms. The phoneme boundaries and duration were extracted by Montreal forced alignment (MFA) [30] tool. Pitch was extracted by the PyWORLD 1 toolkit.
The pre-trained "bert-base-chinese" model [31] released by Huggingface 2 was used to extract cross utterance sentence embedding in the CU text encoder. The BERT model consists of 12 Transformer layers, each layer contains 768 hidden units and 12-heads are used for each attention layer. It is worth mentioning that the parameters of the BERT model are fixed during training. The multi-head attention in the CU text encoder has 4 heads and 512 hidden units. In our experiments, PBEs are generated by 2 preceding sentences, 2 following sentences, and the current sentence. The masked melencoder is constructed by 2 Conv1d layers, with a filter size of 256 and kernel size of 3 in each layer.
The proposed model is trained by Adam [32] optimizer, with a batch size of 16 and noam [24] learning rate schedule. The learning rate is warmed up to a maximum value of 10 −3 in the first 4000 steps and then exponentially decayed. Adam optimizer is configured with β1=0.9, β2=0.98 and =10 −9 . Mean squared error (MSE) is used as the training objectives for pitch, energy, and duration prediction, and mean absolute error (MAE) is used for the mel-spectrogram reconstruction. The model is trained by a total of 200,000 steps, with L2 parameter regularization of weight 1e-5. A Fastspeech2 model was trained and used as the baseline model. 3

Subjective Evaluation
To evaluate the quality of the proposed MaskedSpeech model, the Mean Opinion Score (MOS) evaluation is conducted to evaluate the naturalness and expressiveness of the synthesized speech. 20 sentences were randomly selected from the test set and synthesized by different models, each audio is rated by 10 native Chinese listeners. The listeners are asked to rate the speech's naturalness and expressiveness on a scale from 1 to 5 with intervals of 0.5. Score 1 means the speech is unnatural completely and score 5 means the speech is completely natural. Tab. 1 shows the MOS results. Our proposed MaskedSpeech model achieved a higher MOS score than the baseline Fastspeech2 model in terms of naturalness and expressiveness. This indicates that with the help of both contextual semantic and acoustic features, the proposed model could improve the prosody generation.

Ablation Study on Contextual Features
In this section, an ablation study is conducted to evaluate the effectiveness of both contextual semantic and acoustic features. With the incorporation of contextual text and speech, it's expected to help the synthesized audio to have a suitable and consecutive speaking style and consistent prosody contour under the condition of neighboring sentences. To evaluate the effectiveness of these contextual features, two AB preference tests were conducted for the ablation study. First, we remove both the CU text encoder and masked melencoder from MaskedSpeech, which is a pure Fastspeech2 model. Then, we only remove the masked mel-encoder, which is a PBEbased model. The MaskedSpeech was compared with Fastspeech2 and the PBE-based model separately. To evaluate the style consecutiveness and prosody consistency of synthesized speech with contextual sentences, the synthesized audio is concatenated with its previous and following recordings, then the judges are asked to select which one is more natural and expressive according to the ground truth neighboring recordings. Tab. 3 shows the results of AB preference tests. MaskedSpeech outperforms the Fastspeech2 model significantly, which proves the effectiveness of using contextual semantic and acoustic features. Furthermore, MaskedSpeech outperforms the PBE-based model significantly, which proves the fine-grained prosody representations learned from the masked acoustic features can further improve the prosody modeling.
In addition, an objective evaluation was conducted on these three models. The mel-spectrogram distortion (MSD), the root of mean square errors of F0 (F0 RMSE), the error rate of voicing/unvoicing flags (VUV), the Pearson correlation factor of F0 (F0 CORR), and F0 standard deviation (F0 SD) were adopted as the objective evaluation metrics. The predicted mel-spectrograms were aligned towards their corresponding natural ones by dynamic time warping (DTW). The F0s were aligned towards natural ones following the DTW paths from mel-spectrogram alignment. Tab. 2 shows the objective results. Compared to the F0 SD of 45.25Hz for natural recordings, the F0 SD metric from MaskedSpeech is the closest, which is 41.462Hz. While the MSD of MaskedSpeech is the highest among the three models, which may be caused by the higher variation of F0. In addition, MaskedSpeech achieves the best F0 RMSE and F0 CORR among the three models. The objective results further proved the advantages of MaskedSpeech on prosody modeling.  Fig. 2 shows the mel-spectrograms of the synthesized speech of three different models and the natural recording. Each sub-figure shows the concatenated mel-spectrograms of previous, current, and following sentences with distinct silence as the boundary, only the central speech is the synthetic audio. Fig. 2 (d) shows that the speaking speed of the nature recording is relatively faster and the fundamental frequency is lower. MaskedSpeech perceives this property according to the contextual semantic and acoustic features and generates speech with a relatively faster speaking rate and lower pitch.

CONCLUSION
A context-aware speech synthesis system is proposed in this paper, with improved prosody by utilizing the contextual semantic and acoustic features from neighboring sentences. The proposed model takes the concatenated and masked mel-spectrograms as augmented input and learns to reconstruct the masked mel-spectrograms. The phoneme encoder takes account of the phonemes from context sentences and those of the current sentence. A CU text encoder is used to extract cross-utterance semantic representation. Experimental results showed the effectiveness of the proposed MaskedSpeech model.