{'fraction_non_alphanumeric': 0.029744014923826303, 'fraction_numerical': 0.010208311742149445, 'mean_word_length': 5.059340659340659, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
s-Transformer: Segment-Transformer for Robust Neural Speech Synthesis

Neural end-to-end text-to-speech (TTS) , which adopts either a recurrent model, e.g. Tacotron, or an attention one, e.g. Transformer, to characterize a speech utterance, has achieved significant improvement of speech synthesis. However, it is still very challenging to deal with different sentence lengths, particularly, for long sentences where sequence model has limitation of the effective context length. We propose a novel segment-Transformer (s-Transformer), which models speech at segment level where recurrence is reused via cached memories for both the encoder and decoder. Long-range contexts can be captured by the extended memory, meanwhile, the encoder-decoder attention on segment which is much easier to handle. In addition, we employ a modified relative positional self attention to generalize sequence length beyond a period possibly unseen in the training data. By comparing the proposed s-Transformer with the standard Transformer, on short sentences, both achieve the same MOS scores of 4.29, which is very close to 4.32 by the recordings; similar scores of 4.22 vs 4.2 on long sentences, and significantly better for extra-long sentences with a gain of 0.2 in MOS. Since the cached memory is updated with time, the s-Transformer generates rather natural and coherent speech for a long period of time.


Introduction
End-to-end sequence-based TTS models such as Tacotron [1] [2] and Transformer TTS [3] [4] have successfully applied to model text, speech pairs. With the help of high quality neural vocoder like WaveNet [5], LPCNet [6], GAN based vocoder [7], and etc, the generated speech quality could be very natural. As a unified framework, the seq2seq TTS model has largely improved the model capability, taking advantages of direct modeling from text input to speech and whole utterance modeling with either RNN attention or Transformer multi-head attention.
However, it is still challenging for the seq2seq TTS model to generate extra long sentences, as the ability of modeling complex and long-term dependency remains a research problem. RNNs are difficult to optimize due to gradient vanishing and explosion [8] while the sequence length increasing. Transformer [9], based solely on attention mechanisms, with global dependencies between input and output, owing efficient computation and high parallelization, requires large memory growth of sequence length and is difficult to optimize during training on the various length speech corpus, especially when the acoustic feature number arrives to thousands of frames.
How to capture the long-term dependency has been a generic challenge for the sequence transduction model. Various attention mechanisms are tried in the encoder-decoder attention model, like stepwise monotonic attention [10], forward-Samples are available at https://vancycici.github.io/sTransformer/ attention [11], monotonic chunkwise attention [12], and etc. However, these attention mechanisms are usually based on previous state calculation, and not easy to apply into the high parallelized attention calculation of Transformer model. Sparse-Transformer [13], introducing sparse factorization on the attention matrix, gains the ability to produce thousands of timesteps on images, audio and texts, nevertheless it only works for the self-attention, which is not direct suitable for encoder-decoder attention as different length pair as text, speech . In the speech recognition tasks, there are many works on the streaming end-to-end model [14], using CTC plus Transformer [15] [16] or RNN-Transducer [17] plus LAS [18] [19]. These two-stage flows need to be jointly decoded or rescored later. Transformer-XL [20], incorporating a segment-level recurrence mechanism and a novel positional encoding schema, achieves excellent performance in language model task for extra long sequence. It reuse the hidden states obtained from previous segments and builds up a recurrent connection between segments. Thus, modeling very long-term dependency becomes possible because information can be propagated through the recurrent connections.
Inspired by the segment streaming model, we think it would be feasible to model speech utterances into segments without disrupting the temporal coherence. We have tried segment based Tacotron model using partial conditioning with incremental prediction [14], but due to the size limitation of previous RNN state, the experiment reports inferior performance. Accordingly we turn to Transformer-XL like model which is flexible and high efficient, and propose s-Transformer for speech synthesis. This recurrence mechanism is incorporated in both Transformer encoder and decoder self-attention, as text input and spectrogram output segments, with previous segments computed as hidden states cached as extended context, while the encoder-decoder attention aligns in segment-level. The cached memory lengths for encoder and decoder are setting differently regarding the assumptions that text sequence usually has a longterm dependence while the speech spectrogram is more local related. We set a large cached memory size for encoder, and a smaller one for decoder. With the decoder cached memory, it allows the adjacent acoustic features between segments to participate in the computation as segment overlap, to relax the strict alignment of segment and maintain the coarticulation in continuous speech. As the cached memory size is fixed, the memory will update while segment passing through and "forget" the "too far" segment when it exceeds the memory capacity. What's more, with the relative positional self attention, it gains generalized attribute regardless of various speech utterance lengths. The s-Transformer can generate extra long utterance which is even unseen in the corpus and obtains the appealing modeling capability beyond sentences like paragraphs.

s-Transformer
The proposed s-Transformer based speech synthesis system is shown in Figure 1. Given an input text sequence em-  . . x τ,l } and corresponding output chunk yτ = {yτ,1, yτ,2, . . . y τ,l }. Different from Transformer TTS modeling on the whole utterance level, it applies the s-Transformer model on the speech segment, keeping the segments internal order in the utterance. In the Transformer encoder and decoder blocks, the self-attention modules are calculated with previous segment recurrence as extended context with the help of cached encoder and decoder memories. The Transformer encoderdecoder attention remains the same to produce variable output lengths by using the query vectors and decoder states to generate the attention matrix to a sequence of encoder state. Although the gradients are calculated in the segment, the additional extended context allows the network to exploit the longrange context of previous history information. In other words, the s-Transformer essentially works similar as the monotonic chunk-wise attention with segment recurrence, to ease the optimization for s-Transformer architecture while learning longterm dependency.

Segment level recurrence
With the recurrence mechanism to Transformer architecture for consecutive segments, the hidden state sequence computed for Current segment Previous segment −1 Next segment +1 Figure 2: Illustration of segment level recurrence for self attention, with an example of segment length 4.
the previous segment is cached to be reused as an extended context when the model processes the current segment. Let the two consecutive segments be as xτ = {xτ,1, xτ,2, ..., x τ,l } and xτ+1 = {xτ+1,1, xτ+1,2, ..., x τ +1,l } respectively, denoting the n-th layer hidden state sequence produced for the τ − th segment s τ . Then, the n-th layer hidden state for segment sτ+1 is calculated as followings: Where the function SG(·) stands for stop gradient, the notion [hv • hu] indicates the concatenation operation for the two sequences along the length dimension as extended context, and W denote the Transformer model matrix parameters. The critical difference is the hidden stateh n−1 τ +1 which is extended with previous segment. Correspondingly, the current segment based key k n τ +1 and value v n τ +1 are derived from this extended hidden state, while query q n τ +1 remains the same computation from standard hidden state as h n−1 τ +1 . The cached memory could contain multiple previous segments, which extends the context information beyond just two consecutive segments. It's noticed that the recurrent dependency between connected layers shifts one layer downwards per-segments, which is close to global dependency as Transformer self-attention, using local query to compute the attention with extended hidden state derived key and value. Consequently, the largest dependence length grows linearly with number of layers, segment length and cached memory length.

Relative positional self attention
In order for the Transformer model to make use of the sequence order, the relative or absolute positional embedding is injected with the input embeddings at the bottoms of the encoder and decoder stacks. In Transformer TTS, we use scaled positional encoding [3] instead of fixed positional embeddings, to adaptively fit the scale of both encoder and decoder, regarding the range of text and speech sequence are quite different. The scaled weights fit for both encoder and decoder positional embedding are trainable. When we reuse the extended context into hidden state, we should also handle the coherence of position information. In the segment positional embedding, information of relative position in the internal segment sequence is injected by triangle positional embeddings. Meanwhile, similar as [21][22] introducing relative position representations to allow attention to be informed by how far two positions are apart in the sequence, we propose to add a trainable bias matrix to learn the relative position embedding for the extended context between a query and key respectively. In the standard Transformer TTS, encoder self-attention has a masking bias matrix to mask the padding input embeddings, while decoder self-attention has a casual masking bias matrix to prevent the usage of future mel-spectrograms. Let the masking bias matrix denoted as B cur which is based on current segment input embeddings, the trainable bias matrix denoted as B rel for the extended context, an L × (L + M ) dimensional logits matrix multiplied by Q and K, which modulates the attention probabilities for each head as: indicates the concatenation of the relative extended bias matrix and corresponding bias matrix along the time length dimension.

Chunk stop token
Similar as Tacotron2, Transformer-TTS model also employs an utterance stop token to help autoregressive generation stop emitting output while synthesis. As in the s-Transformer, it should have the capability to know when to stop generating output for this segment. We propose to add a chunk stop token network which works together with utterance stop token. The chunk stop network is a Linear network similar as stop network predicted from decoder output. The final stop logit is weighted summarized by both stop token and chunk stop token jointly to decide when to stop for this segment no matter it's beginning, middle or ending part in the utterance.

Chunk speaking rate
To improve the various speaking rate for multiple speech segments, we propose to add a chunk speaking rate feature into training. The chunk speaking rate is calculated as: L chunk and T chunk represent the chunk input embedding length and corresponding mel-spectrogram frames. The chunk speaking rate is calculated as a feature during the training, adding with encoder output. Additionally, there is another chunk speaking rate network as Linear network trained from encoder output. During inference, the chunk speaking rate could be estimated from encoder output and averaged according to the encoder length dimension axis.

Sentence feature
There are some global sentence features, such as the sentence type which is implied in the "future" segment, missing in the s-Transformer while model the current segment. To address this limitation, we collect the sentence feature as additional global embedding input, pass through another Pre-net, add with the segment input embedding as encoder input.

Experiments
We apply Transformer TTS as baseline and s-Transformer on a professional enUS female speaker, 45. 8

Baseline
The baseline model have the similar architectures and hyper parameters as Transformer TTS [3]. Both encoder and decoder are composed of 6 layers with hidden size as 512 and each multihead attention has 8 heads, including encoder-decoder attention.
We use phoneme as input and 80-channel mel-spectrogram as output, the speech frame shift is 12.5ms. Dynamic batch size is used to support for efficient parallel computing. The model is trained on NVIDIA V100 4xGPU 16GB, with maximum batch size 64 on each GPU, using Adam optimizer β1 = 0.9, β2 = 0.999, = 10 −8 and scheduled learning rate exponentially decay. At first, we remove the long utterances (>800 frames as 10 seconds, about 20% in the corpus) to get a stable baseline model, for some long utterances in the corpus will lead the encoder-decoder attention hard to learn. Then, we use all utterances to refine the model. To help the baseline model on extra long sentence, we augment the corpus about 40% utterances to increase the long sentences percentage by merging two sentences into one, from 1200 to 2400 frames length utterances increase about 20% for further model refinement.

s-Transformer
We employ the same parameters as Transformer TTS, with 6 layers for both encoder and decoder with hidden size as 512, self multi-head as 8, but the encoder-decoder multi-head attention number is reduced to 4 considering the segment alignment task would be easier. The encoder and decoder Pre-net and output Pos-net are the same as Baseline.

Order chunk reader
There are some additional parameters for s-Transformer: chunk size for input segment, cached encoder and decoder memory length. The speech corpus is aligned by a speech recognition model on phoneme duration. Chunk size for input segment is calculated for the input phonemes including punctuation, delimiter and etc., and corresponding me-spectrogram segment is obtained according to the phoneme alignment. The chunk size is not exactly segmention point, but to find the center phone.
To preserve the speech coarticulation for phonemes in a word, a search window from center phone left to right is set as 20 to find an appropriate segmentation point, preferring punctuation or word boundary. The data reader supports to fetch the segments in utterance order within a fixed batch size.

Training and synthesis
s-Transformer performs training iteration on each segment batch with fetched cached encoder and decoder memories. The cached memory will be zero initialized at each utterance beginning. We conduct the s-Transformer training on NVIDIA V100 There is no need to prune the training corpus for s-Transformer is robust to various speech utterance lengths. We explore the following parameter settings for the experiments: chunk size=60, decoder memory=4, encoder memory lengths as 120 and 180 respectively. During synthesis, the phoneme input is also followed the same strategy for the segmentation. The whole utterance is generated segment by segment and merged in order as one utterance.

Evaluation
In order to test the general voice quality of TTS model and the capability of long-term dependence, we prepare 3 different domains of test set, each set contains 30 sentences, content from news and audiobooks, but different on the text length (total word number) and rough estimated speech length according to synthesized speech. Firstly, we conduct a group of CMOS tests to compare the voice quality between different Baseline and s-Transformer models. According to Table1 CMOS results, we observe that comparing with Baseline(<800), the s-Transformer with encoder memory size 120 could achieve equal naturalness on short sentences, obvious better voice quality on long sentences, due to Baseline model is trained on a tailored corpus without over 800 frames long utterances. After refinement with all corpus, the Baseline model is improved on both short and long sentences. The s-Transformer with encoder memory size 180 improve the voice quality on both short and long sentence comparing with Baseline(all), as expected that longer context would help to improve the overall prosody and quality. To further test the capability on extra-long sentence and impact of training corpus, we perform model refinement on the augmented corpus included a certain degree of augmented extra-long training utterances for both Baseline and s-Transformer. We still observes many attention errors occurs when the input text is too long for Baseline model, especially the intelligibility drops a lot for Baseline model synthesis beyond 1500 frames, while s-Transfomer performs not much change. After removing some serious attention error samples in the extra-long test set, the CMOS=0.415 indicates that s-Transformer has obvious advantage on the extralong sentences.
Then, we select the Baseline (all) and s-Transformer with encoder memory 180 (all) for short and long sentences, and Baseline(aug.) and s-Transformer(aug.) for extra-long sentences to perform MOS test, to evaluate the general naturalness with the recording in similar domain. For the short and long domains, Baseline and s-Transformer gain equal 4.29 MOS score, very close to the recording as gap=0.03 on short sentences, bigger gap=0.2 on long sentences. Specifically, for extra-long test set, s-Transformer significant outperform Baseline with MOS score 3.99 versus 3.79. It performs very stable on the extralong sentences, and in theory, it even be able to generate speech as long as you want. However, due to extra-long sentence like audiobook content is very challenging to obtain rich prosody, s-Transformer still has MOS gap 0.6 with recording, suggesting that even longer-term context by enlarging the cached memory and extracting higher level history and future information are potential beneficial. At last, we select a extra long sentence sample as case study to observe the encoder-decode attention which represent the alignments for baseline and concatenate segment alignments for s-Transformer. In the Figure 3, comparing with Baseline, in particular the alignment over 1000 decoding steps, s-Transformer alignment is much clearer and smoother, which is essentially like monotonic chunk-wise attention.

Conclusions
In this paper, we propose a novel architecture as s-Transformer model for speech synthesis, which utilizes segment recurrence to capture long-term dependence and performs segmentlevel encoder-decoder attention to handle the difficulty of long pair text, speech . Our experiments show the proposed s-Transformer can achieve equal naturalness as standard Transformer TTS on general test set, in particular stable and obvious better quality on extra-long sentences, even unseen long speech as couple minutes as thousands frames. With this segment recurrence mechanism, it builds up a framework for long speech synthesis even beyond sentences.