{'fraction_non_alphanumeric': 0.02630202886425434, 'fraction_numerical': 0.01260196611587534, 'mean_word_length': 4.707251566696509, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Real-Time MRI Video synthesis from time aligned phonemes with sequence-to-sequence networks

Real-Time Magnetic resonance imaging (rtMRI) of the midsagittal plane of the mouth is of interest for speech production research. In this work, we focus on estimating utterance level rtMRI video from the spoken phoneme sequence. We obtain time-aligned phonemes from forced alignment, to obtain frame-level phoneme sequences which are aligned with rtMRI frames. We propose a sequence-to-sequence learning model with a transformer phoneme encoder and convolutional frame decoder. We then modify the learning by using intermediary features obtained from sampling from a pretrained phoneme-conditioned variational autoencoder (CVAE). We train on 8 subjects in a subject-specific manner and demonstrate the performance with a subjective test. We also use an auxiliary task of air tissue boundary (ATB) segmentation to obtain the objective scores on the proposed models. We show that the proposed method is able to generate realistic rtMRI video for unseen utterances, and adding CVAE is beneficial for learning the sequence-to-sequence mapping for subjects where the mapping is hard to learn.


INTRODUCTION
In recent years, real-time Magnetic Resonance Imaging (rtMRI) has been emerging as an important tool in understanding speech production [1]. The rtMRI of a speaker allows visualising the dynamics of various speech articulators, irrespective of language or health conditions. While other data sources such as Electromagnetic articulography (EMA) [2], X-ray [3] and Ultrasound [4] can provide data at a higher frequency, rtMRI captures various regions in the midsaggital plane such as the pharynx, larynx, velum along with jaw and the articulators in the oral cavity. EMA tracks sensors on a few articulators in the mouth, hence a complete view of different articulators is not present. Similarly, ultrasound imaging doesn't capture structures such as palate or pharyngeal walls [5]. rtMRI is also non-invasive, which is not the case for EMA. Additionally, there has been work [6] at extracting rtMRI frames at a higher sample rate.
The rtMRI allows for understanding the movement and coordination of various speech articulators. Visualisation of articulators been used to aid in learning pronunciation [7,8] and air tissue boundaries obtained from rtMRI has been used in various applications such as speech synthesis [9], speaker verification [10], phone classification [11] etc. Different works [12] have also looked at understanding articulation using rtMRI data.
While rtMRI has its benefits, there is still considerable effort and apparatus required to collect rtMRI data. Hence, there have been efforts to synthesise rtMRI frames from other modalities. Previously, there have been works on obtaining word level rtMRI frames using concatenative approaches [13], [14]. In these methods, a phone-specific image repository is created from the training set by aligning frames to phonemes based on forced alignment. During test time, frames are selected for the required phonemes by employing a dynamic programming approach to ensure smoothness across frames. This is followed by interpolation to match the duration as required by forced alignment. The synthesis is performed on a word level and the authors found that the synthesis procedure was able to produce a realistic video but having more phonemes in a word reduces the quality of synthesis. Hence, it is not ideal for longer words or full utterances.
We explore the use of end-to-end deep learning models in order to learn the mapping from phonemes to rtMRI video, on a full sequence. Deep learning methods are appropriate for this task since the models can learn across long sequences of arbitrarily different lengths, and can learn between modalities and generalise well to unseen situations. In recent years, transformer neural networks have been shown to perform well on various speech-based tasks involving sequence-to-sequence modelling such as speech synthesis [15], acoustic to articulatory inversion [16] etc. Self-attention operation in transformer networks enables learning the dependencies between positions in a sequence, making it suitable for learning good phoneme-level features. Convolutional neural networks have been commonly used as image decoders in various image prediction tasks such as U-Net [17], SegNet [18] and combinations of 2D and 3D convolution have been utilised to learn spatial and temporal information in video prediction tasks [19]. A mix of 2D and 3D convolutions is suitable for decoder since rtMRI synthesis requires the articulators to be anatomically accurate w.r.t. the uttered phoneme while having smooth transitions in phoneme boundaries. We also incorporate a conditional variational autoencoder (CVAE) to incorporate frame-level features which encode prior information on articulators for corresponding phonemes. This type of sampleconditioned training methodology has helped in image prediction in specific domains requiring anatomical correctness such as cardiac image segmentation [20].
To summarise, we treat phoneme to rtMRI synthesis as a sequence-to-sequence prediction task. We use a transformer neural network as an encoder to represent the phonemes of the full input sequence and use a series of convolutional neural networks as decoders to predict the rtMRI frames. We also add to this approach by conditioning the image decoder with intermediary features from a pretrained CVAE, to aid in learning the mapping. We perform a subjective evaluation on the synthesis of all subjects' data, along with an objective evaluation with dice score on air tissue boundary (ATB) masks from a pretrained SegNet, which is trained on rtMRI to ATB masks.

DATASET
We use rtMRI data from USC-TIMIT [21] speech production database in our work. The dataset consists of rtMRI data of 10 American English speakers (5 male and 5 female). The MRI is obtained at a frame rate of 23.18 with frames of shape 68 x 68 x 3. The audio is recorded simultaneously at 20kHz frequency, from 460 utterances from the MOCHA-TIMIT corpus. [22]. Phoneme alignment is obtained by performing forced alignment with Kaldi [23], and the aligned is processed to align with rtMRI frames at 23.18 Hz. In our experiments, we use data from 8 speakers -M1, M2, M3, M4, F1, F2, F3, F4. Two speakers were left out for unseen speaker studies.

PROPOSED METHODOLOGY
We consider the problem of rtMRI synthesis as a sequence-tosequence problem with input text sequence of length n and output rtmRI video of shape n x 3 x 64 x 64. As shown in Fig. 1, we use a transformer architecture to represent the input phoneme sequence. Transformers are an ideal choice since they have been shown to perform well in learning text features, especially in multi-modal scenarios [15,16]. We then use a custom convolutional decoder consisting of 2D and 3D convolutional neural networks (CNN) to represent the frames. The CNN decoder is designed to upsample the representation to the image size while maintaining the temporal context learnt from the transformer encoder. In the following sections, we go into detail about the encoder and decoder architecture.

Transformer Encoder:
We use 12 transformer layers as the encoder for the sequence-tosequence model. The input phoneme labels are vectorised with a learnable embedding which is then given to the transformer. Transformer layers start with a self-attention layer. In self-attention, three learnable projections of the input called key, query and value are obtained. The dot product is taken between the key and query, followed by softmax activation. This acts as the attention matrix which learns dependencies between phonemes. This is then multiplied by the value matrix, which results in learning the representation of the utterance. After self-attention, the results are passed through feed-forward networks to combine the information learnt in the selfattention operation. There are additional features such as residual connection, layer normalisation and positional encoding which make the transformer a powerful representation learner. The final transformer layer produces the features with D dimension, which is used to extract frame-level features in the decoder.

Convolutional Decoder:
With the output of transformer encoder, we first rearrange the features as n x d x 8 x 8 which can be considered as feature maps representing low level frame features. This is then passed through two 3D convolution layers with residual connections (ResBlock) to represent the temporal information learnt from the transformer after being reshaped as an image. Each ResBlock contains two sets of 3D convolution layers with a kernel size of 3, batch normalisation and ReLU activation. 3D CNN learns over the sequence, with which it learns to retain temporal information. The ResBlock output is then upsampled with two transposed convolution layers, which is represented as Tconv in Fig. 1. Finally, two more ResBlocks are used to maintain the temporal information at the output and reduce the number of channels to 3 as present in the image. These series of 2D and 3D convolutional layers act as our decoder capable of producing realistic rtMRI frames while retaining temporal context from the input phoneme sequence. We label this sequence to sequence model as s2s.

Variational prior:
In our experiments, we found that the s2s model works well in most cases but fails to produce realistic frames in subject M1. In order to avoid such issues and provide context for the model to learn the multi-modal mapping, we utilise features from conditional variational autoencoder (CVAE) [24] as shown in Fig. 2. CVAE is used as a lightweight convolution-based generative model, trained on frames of rtMRI, with the VAE latent space conditioned on the phoneme corresponding to the frame. It is optimised with the ELBO loss as well as reconstruction loss on frames [24]. This allows us to conditionally sample frames from phoneme labels. For all the phonemes in the s2s model, we extract the representation from the 3nd decoder layer (denoted as F in Fig. 2) of CVAE in the sampling process, and these features are concatenated after the first 3D CNN block in the decoder of s2s, as shown in Fig. 1. We label this variant of s2s as s2s-v. Training is done by conditioning latent space with phoneme labels. During inference, given phoneme and sample from the normal distribution, a frame is synthesised. An intermediary feature F is used for s2s-v.
It is well known that VAEs tend to produce blurry images. Also, while the predicted images from CVAE might represent the shape expected for a phoneme, it will not be able to predict the rtMRI frame with context from neighbouring phonemes. Due to these reasons, we make a design choice to extract an intermediary representation in the sampling process from CVAE, rather than the output image. These extracted features denoted by F, are concatenated in s2s-v as shown in Fig. 1.

EXPERIMENTAL SETUP
We use data from 8 subjects (4 male, 4 female) and train subjectspecific models. We use 80% of data for training, 10% for validation and the remaining 10% for testing. The sentences used for all the splits are common across subjects. Hence, we train 8 models and evaluate on the same speaker, unseen sentence configuration. We resize all images to 64 x 64 to simplify the upsampling steps in decoder and normalise them between [0,1]. The input phoneme sequence is aligned with rtMRI frames such that there is a one-to-one mapping between the two modalities.
For training CVAE, we consider the data at the frame level and shuffle all the frames within the training and validation set. The phoneme aligned to each frame is used as the conditioning in the latent space of VAE as shown in Fig. 2. We use variational and reconstruction loss to train the VAE and train it only for 5 epochs since it converges at a very fast pace and is able to generate good frames within this time period.
For training the sequence-to-sequence model, we train on full videos, with a batch size of 1. This is performed since the memory usage of a single video can be significant and training one sequence at a time avoids zero padding. The transformer encoder and Fig. 3. Figure represents the 3 masks used for ATB segmentation from rtMRI frames, the first row corresponds to a frame from a real rtMRI video, and the second row corresponds to the predicted frame from s2s model for the same utterance in subject F4. (a) mask1predicted mask from both frames for the region above the palate. (b) mask2 -predicted mask from both frames which capture the tongue, jaw etc. (c) mask3 -predicted masks from both frames in the region of the pharyngeal wall. dice scores between the frames are shown at the top. the 3D ResBlock learn as the batch size of 1, whereas, the 2D transpose convolution (TConv) responsible for upsampling, views the sequence length as a batch, then learns and upsamples over independent frames. The final 3D ResBlock again acts on a batch size of 1, applying 3D convolution on the sequence. We use a Mean Squared Error loss for s2s and s2s-v models. All models are trained with Py-Torch, and the Adam optimizer is used with a learning rate of 0.0001. The checkpoint is stored based on early stopping criteria, i.e. when validation loss does not decrease after a few epochs. For evaluating the synthesized video, we use objective as well as subjective metrics.

Objective evaluation:
We train a SegNet [18,25] from ground truth frames to air tissue boundary (ATB) masks. We use the SegNet variation from [26] which has one encoder and three decoders, which predicts three ATB masks in a single model. The SegNet model is trained on individual rtMRI frames, from nine rtMRI videos from each subject after resizing the data to 64 x 64 to match the s2s shapes. It is optimised with cross-entropy loss over the mask labels. We first train a pooled model combining data from 10 subjects, then fine-tune it on the frame-mask pair of each subject. Using this fined-tuned subjectspecific model, we predict the masks from real rtMRI video and s2s, s2s-v model predictions. We then compute the dice score between the masks from real videos and the masks from the proposed method. Dice loss has been previously used to improve the segmentation of ATB masks [26]. Dice score is a similarity coefficient which quantifies spatial overlap and provides a score between [0,1], with 1 being an exact match. This objective is an indicator of the accuracy of positions and shapes of various articulators in the synthesised videos. Fig. 3 represents the three different ATB masks on which the score is reported. We compute the dice score across all the frames in an utterance, and then take the average across all the utterances in the test set of a subject. With this setup, we report the objective evaluation scores for all subjects, for 3 ATB masks shown in Fig. 3.
Subjective evaluation: With subjective evaluation, we aim to measure the capability of the models to produce realistic rtMRI video on the full sequence, when compared to the actual videos. We also look at identifying the need for s2s-v variation of the s2s model. Hence, for the sub- jective test, we show the ground truth reference for an utterance and as well as the videos produced by s2s and s2s-v for the same utterance. The evaluator has the select the video which resembles closely to the ground truth. Along with it, the evaluator also rates the selected video on a scale of 1 to 5, with 1 being of poor quality and 5 reflecting excellent quality. The test was conducted on an interface built with Jupyter notebook in python. The evaluator could play any video, any number of times. They were encouraged to focus on the movements of various articulators and score accordingly. We performed this test with 10 evaluators, each of them scoring on the synthesis of 5 random sentences from the test set for each speaker. GitHub codes for experiments and generated samples are available at https://github.com/bloodraven66/text to rtmri synthesis.

Subjective evaluation
As shown in the table, we conduct two types of subjective tests. Given the reference video identified as a reference, and provided with samples from the 2 models, the evaluator has to choose the sample which resembles the reference closely. Further, the evaluator rates the chosen sample. From Table 1, we can observe that subject M1 s2s-v is heavily favoured. For other subjects, we find a fairly consistent preference for both models. Looking at the mean opinion score, we observe that M1 has a lower score of 3.39 compared to the rest of the speakers, and this score was possible with the s2s-v model. Apart from this, we have MOS score greater than 3.67 for each of the 7 other speakers. With these numbers, we can infer that both models can be fairly consistent and produce realistic videos. Additionally, s2s-v models are beneficial when s2s models are not able to produce realistic frames. Table 2 represents the dice score evaluation on the 3 masks for both s2s and s2s-v models compared against the ATB predictions from the real rtMRI samples. We can observe that for subject M1, there is a slight increase in the score in all cases, with s2s-v dice scores being higher. The difference in the values in M1 between the two models is higher compared to other subjects. This suggests the preference given to M1 in subjective scoring. Overall, all values are greater than 0.89, suggesting good overlap between the masks of ground truth videos and predicted rtMRI frames which indicate that the synthesised frames have realistic shapes of different articulators. Fig. 4 shows the reference as well as predicted rtMRI frames from both the s2s and s2s-v models. We can observe the movement of specific articulators such as velum based on the spoken phoneme. We can observe both the s2s and s2s-v model results are extremely similar, indicating s2s models are good enough in most cases and s2s-v is useful when s2s fails. , showing the frames from phonemes. Row 1 represents the reference video, row 2 is the seq2seq model prediction, row 3 is seq2seq-v prediction. We can observe the lowering of velum due to nasal sound being produced.

CONCLUSIONS
In this work, we view rtMRI synthesis from phonemes as a sequenceto-sequence problem. We use a transformer encoder and convolutional decoder s2s to predict full sequence rtMRI video, on 8 subjects. We further condition the s2s model with latent features from phoneme-conditioned variational autoencoder(CVAE) in order to make the training robust. We evaluate with subjective and objective metrics based on air tissue boundary(ATB) segmentation. With the least MOS score of 3.396 and least dice score of 0.89, we show that the synthesised samples are of reasonably good quality and also show the benefit of having s2s-v modification. In the future, we plan to focus more on the objective metric, to be a more precise score, and also capture the errors in temporal continuity of different articulator contours from the synthesised videos.