{'fraction_non_alphanumeric': 0.03476960821896803, 'fraction_numerical': 0.024419228705052518, 'mean_word_length': 4.590870124303472, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
SSR7000: A Synchronized Corpus of Ultrasound Tongue Imaging for End-to-End Silent Speech Recognition

This article presents SSR7000, a corpus of synchronized ultrasound tongue and lip images designed for end-to-end silent speech recognition (SSR). Although neural end-to-end models are successfully updating the state-of-the-art technology in the field of automatic speech recognition, SSR research based on ultrasound tongue imaging has still not evolved past cascaded DNN-HMM models due to the absence of a large dataset. In this study, we constructed a large dataset, namely SSR7000, to exploit the performance of the end-to-end models. The SSR7000 dataset contains ultrasound tongue and lip images of 7484 utterances by a single speaker. It contains more utterances per person than any other SSR corpus based on ultrasound imaging. We also describe preprocessing techniques to tackle data variances that are inevitable when collecting a large dataset and present benchmark results using an end-to-end model. The SSR7000 corpus is publicly available under the CC BY-NC 4.0 license.


Introduction
A silent speech interface (SSI) (Denby et al., 2010) enables us to speak or use voice interfaces without uttering an audible sound. The essential purpose of SSI is to expand the range of applications of voice interfaces in computing. Voice interfaces (Porcheron et al., 2018;Seaborn et al., 2021) based on automatic speech recognition (ASR) are intuitive interfaces that most people can use without training. It is like asking someone else to do things for us. However, the intrinsic nature of vocalization presents various constraints. For example, it is difficult to use in noisy environments. Caution is also warranted when handling information that may jeopardize privacy or confidentiality. The SSI removes this limitation by enabling non-voice interactions. It also allows communication for users in hands-busy settings or for those with a low voice or no voice due to tracheostomy, amyotrophic lateral sclerosis (ALS), or dysarthria.
The key technology for implementing SSI is silent speech recognition (SSR). Silent speech is defined as involving only articulatory movements without vocalization or the use of vocal cords. Traditionally, sensors such as surface electromyography (Maier-Hein et al., 2005;Kapur et al., 2018), electroencephalography (Porbadnigk et al., 2009), a front camera for lip reading (Wand et al., 2016;Assael et al., 2016b;Sun et al., 2018), and ultrasound imaging (Kimura et al., 2019;Cai et al., 2011;Ji et al., 2018a) have been used for SSR. Among these, ultrasound imaging is superior as a non-invasive and safe means of obtaining detailed images of the body, as it is also used during pregnancy (Denby et al., 2010). It can capture tongue movements, which play a vital role in articulation. In addition, the ultrasound probe, the sensor for ultrasound imaging, can be flexible and miniaturized (approx. 1 cm × 2 cm). These are essential factors for future use in wear-able applications. Several studies have focused on SSR based on ultrasound tongue imaging (UTI), and the current state-ofthe-art (SOTA) method (Ji et al., 2018b) for this task uses the cascaded DNN-HMM model of speech recognition (Ji et al., 2018a). On the other hand, in the field of ASR, end-to-end models Chiu and others, 2018) based on connectionist temporal classification (CTC) (Graves and Jaitly, 2014) or attentionbased encoder-decoder (Chan et al., 2016) have become mainstream due to their significantly better performance for large speech corpora. Some of these techniques have also been adopted in the field of lip reading (Assael et al., 2016a;Afouras et al., 2018), which is similar to silent speech recognition tasks, and have achieved SOTA performance with large-scale datasets (Chung and Zisserman, 2016;Alghamdi et al., 2018).The emergence of large datasets has attracted many researchers to the field of lip reading and accelerated research in this area. However, the benchmark dataset of the UTI-based SSR, the Silent Speech Challenge (SSC) dataset (Denby et al., 2013), is relatively small compared with the corpora used for other speech recognition tasks. It is therefore not suitable to exploit performance from end-to-end models. In this study, we constructed SSR7000, a large-scale corpus of synchronized ultrasound tongue and lip images designed for end-to-end UTI-based SSR. Our dataset comprises approximately 7484 UTI and lip images of silent speech by a single native speaker of English. Table 1 presents a comparison of SSR7000 with other corpora for UTI-based SSR. Our dataset is characterized by a large sample size for a single speaker and a realistic variance among samples, assuming the stories using end-to-end models. The SSR7000 appears to be an extension of SSC (Denby et al., 2013). It shares the same number of speakers and part of the Table 1: A comparison of UTI-based corpora. Our SSR7000 corpus is characterized by the maximum number of utterances per person. It is approximately three times bigger than SSC (Denby et al., 2013). The TaL (Ribeiro et al., 2021) (Denby et al., 2013). This is primarily due to the fact that we did not perform a strict calibration to suppress variance, unlike SSC, for each collecting session, considering that calibration will be a barrier when collecting samples even larger than the SSR7000 or when collecting from multiple speakers in a future study. Additionally, for the application used in wearable computing, calibration is not practical for each instance. SSR7000 also provides a preprocessing challenge to reduce data variance, and this paper presents a benchmarking method for this. The Ultra-Suite repository (Eshky et al., 2018) contains ultrasound and speech data from 58 children with normal development and from 28 children with speech disorders receiving speech therapy. The most recent TAL corpus (Ribeiro et al., 2021) consists of TaL1, a set of six recording sessions of one male native English speaker who is a professional voice talent, and TaL80, a set of 81 recording sessions of a male native English speaker with no professional voice talent experience. The TaL corpus is similar to SSR7000 in that it uses the same fixing device and recording software. However, the TaL corpus is not strictly a silent speech corpus (participants uttered voice), as it is also intended for use in articulatory-to-speech mapping (Hueber et al., 2011;Porras et al., 2019), language learning (Wilson and Gick, 2006;Gick et al., 2008), and phonetics research. Our dataset contains the largest number of utterances per speaker.
Our main contribution is as follows: 1) we have designed and constructed a new dataset "SSR7000" for SSR using end-to-end models, 2) we describe a strategy for recording a large-scale SSR dataset and preprocessing techniques to handle data variances, and 3) we present benchmark results using an end-to-end ASR model. In Section 2, we delineate the data collection method, the characteristics of the data, the preprocessing techniques based on the data properties. In Section 3, we demonstrate how to extract features from the preprocessed data and benchmark the recognition of these features using ESPnet (Watanabe et al., 2018), a speech recognition toolkit. The raw image data, the preprocessed data, and the feature extracted data of the dataset have been packaged and made public. The recognition part of the dataset is available in a form that anyone can reproduce using Google Colab 1 .

Dataset Collection
Our SSR7000 corpus is a recording set consisting of 7484 utterances by a single male native English speaker. In this paper, we split the dataset into 7384 training data (100 for validation) and 100 testing data. All utterances were recorded in a silent manner, where the participant did not speak aloud but only moved his articulatory organs. We used an UltraFit system (Spreafico et al., 2018) ( Fig. 1) for data acquisition.
The system is comprised of a 3D-printed adjustable helmet housing a convex-array ultrasound probe (opening angle: 104°, frequency range: 5-10 MHz, and piezo elements: 128) to the chin of the participant and an NTSC micro-camera for capturing from the front so that the participant's lips were entirely visible in the image. The Articulate Assistant Advanced (AAA) software (Articulate Instruments Ltd, 2021) was used to record and synchronize the dataset. Ultrasound images were recorded with a field of view of 92 degrees, outputting videos with a resolution of 640 × 445 pixels at 63.51 fps. Lip images were recorded using the micro-camera, outputting videos with a resolution of 640 × 480 pixels at 59.94 fps (greyscale interlaced). The two video streams were synchronized using the SynchBrightUp unit, which is triggered by an audio beep that superimposes a white mark on the video signal and generates a pulse on the audio channel, thereby aligning the first few frames. Given that the fatigue of the participant could affect the articulation, we limited the collection time to 1 h per session and 3 h per day. We also avoided collecting data for more than 3 days in a row. In addition, we removed the equipment every time we took a break. Between sessions, we did a simple check to ensure that the tongue and lips were visible to the sensors (camera and probe). The SSC (Denby et al., 2013) data collection included a recalibration process to adjust the tongue and lip positions interactively using modules provided by Ultraspeech for each session to make the positioning consistent. Consistent positioning with strict recalibration is important to achieve high accuracy on the test set, but we omitted it based on the story the SSR7000 supposes, the large dataset for end-to-end models. For this reason, SSR7000 has a larger variance than SSC. This can be observed in the difference in clarity between the SSR7000 "average faces" (Fig. 4-A) and the SSC "average faces" (Fig. 6). This is an important property difference between SSC (Denby et al., 2013) and SSR7000.
Figure 1: UltraFit stabilizing helmet, which fixes the video camera in front of the participant and fixes an ultrasound probe to the chin of the participant. The same system was used for UltraSuite (Eshky et al., 2018) and TaL (Ribeiro et al., 2021).
For the recording prompts, we chose the TIMIT corpus (Garofolo et al., 1992) as the SSC (Denby et al., 2013) because it includes phonetically balanced 2342 sentences and is suitable for training data. To further extend this, 5042 new sentences were selected from wsj0 (Garofalo et al., 2007) corpus, adding up to 7384 sentences (50 sentences from each corpus are used for validation). For the test set, we selected the same 100 sentences as SSC from the wsj0 (Garofalo et al., 2007).
The sentences for the test data are fully independent of the 7384 sentences in the training and validation data. Since the scripts from the WSJ corpus are generally longer than those from TIMIT, we set a maximal duration of 12 s for the recordings compared to the 8 s used in the SSC dataset (Denby et al., 2013). All 7484 sentences were captured in approximately 50 sessions. As mentioned above, the camera and the probe positions differed slightly for each session.

Ultrasound Tongue Images (UTIs)
Fig. 2 depicts samples of the captured ultrasound tongue image (UTI) sequences. UTIs were captured using a high-gain setting. As with a normal RGB camera, a high-gain setting on the ultrasound imaging probe will make the image brighter, while a low gain setting will make it darker. Since it is difficult to always guarantee the right gain setting in a large dataset, we used a high-gain setting to reliably capture the tongue throughout many sessions over several weeks. Although this high-gain setting resulted in substantial white noise, as the first row of Fig. 2 shows, in all sessions, we were able to avoid the worst-case scenario in which the gain was too low to capture the tar-get tongue. However, the second row of Fig. 2 indicate that when we applied feature extraction using discrete cosine transform (DCT), there was a minimal difference between the reconstructed images, meaning that DCT did not extract important features. Therefore, we designed a filter that removes white noise and emphasizes only the target tongue.
Figure 2: Ultrasound tongue images (UTIs) and reconstructed images using discrete cosine transform (DCT). The first row shows the raw images of the UTIs, which were captured at a high-gain setting and had white noise. The second row shows reconstructed images of those from the first row. The third row shows the UTIs after applying moving average filtering. The reconstructed images in the fourth row are increasingly distinguishable from those without filtering.

Filtering UTI
In still images, white noise is difficult to distinguish from the tongue. However, since white noise is inconsistent over time, it is easy to distinguish the two in video. Therefore, we set a high brightness threshold for each image and performed moving average filtering to emphasize the consistent capture of the tongue over time. Fig. 2 compares filtered (the third row) and raw data (the first row). Considering the frame rate of the ultrasound videos (60 fps), we set the slide window size to 5 and discovered that the noise reduced substantially. The images in the second and fourth rows are the reconstructed images extracted using DCT. In the filtered images, the white noise almost disappears, and the tongue features are emphasized. The fourth row in Fig. 2 shows that filtering succeeded in emphasizing the important DCT features.

Lip Images
In the top row of Fig. 3, raw lip images taken from different sessions are depicted. The lip positions changed observably in each session. Fig. 4. shows the "average face" from the training data, which is calculated by aggregating the first frame from each of the 7384 utterances and then dividing their sum by the number of utterances (7384). The "average face" in the first frame of the training data is highly blurred. This means that Figure 3: The visualization of cropping for lip videos. We detected ROIs in lip videos using a neural image tracking algorithm and cropped them out for better recognition results.
the lip positions varied and that there was a high variance in the raw data. When comparing average faces from the training data ( Fig. 4.A) to those from the test data ( Fig. 4.B), it is evident that the test data were not in the distribution of the training data.

Detecting Region of Interest and Cropping
Based on the above observations, we found it necessary to locate the lips and cut out the region of interest (ROI) to improve recognition results. To determine the ROI in the lip videos, we employed a deep-learning image tracking algorithm, GOTURN (Held et al., 2016), to estimate the position of the lips for each frame. For each of the lip videos, we first computed a video-level lip-bounding box by averaging the tracking results at each frame. Subsequently, the bounding box was resized to 80 × 120 and finally fine-tuned manually to ensure that the lips were located approximately at the center. Fig. 3 illustrates how the lip ROIs were detected and cropped out. Fig. 4.C shows the "average face" from the training data with the detection of the ROI and cropping, while Fig. 4.D shows that of the test data. The average training face became much clearer without ROI-cropping (A). The face from the training data (C) and the test data (D) also became more similar.

Dataset
The SSR7000 is publicly available 2 and it is the first to provide raw data without any preprocessing, which is useful for those interested in improving preprocessing. For those more interested in the recognizer rather than the preprocessing, we have provided the preprocessed data described in this paper. The corpus is publicly available under the CC BY-NC4.0 license.

Recognition Pipeline
Our recognition pipeline uses a hybrid CTC/attentionbased end-to-end ASR model . We implemented this model based on the VoxForge recipe of ESPnet (Watanabe et al., 2018) with some 2 https://github.com/supernaiter/ ssr7000 Figure 4: The "average face" of our SSR7000 dataset, which was calculated by adding up the first frame from each of the utterances, then dividing it by the number of utterances. A) shows the "average face" from the training data without cropping or preprocessing. B) shows that from the test data without preprocessing. C) shows that from the training data with preprocessing, and D) shows that from the test data with preprocessing. The more blurred the average face, the greater the variance in the data.
modifications. The detailed model architecture, parameter settings, and configurations are publicly available in the repository alongside the dataset. As indicated in Fig 5, we utilized the DCT features as inputs to the network. We used SpecAugment (Park et al., 2019) to apply temporal and frequency augmentation, which includes a random time warp (shifting the data sequence along the time axis) for up to five frames, two random time masks (replacing the data in a random time range with zeros) with a length of up to 40, and two random frequency masks (replacing the data in a random frequency range with zeros) up to a width of 5 for DCT 20, 10 for DCT 30, 30 for DCT 60, and DCT 120, respectively.

Training
The training was run on Ubuntu 18.04 with a GTX1080Ti GPU and converged in approximately 4 h. The model that had the highest accuracy on the validation set was applied to the test data and to calculate the error rates.  Figure 5: Our recognition pipeline using an end-to-end ASR model. Note that after the feature extraction step, we show images reconstructed from DCT features instead of DCT features themselves for visualization. The lip and tongue images are reconstructed using 30 DCT coefficients.

Results
We used word error rate (WER) and character error rate (CER) as metrics. While the phoneme level result has been indicated along with WER in past studies on SSR based on UTI, we show CER instead because our recorganizer uses characters as tokens. Table 2 shows a comparison of the preprocessing performed. Evidently, each preprocessing improved the results as expected. The improvements in preprocessing were substantial (7% with ROI-C, 4% improvement with filtering on both DCT-60 and DCT30 condition), while the change in the error rate was quite minimal when employing various E2E models. This suggests that the main focus of SSR7000 is preprocessing. In particular, ROI-cropping was done semi-automatically by OpenCV, so it can be expected to be greatly improved by aligning the ROI by hand or by inventing superior methods.  Table 3 shows the results of investigating the effect of the number of data on the recognition. For the experiments, 60 DCT features were used. Subsets of 1000, 3000, and 5000 data were randomly selected from all training data. We repeated the random sampling and training process several times to diminish the noise in the results. Overall, we can see a linear improvement in the error rate as the amount of training data increases. This supports our idea of increasing the number of data for the E2E model. The decrease in the error rate has not yet converged, suggesting that adding more data may increase the accuracy. Data augmentation of raw image data as well as SpecAug should be also effective.  Table 4 shows the variation in the error rate according to the number of dimensions of the features obtained by DCT. We first attempted 20, 30, and 60 dimensions, and found that the error rate tended to decrease as the number of dimensions increased. When we tried 120 dimensions, however, the error rate increased. Ji et.al (Ji et al., 2018a) reported that 30 dimensions is optimal for the SSC dataset, but when using a more expressive end-to-end model with a large amount of data, as in our current experiment, it is suggested that a larger number of DCT dimensions is appropriate. Based on this result, we set the DCT dimension to 60 in the other comparison experiments.  Table 5 indicates the results of the recognition experiments with lip images and UTIs alone (DCT-60 was used). The lip images and UTIs have been preprocessed respectively. As the "Lip and UTI" column shows, the two modalities were synergistic and had a better error rate than those of UTI or the lip images alone. The lip images alone had a good error rate of 22.7% CER and 46.1% WER. On the other hand, as in Table 5, UTIs alone had a high error rate;72.0% WER. However, considering that the UTIs had greater accuracy than the lip images when tested alone in the TaL corpus (Ribeiro et al., 2021), the gain might have been set too high. 4. Discussion

Comparison with SSC
Although the SSC dataset and the SSR7000 were created using different equipment and under different conditions, the test sentences are the same; thus, our results with SSR7000 can be fairly compared to that of previous work with SSC. We previously performed recognition tasks on the SSC dataset using the same pipeline and recorded an error rate of 10.1% CER and 20.5% WER (Kimura et al., 2020) (The best result 6.4% WER is reported by Ji et.al (Ji et al., 2018a)). This is about half the error rate of the SSR7000's best results of 17.6% CER and 37.6% WER, even though the SSR7000 contains roughly three times as much training data. We tried not only the hybrid ctc/attention model , but also the pure attention architecture with the same hyperparameters, and the former model produced the best result above. Therefore, the causes of discrepancy should lies before the recognizer. Fig. 6 which shows that the lip positions are quite consistent through training data, and it seems quite similar with that of test data. On the other hand, the SSR7000 average face shown in Fig. 4-A is very blurry. This is due to the fact that the lip position is different for each session, as shown in Fig. 3. The preprocessed image of the SSR7000 shown in Fig. 4-C is somewhat clearer than the raw image, but still blurrier than the SSC image (Fig. 6). This strongly depends on the quality of the calibration during the session; how rigorous the calibration is depends on the story the dataset is supposed to tell. The SSR7000 was intended to be the first model on a large dataset to exploit the performance of endtoend ASR models, so only a simple calibration was performed.
There is also room for improvement in the fixation devices for the ultrasound probe and camera. The 3D printed helmet-type fixation device used in this study could not hold the sensors in the same position for a long period of time, and the positions of the sensors moved even during the single session. If we can develop a fixation device that can be easily installed in the same position every time, t will help the calibration between sessions. Compared to the SSR7000, the SSC (Denby et al., 2013) has a wider angle lens positioned closer to the lips; the SSC is thereby able to successfully capture the frontal tongue movement in addition to the lip move-ment. The discrepancy between SSR7000 and SSC may propose to adopt SSC's style to capture lip images. Figure 6: "Average faces (lips)" of the SSC dataset.
That of the training data is clearer than that of SSR7000, and the training data and test data are surprisingly similar.

Preprocessing
As mentioned above, our dataset has a large variance in lip position, which introduces a challenge regarding the method to suppress it (ROI-cropping). We have shown a first benchmark using an existing algorithm provided by OpenCV (GOTURN (Held et al., 2016)). The lip tracing was done automatically, except for the manual adjustment of the few sessions. In order to get the best offline results, it would be useful to use a crowd worker to mark the coordinates of the corners of the mouth for all images. This allows normalization of rotations and size changes, which was difficult to do with GOTURN.

Feature Extraction
We used the discrete cosine transform used in Ji et al.'s work (Ji et al., 2018a) and TaL (Ribeiro et al., 2021) for feature extraction, but there is still room to experiment with various feature extraction methods; for example, principal component analysis is the first other method to consider. More recent methods, such as using an auto encoder and a variational auto-encoder using neural networks, are good candidates. The use of pre-trained weights (Feng et al., 2020) for the lip images established in the field of lip reading may also be useful.

Channel Attention
In this pipeline, the lip image features and the UTI features were just stacked and fed into the recognizer (for example, when using DCT-60, the stacked features were 120 dimensions). However, the lip image and the UTI should have different pronunciation strengths. For example, "p", "b", and "m" are not observable from the UTI, but can be inferred from the lip images. On the other hand, pronunciations that mainly use the tongue, such as "r" and "l", cannot be observed from outside the body, so the UTI is important. To reflect these characteristics in the recognizer, it may be effective to train different recognizers for each feature in advance and integrate them afterwards, or to incorporate a mecha-nism such as a channel attention module (Woo et al., 2018).

Conclusion
This paper presented a large dataset for the end-toend speech recognition model, SSR7000, which comprises 7484 silent speech utterances synchronized with UTIs and Lip Image. Among existing UTI-based corpora, our SSR7000 has the largest number of utterances per person. We also introduced a benchmark preprocessing method and included preprocessed images from the dataset. Silent speech recognition experiments using the E2E model of hybrid CTC/attention were performed and benchmarked. This model will be released with SSR7000. The model will be released together with the SSR7000 so that people who are interested in preprocessing, recognizers, or other techniques can try them without the difficulty of implementing the pipeline.