{'fraction_non_alphanumeric': 0.03134201400248642, 'fraction_numerical': 0.012497546293267029, 'mean_word_length': 4.782822550132425, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Data Augmentation with Locally-time Reversed Speech for Automatic Speech Recognition

Psychoacoustic studies have shown that locally-time reversed (LTR) speech, i.e., signal samples time-reversed within a short segment, can be accurately recognised by human listeners. This study addresses the question of how well a state-of-the-art automatic speech recognition (ASR) system would perform on LTR speech. The underlying objective is to explore the feasibility of deploying LTR speech in the training of end-to-end (E2E) ASR models, as an attempt to data augmentation for improving the recognition performance. The investigation starts with experiments to understand the effect of LTR speech on general-purpose ASR. LTR speech with reversed segment duration of 5 ms - 50 ms is rendered and evaluated. For ASR training data augmentation with LTR speech, training sets are created by combining natural speech with different partitions of LTR speech. The efficacy of data augmentation is confirmed by ASR results on speech corpora in various languages and speaking styles. ASR on LTR speech with reversed segment duration of 15 ms - 30 ms is found to have lower error rate than with other segment duration. Data augmentation with these LTR speech achieves satisfactory and consistent improvement on ASR performance.


INTRODUCTION
Perception and recognition of speech requires tremendous amount of processing in human brain. Cross-disciplinary studies investigated the relation between speech sound and human behaviour. To understand how human listeners response to different speech stimuli, unintelligible speech is often used. Speech reversal refers to the operation of reversing a speech signal, which renders an audio signal that may contain unintelligible speech. In studies related to irrelevant speech effect (IFS), [1], time-reversed speech was found to affect short-term memory of human listeners. In neuro-physiological research [2], when subjects listened to time-reversed speech, observation on neural responses revealed that human brain discriminates speech stimuli based on acoustics instead of linguistic features. In relation to speech technology, time-reversed speech was applied to train speech enhancement model [3] to improve the quality of enhanced speech.
While time-reversed speech is generally unintelligible, speech intelligibility, referring to percentage of correctly recognised syllables or words, was found to withstand time reversal performed in local segments [4]. This is known as locally-time reversed (LTR) speech. Let a 5-second speech utterance be divided into 250 segments of 20 ms in length. If the samples in each segment are reversed in time order, listeners still consider the manipulated speech highly intelligible. Cognitive restoration of LTR speech was further investigated in a multilingual study [5]. The speech intelligibility degraded with the duration of time-reversed segment increasing across different languages. The intelligibility remains at above 90% as the reversed segment duration is shorter than 50 ms, and drops to below 25% as the duration is longer than 100 ms. This observation was in agreement with other monolingual studies on LTR speech [6][7][8][9].
Being analogous to human speech recognition (HSR), ASR aims to transform human speech into text. It could be cast as a task of pattern recognition. Motivated by human perception of LTR speech, we are interested in asking the following questions: how would a general-purpose ASR system perform on LTR speech? Apparently, local time reversal alters the spectral characteristics of natural speech. The alteration increases the discrepancy between training data of natural speech and test data of LTR speech. If reversed segment duration is relatively short, e.g. below 50 ms, acoustical characteristics of phonemes, syllable structure, word order, are largely preserved in LTR speech. We expect the natural and the LTR speech are similar. The ASR performance on LTR speech would be close to that on natural speech.
State-of-the-art ASR systems are predominately based on the data-driven deep neural network (DNN) approach. Using data augmentation strategies to escalate the quantity and variability of training data is effective to improve the performance of DNN models. Data augmentation does not require additional acoustic data, and is fairly easy to be implemented. Commonly used methods include vocal-tract length perturbation [10], speed perturbation [11], spectral masking, swap- ping and stretching [12][13][14]. Data augmentation with LTR speech has the potential to help improve ASR performance. The present study starts with an investigation on the effect of LTR speech on ASR. The performance of state-of-theart end-to-end (E2E) ASR systems on LTR speech is evaluated with speech corpora of different languages and speaking styles. The E2E design is chosen as it was shown to better match human performance than other predominant systems in psychometric experiments [15]. We find recognition on LTR speech with particularly short and particularly long reversed segment duration is prone to erroneous results. Based on the observation on ASR for LTR speech, we propose a speech data augmentation method, which combines LTR speech with natural speech in ASR training. Extensive ASR experiments are carried out to confirm the effectiveness of this strategy.

Speech corpora
ASR experiments are carried out on natural and LTR speech with four speech databases. They are TIMIT [16], WSJ [17], Aishell [18], Ted-lium2 [19]. The databases are chosen based on the consideration of language diversity, speaking style and amount of speech data. TIMIT is a read-speech corpus of English. The corpus contains phonetically-balanced utterances. It was widely used to support research on acoustic-phonetics and continuous speech recognition. WSJ is a read-speech corpus of English news, covering a large vocabulary size. The corpus was extensively used for benchmark evaluation of ASR systems. Aishell is an open-sourced corpus of Mandarin read-speech. The content covers a wide range of topics. Ted-lium2 is a corpus of spontaneous English speech. The speech was extracted from recordings of public talks, and the content was not well organized. Details of the four databases are summarised as in Table 1.

Locally-time reversed speech
The narrowing performance gap between ASR and HSR has motivated exploratory studies on the proximity of ASR and human auditory system [20,21]. Despite that some of the latest ASR systems outperform human performance on specific tasks [14,22], ASR performance is clearly inferior to human auditory system under challenging acoustic conditions, i.e. when input speech is clipped, spectrally modulated, bandpass filtered, or masked by noise [15,23]. The present study is focused on the effect of LTR speech on E2E ASR. LTR  speech is a type of temporally distorted speech. Local reversal with segment duration from 5 ms to 50 ms, in the step size of 5 ms, is applied to the speech data in TIMIT, WSJ, Aishell and Ted-lium2. In [24], the effect of LTR speech on ASR was investigated with the English corpus Ted-lium2 and the Japanese corpus CSJ [25]. LTR speech with segment duration of 25 ms, 50 ms, 70 ms and 100 ms were attempted. The ASR performance was found to degrade substantially when the reversed segment duration is longer than 50 ms. Figure 1 illustrates the spectrograms of natural and LTR speech of the same utterance. It is noted that the temporal order of phonemes is mostly preserved in the LTR speech. In the case of 5 ms local reversal, the acoustic properties of speech, e.g. pitch and formant contours, are locally altered. Local reversal in short segments tends to cause more noise in speech. This is related to the discontinuities of speech samples at the segment boundaries. Reversal in relatively longer segments, e.g. 50 ms, does not cause much noise, but inclines to disrupt the smooth transition between speech sounds.

E2E ASR system
The ASR model is implemented with an encoder-decoder architecture. It is trained toward two objectives, namely the connectionist temporal classification (CTC) and the attentionbased classification [26]. Tokens such as English/Chinese characters, phonemes and graphemes, are extracted from given speech transcriptions as the output labels (or symbols). CTC aims to map input acoustic features to output token sequence via the encoder. Assuming a ground-truth token sequence y has K distinct token labels, an intermediate token sequence π = (π 1 , ..., π T ) of length T is used to represent where y is the prolonged version of y with blank tokens inserted in-between ground-truth tokens, and at the beginning and the end of y. Φ(y ) denotes all possible sequences of y . The decoder predicts output token sequence in an autoregressive manner with the attention mechanism. Let y u be the u-th ground-truth token, the decoder loss is expressed as, The two loss functions are fused in a multi-task manner by, where λ ∈ [0, 1] is the task weight. In the decoder, the language model (LM) can be incorporated by shallow fusion. The ASR system aims to determine the best token sequencê y as, where Y * denote the complete set of hypotheses. α and β are hyper-parameters that control the contributions of CTC, attention and language model (LM).
Training of the E2E models is implemented with the ESP-NET toolkit [28] with the Adadelta optimizer [29]. The input acoustic features are 80-dimensional Mel-scale filter-bank (Fbank) coefficients with mean and variance normalised. The model architectures and training parameters are summarised as in Table 2. Recurrent neural network LMs (RNNLMs) are  Table 3. In addition, a 4-gram LM is trained on Aishell and jointly used with the RNNLM.

ASR performance on locally time-reversed speech
The ASR systems are first trained on natural speech and used to decode input utterances of LTR speech. The segment duration for local reversal varies from 5 ms to 50 ms. The recognition performance is measured in terms of phone error rate (PER) in TIMIT, character error rate (CER) in Aishell, and word error rate (WER) in WSJ and Ted-lium2. Except the TIMIT phone recognition task, shallow fusion with RNNLM is applied in the decoding. The results are shown as in Figure 2. It must be noted that many previous studies on LTR speech [5][6][7][8]24] assumed minimum segment duration of 20 ms. The effect of local time reversal in notably short segments, i.e. below 20 ms, has not been well understood. The error rate increases first from no reversal to reversal with 5 ms segment duration. When the segment duration further increases to 10 ms and 20 ms, the error rate drops. When the segment duration increases from 20 to 50 ms, the error rate goes up steadily. These trends are consistent among different corpora regardless of the model architecture. The error rate peak at 5 ms could be caused by the additional noise and changes of acoustic properties in LTR speech. The dissimilarity between training and test data deteriorates the recognition performance.
Different types of ASR errors are analysed for LTR speech and natural speech. In the case of TIMIT, there are frequent errors concerning nasal substitution (/m/ → /n/), and  For LTR speech, ASR is more prone to minimal-pair errors when the reversed segment duration is 10 ms or below. Two Chinese characters are said to be a minimal pair if they differ by a single phonological element. For instance, the two differ in tone, e.g. 'shi4' (city) and 'shi2' (time); or differ in one phoneme, e.g. Mandarin particles of 'le' and 'de'. More deletion errors are found in ASR when the reversed segment duration is 15 ms or above. The recognition performance and error analysis of LTR speech help understand the relation between LTR speech and general-purpose ASR systems. The results suggest that if appropriate reversed segment duration is chosen, the mismatch between natural and LTR speech can be alleviated.

Data augmentation with LTR speech for ASR training
Utterances of LTR speech are divided into 5 sets according to the reversed segment duration, as shown in Table 4. Each set represents different levels of intelligibility. As an attempt to data augmentation, each set of LTR speech utterances are combined with natural speech in ASR training, creating a 3fold training set. Data augmentation with LTR speech is compared with speed perturbation and SpecAugment, which are the most widely used strategies of data augmentation. With speed perturbation factor of 0.9, 1.0 and 1.1, 3-fold training set is created, and has the same quantity of training data as data augmentation with LTR speech. SpecAugment warps and masks the time and frequency domain of the input F-bank coefficients. It does not increase the size of original training data. The model training follows the configurations in Table  2. The systems trained on natural speech are regarded as the baseline systems. Recognition performance on natural speech are given as in Table 4.
Data augmentation with LTR speech improves the ASR performance over the baseline system on all the four corpora. This confirm the feasibility and efficacy of deploying LTR speech in ASR training. For each corpus, the best performance attained by data augmentation with LTR speech is comparable to data augmentation with speed perturbation. Data augmentation with LTR speech outperforms SpecAugment on TIMIT, WSJ and Aishell, but not on Ted-lium2.
When shallow fusion with LM is applied, using LTR speech with reversed segment duration shorter than 30 ms (Set 1-3) performs better than with longer segment duration (Set 4-5) on WSJ, Aishell and Ted-lium2. Without shallow fusion, more satisfactory performance is obtained via LTR speech with reversed segment duration of 5 ms and 10 ms (Set 1). For TIMIT, the best result is achieved by using LTR speech reversed segment duration of 15 and 20 ms (Set 2). Consistent improvement is observed in using LTR speech with reversed segment duration of 15 ms -30 ms (Set 2-3). These results are in agreement with Figure 2, in which the error rates are comparatively low. LTR speech with this range of reversal contains less noise, and retains smoother transition of speech sounds than longer segment duration. Local reversal with 15 ms -30 ms is recommended for efficacious data augmentation with LTR speech.

CONCLUSION
The effect of LTR speech on E2E ASR performance, and the feasibility of using LTR speech in ASR training are studied in this paper. Using general-purpose ASR systems trained on natural speech, recognition on LTR speech with relatively short and long reversed segment duration is prone to erroneous results. Data augmentation with LTR speech is confirmed to improve ASR performance on natural speech. Using LTR speech with reversed segment duration of 15 ms -30 ms consistently improves ASR performance on corpora of different languages and speaking styles. In future work, the intelligibility of LTR speech with short reversed segment duration, e.g. below 20 ms, will be studied via subjective rating. The use of LTR speech will be generalized to other topics in speech technology, such as speaker verification, speech enhancement and self-supervised learning of speech representation.