{'fraction_non_alphanumeric': 0.029957375277661045, 'fraction_numerical': 0.01008584979287987, 'mean_word_length': 4.8737658674189, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Manner of Articulation Detection using Connectionist Temporal Classification to Improve Automatic Speech Recognition Performance

Conventionally, the manner of articulations in speech signal are derived using discriminative signal processing techniques or deep learning approaches. However, training such complex systems involves feature extraction, phoneme force alignment and deep neural network training. In our work, we initially detect the manner of articulations without phoneme alignment using an end-to-end manner of articulation modeling based on connectionist temporal classification (CTC). The manner of articulation knowledge is deployed in the conventional character CTC path to regenerate the new character CTC path. The modified manner based character CTC is evaluated on open source speech datasets such as AN4, LibriSpeech and TEDLIUM-2 and it outperforms over the baseline character CTC.


INTRODUCTION
Automatic speech recognition (ASR) has emerged as one of the active areas of technology with a lot of research and development efforts mainly in speech processing communities. The conventional acoustic models (AMs) of an ASR system followed a generative approach based on Hidden Markov Models (HMMs) [1] where the emission probabilities of each state were modeled with a Gaussian Mixture Model (GMM). Later, the performance of ASR has been improved dramatically by the introduction of deep neural networks (DNNs) as acoustic models [2] [3]. In the hybrid HMM/DNN approach, DNNs are used to classify speech frames into clustered context-dependent (CD) states (i.e., senones). On a variety of ASR tasks, DNN models have shown significant gains over the GMM models. However, the HMM-GMM or HMM-DNN pipelines are highly complex as it involves multiple training strategies (such as CI phones, CD senones), various parameters (such as dictionaries, decision trees) and We thank Tata Consultancy Services (TCS) for sponsoring the research under TCS-Research Scholar Programme. the performance depends on optimal choice of number of senones, Gaussians, etc.
More recent work has been focused on solutions which involves a simple paradigm which come close to end-to-end systems. On this aspect, Graves et al. [4] introduced the connectionist temporal classification (CTC) objective function to infer speech-label alignments automatically without having to rely on an alignment between the audio sequence and the symbol sequence. This CTC technique is further investigated in [5] [6] on large-scale acoustic modeling tasks. On the other hand, the attention-based method [7] exploits an attention mechanism to perform alignment between acoustic features and recognized symbols. However, the basic temporal attention mechanism is too flexible in the sense that it allows extremely non-sequential alignments. This is rational for applications such as machine translation where input and output word order are different [8]. However in phone attribute detection, the acoustic features and the corresponding outputs proceed in a monotonic way. Since CTC permits an efficient computation of a strictly monotonic alignment using dynamic programming, we propose to train a CTC-based manner of articulation detector to detect vowels, semi-vowels, nasals, fricatives and stop consonants.
Speech attributes were detected using discriminative features at the front-end and training the classifier [9]. Signal processing approaches are used for automatic and accurate detection of the closure-burst transition events of stops and affricates [10]. Later deep learning techniques were used to detect speech attributes [11]. However, training such complex systems involves feature extraction, phoneme force alignment and deep neural network training. Recently, Cernak et al., [12] exploited a solution to train a nasal sound detector without phone alignment using an end-to-end phone attribute modeling based on the connectionist temporal classification.
Recent success on nasal detection using CTC motivated us to extend their framework for detecting five broad manners of articulation namely vowel, semi-vowel, nasal, fricative and stop consonant. In the first part of our work, we extend CTC based nasal and non-nasal detection framework [12] to detect five broad manners of articulation. Later, we propose a mech-anism where the manner of articulation detection knowledge is deployed in decoding a speech utterance and study its impact in ASR performance. The main addressed question of this work was if it is possible to train a CTC manner detector without phone alignment and embed this knowledge in ASR decoding.

MANNER OF ARTICULATION DETECTION USING CTC
The character level transcripts are mapped to five different manners of articulation {V, $, N, F, S} that represents vowel, semi-vowel, nasal, fricative and stop consonant respectively. Figure 1 illustrates a toy example of obtaining manner of articulation labels from the character labels. The connection- ist temporal classification (CTC) approach [5] is an objective function that allows an end-to-end training without requiring any frame-level alignment between the input and target labels. It introduces an intermediate representation called the CTC path. The label sequence can be represented by a set of all the possible CTC paths that are mapped to it.
For an input sequence X = (x 1 , ..., x T ), the conditional probability P (z|X) is then obtained by summing over all the probabilities of all the paths that corresponding to the target label sequence y after inserting the repetitions of labels and the blank tokens, i.e., where φ(z) denotes the set of all possible paths that correspond to z after repetitions of labels and insertions of the blank token. The conditional probability of the labels at each time step, P (y t |x t ), is generally estimated using a RNN (LSTM/GRU). The model can be trained to maximize Equation 1 by using gradient descent, where the required gradients can be computed using the forward-backward algorithm [4]. In order to detect manners of articulation using CTC, the softmax output nodes are set to M +k where M is is the number of manners and k characters are added to include blank (<) or space (>). The manner of articulation detector using CTC is shown in left part of Figure 2. The bottom of the network is two layers of convolutions over both time and frequency domains. Temporal convolution is commonly used in speech processing to efficiently model temporal invariance for variable length utterances. Convolution in frequency attempts to model spectral variance due to speaker variability and it has been shown to further improve the performance [13]. Following the convolutional layers are bidirectional recurrent layers. After the bidirectional recurrent layers, a fully connected layer is applied and the output is produced through a softmax function computing a probability distribution over the target labels blank, vowel, semi-vowel, nasal, fricative, stops, space. The model is trained using the CTC loss function. To accelerate the training procedure, Batch Normalization [14] is applied on hidden layers.

CHARACTER CTC DERIVED FROM MANNER OF ARTICULATION CTC
We separately train manner of articulation CTC detector and the conventional character based CTC detector. Figure 2 shows the overview of obtaining character CTC by combing the manner of articulation posteriors and the baseline character posteriors. The basic version of obtaining manner based character CTC is shown in Algorithm 1. The modified character index is initialized with blank labels (line 4) and the previous most probable character index is initialized with space index (5). Then, we find the frame level index by calculating the most probable manner of articulation portions obtained from posteriors manner, (postM anner) (6). We find the non-blank index,nBInx in the manner CTC and their start and end frame durations, nBSeg (7). The algorithm iterates over all non-blank portions obtained from the manner CTC. We iteratively find the index of the most probable character segment by forcing the blank probabilities to zero (9). This is to ensure that the nonblank segment derived from manner CTC always emits a nonblank character. Later, in the non-blank manner segments, we find the most probable characters and are sorted according to the frequency of occurrence (sort f ) within the manner alignments (10). In the next phase, we find the characters excluding the previous index so that a new symbol is generated. We compare the current character generated for a manner peak with the previous character index and is ensured that the current character is always new as compared to the previous character (11)(12). The current index now acts as a previous index charList ← sort f (Ch ∈ prevInx) 11: currInx ← charList [1] 12: modCharInx(nBSeg) ← currInx 13: prevInx ← currInx 14: modChStr ← sym2Char(modCharInx, labelsChar) 15: return modChStr and the process is repeated for all frames (13). After the last time-step, the best character indices are converted to character symbols and is returned as a result (14)(15). The basic idea of the proposed method is to observe the non-blank peaks in the manner CTC and it is forced to emit a non-blank symbol thereby some of the characters missed out from blank dominating probabilities are nullified.  Fig. 3. (a) Manner of articulation predicted labels using manner CTC detector (b) Most probable manner index at frame level (c) New character index initialization to blank symbols (d) Sorted character indices obtained using character CTC detector for each frame (e) Most probable character index using state-of-the art character CTC detector (f) Character indices derived from manner of articulation CTC detector manners of articulation. We denote the most probably predicted blank symbols in black color and space character by dotted rectangle for illustration. We consider only those portions where the manner CTC detector emits non-blank sym-bols. The most probable manner index at frame level is shown in Figure 3(b). We initialize the manner based CTC characters to blank symbols as shown in Figure 3(c). Figure 3(d) shows the sorted list of character indices for each frame based on character posterior probabilities. They are generated using baseline character CTC detector. We denote the characters C 1 and C 29 for representing blank and space characters respectively for illustration and C 2 -C 28 for representing non-blank characters. The greedy search on these character probabilities picks up the most probable character index at each frame and it is represented in Figure 3 (e). The decoded sequence generated using best path decoding or greedy search is C 4 C 29 C 10 .

Illustrative Example
We find the start and end frame index of manner CTC peak. If the manner CTC emits a non-blank symbol, then the character CTC is forced to emit a non-blank character. This can be achieved by iteratively finding the most probable element in decreasing order of the posteriors probabilities and finding the optimal character by sorting the characters according to their frequency of occurrence in each segment. For instance m 1 is the most probable manner index and the corresponding character index is c 1 which is a blank symbol. So we find the second maximum character indices and sort them according to their frequency of occurrence and hence emit a single non-blank symbol. We will also compare the current index emitted with the previous index and if both are same then we emit different index by choosing appropriate symbol. The decoded sequence generated using proposed manner based character CTC is C 3 C 4 C 29 C 10 C 9 .

EXPERIMENTS 4.1. Data
We used three open source databases for training the manner and character CTC systems: (1) AN4 1 -the database contains alpha numeric speech data having 948 training and 130 test utterances. The dataset provides a good sample to achieve deterministic results to scale up with larger datasets. (2) Lib-riSpeech 2 -the data are sampled at 16 kHz, and the training part of the corpus are with size approximately 100, 360 and 490 hours respectively. In our experiments, we use 100 hours train-clean corpus. (3) TEDLIUM-2 3 -the English-language TED talks, with transcriptions, sampled at 16kHz. It contains about 118 hours of speech.

Training
The training phase is based on the open source DeepSpeech 4 network [15], trained with the CTC activation function. The manner detector using CTC starts with two layers of 2D convolutions over both time and frequency domains with 32 channels, 41 × 11, 21 × 11 filter dimensions, and 2 × 2, 2 × 1 stride. Four next bidirectional gated recurrent layers with 400 hidden units are followed by one fully connected linear layer with 7 softmax outputs {blank, , vowel, semi − vowel, nasal, f ricative, stop, space}. The GRU models have around 4.1 millions (M) parameters. The input sequence are values of spectrogram slices, 20 ms long, computed from Hamming windows with 10 ms frame shifts. The output (target) sequence was obtained directly from the letters of the word transcription. We used 50 epochs to train all the models used for further evaluation.

RESULTS
We evaluated both character and manner CTC detectors on the test-clean data set. The manner of articulation error rate (MER) is used to measure the performance of the manner CTC detector. The calculation of MER is similar to that of character error rate where the manner based reference and the obtained transcripts are compared. The reference transcripts is initially changed as per Figure1 . Table 1 shows obtained MERs of the CTC manner of articulation detector obtained on three datasets. and the character error rate (CER) obtained using the baseline and the proposed method. The pre-trained manner of articu- lation models and the baseline CTC models trained with AN4 dataset is made as an open source code 5 . It is observed that the manner of articulation knowledge in modifying the CTC path has significant impact in improving the performance of ASR.

Discussion
The CNN in the used model performed 2D convolution, where the first dimension is frequency and the second dimension is time. A longer stride is usually applied to speed-up training. Using the stride in the time dimension results into time compacting of the input audio, e.g., using the stride of 2 results into 2 times less frames of the output. For applications where time alignment is required, we experimented with the stride of 1 and we observed the convergence in training. Figure 4 shows an example of 5 https://github.com/Pradeep-Rangan/Manner-of-Articulation-Detectionusing-CTC Speech Frame the label index generated on manner CTC, baseline character CTC and the modified manner based character CTC detector. The speech utterance is from AN4 test dataset (test/an4/wav/cen8 − f caw − b.wav) whose content has the sentence "ELEVEN TWENTY SEVEN FIFTY SEVEN". The most probable manner index is derived from the posteriors manner as shown in Figure 4 (a). On top of the figure we illustrate some of the text transcript portions. The baseline character CTC as shown in Figure 4 (b) generates "E NEN TWENTY SEVEN FIFTY SEVEN" leading to false insertion and substitution errors. The non-blank portions in the manner peaks are forced to generate non-blank characters derived from character CTC detector using proposed method. The manner of articulation based character CTC generates the optimal character labels as per the peakiness in the manner portions. Figure 4 (c) shows the modified character index . The decoded sequence obtained using proposed method is : "EREVEN TWENTY SEVEN FIFTY SEVEN". It can be observed that the additional space that was generated using baseline method is nullified using the proposed method. Also the blank character propbabilties that dominated to miss out the substring 'EVEN' is recovered. The generation of the character 'R' may be due to the fact that the manner of articulation has semivowel. The probabilty of occurance of L is less than that of R.

CONCLUSION
This paper has proposed to use the connectionist temporal classification for the end-to-end manner of articulation modeling. The manner of articulation knowledge is deployed in the conventional character CTC path to regenerate the new character CTC path. The modified manner based character CTC is evaluated on open source speech datasets such as AN4, LibriSpeech and TEDLIUM-2 and it outperforms over the baseline character CTC. Application of the proposed manner of articulation CTC detector in weight adaptation of baseline end-to-end ASR training is also planned for future work.