{'fraction_non_alphanumeric': 0.02844699266773251, 'fraction_numerical': 0.013010640057335024, 'mean_word_length': 4.75142675967026, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Language Recognition using Time Delay Deep Neural Network

This work explores the use of a monolingual Deep Neural Network (DNN) model as an universal background model (UBM) to address the problem of Language Recognition (LR) in I-vector framework. A Time Delay Deep Neural Network (TDDNN) architecture is used in this work, which is trained as an acoustic model in an English Automatic Speech Recognition (ASR) task. A logistic regression model is trained to classify the I-vectors. The proposed system is tested with fourteen languages with various confusion pairs and it can be easily extended to include a new language by just retraining the last simple logistic regression model. The architectural flexibility is the major advantage of the proposed system compared to the single DNN classifier based approach.


Introduction
Language Recognition (LR) refers to a machine based approach through which the identity of the language spoken in a speech sample is authenticated. In recent times, tremendous progress have been made in this regard, which is benefited from technological breakthroughs in related areas, such as signal processing, pattern recognition, cognitive science and machine learning. LR systems are usually categorized by perceptual cues they use, such as the acoustic-phonetic approach, the phonotactic approach, the prosodic approach and the lexical approach. However, recent studies have confirmed that acoustic-phonetic and phonotactic features are the most effective language cues and these are the features used typically in the mainstream LR technology [1] [2]. I-vector based approach have been extensively used for LR during the past few years. Such an I-vector system designed for LR task relies on two factors -a front end feature for acoustic representation of the speech signals and a back end model to capture the statistics of distribution of the phonetic events at acoustic feature level. Recent LR experiments have analysed and proved that the shifted delta cepstral (SDC) feature vectors created by stacking delta cepstra computed across multiple speech frames provides best performance in LR [3]. This is because they capture the dynamics over a wider range of speech frames than the first and second order MFCC derivatives. On the other hand typically Gaussian Mixture Model (GMM) based Universal Background Model (UBM) is used to collect sufficient statistics (SS) for such I-vector systems. I-vectors are used for scoring directly or used as input to some secondary classifiers.
However, recently Lei at el., 2014 [4] have proposed an efficient technique of using Deep Neural Network (DNN)s to compute SS for I-vector systems designed for Speaker Recognition (SR) task. Later a few other works are reported the using of such DNN as a means of computing posterior probability of feature vectors in SR task and observed significant improvement in performance [5] [6] [7]. Such improvements inspires this study to use an English DNN as an UBM to address the LR problem. I-vectors employ subspace techniques to model variations from the UBM. Hence our motivation is that if the DNN-UBM models a normal language, the I-vector shall focus more on the variations in the language. Usually such a DNN is trained for modeling acoustic events like tied triphone state or senones in a supervised way using labeled data. The classes corresponding to senone and posteriors are very accurately computed to predict the senone in such a DNN. Hence using the posteriors from DNN class, the I-vector system shall be able to differentiate the minute variations of phonemes occurred due to the change of languages.
The DNN architecture used for this work follows the structure of a Time delay DNN (TDDNN) developed by Peddinti et al., 2015 [8]. The architecture is capable to effficiently model the long term temporal dependencies between acoustic events in Automatic Speech Recognition (ASR) task. It uses a very efficient subsampling technique of selecting time steps to compute hidden activations which reduces computation during training. The TDDNN stucture provides advantage compared to the recurrent structures in terms of less computational time, while effectively learning the temporal dynamics of speech signal just like the recurrent structure.
The LR work reported in this paper trains the TDDNN models using around 1800 hours of the English part of Fisher data and uses the TDDNN to compute posterior probability of feature vectors for I-vector extractor training. The I-vector system is trained using data from around 50 different languages. Finally a logistic regression model is trained and tested on top of the I-vectors using the 14 languages of LRE 2007 evaluation. The results reported in this study are compared with a I-vector system which uses a GMM-UBM. However, the major adavantage of the proposed system is its architectural flexibility to include a new language. Keeping rest of the models as it is, just by retraining the simple logistic regression model a new language can be added to the system. However, in single DNN classifier based system adding a new language requires a retraining of the entire DNN model. The work also reported some intermediate results observed during the LR experiments like use of vocal tract length normalization (VTLN) warping and speaker recogntion style features for LR. The results are included for the case when VTLN warping is performed on the acoustic features and how the results varies when that step is removed. Further features used for speaker recognition, like 60 dimensional Mel Frequency Cepstral Coefficients (MFCC) augmented with delta and acceleration, is also used for the work and compared with the standard SDC.
The rest of the paper is organized as follows: Section 3 provides a description of the baseline system. Description of DNN-UBM based approach is given in Section 2. Experimental set up and related discussions are reported in Section 4. Section 5 concludes the description.

DNN-UBM based approach
A DNN trained as the acoustic model using English speech as part of an ASR, is used as background model for LR in this work. The posteriors of phonetic content like tied triphone states obtained from the output layers of such DNN is used for I-vector system.
In the I-vector model given by Dehak et al. [9], the total factor w is defined by the posterior distribution of the T feature vectors of utterence Y = y1, y2, · · · , yT conditioned to the following Baum-Welch statistics, where, c is the Gaussian index representing j = 1, · · · , M mixture components, P (c|yt, λ) corresponds to the posterior probability of mixture component c generating the vector yt, given UBM λ and µc is the mean of UBM mixture components c.
In practice, the posterior probability of the event that the feature vector yt is generated by the mixture component c, is the alignment of feature vector yt represented by the c th Gaussian. In the DNN based I-vector system, the Gaussian index c is replaced by the class index k of DNN and posteriors from the k th DNN class as the P (c|yt, λ)s in eq.s 1 to 3. Basically the posteriors of the c th Gaussian which is used as the alignments of a feature vector is represented by the senones predicted by the DNN [4].
The DNN architecture used in this study is a multisplice TDDNN [8] which is capable of capturing long term temporal dependency of phonetic contents. To represent long term spectro-temporal dynamics of the speech signal various approaches are proposed in earlier literature. Feature representation like TRAP [10], HATS [11] etc. can be used with standard feed forward model ANN. On the other hand Recurrent Neural Network (RNN) along with Long Short Term Memory (LSTM) architectures are used for modeling of temporal context [12] in deep architectures. But the TDDNN architecture provides some relative advantages over such approaches in terms of computational complexity, training time and parallelization, by utilising a subsampling technique. Here, a narrow temporal context is provided to the first layer and increasingly wide contexts are available to the subsequent hidden layers. The result is that higher levels of the network are able to learn greater temporal relationships. Due to selective computation of time steps such sub-sampling scheme reduces the overall necessary computation during the forward and backward pass computation [8]. The final TDDNN has six layers. If t is some frame, at layers 0, 1, 3 and 4, frames and [t − 7, t + 2] respectively are spliced together to form the subsampling configuration. The DNN configuration can be summarised as follows: P-norm activation function is used in the hidden layers and output is a softmax layer. The network is trained using Natural Gradient for Stochastic Gradient Descent (NG-SGD) algorithm [13]. Initial effective learning rate is 0.0015 and final effective learning rate is 0.00015. The network is trained for 6 number of epochs.

Full covariance GMM-UBM based approach
A Full Covariance GMM-UBM [14] is used to train the I-vector extractor for LR. In such a system, the GMM-UBM plays the role of providing the SS required to train the I-vector model. Since collection of statistics with full covariance GMM is computationally expensive, hence a simplified approach where a tandem of coherent diagonal-covariance and full-covariance models was proposed in [14]. Basically in UBM training first a diagonal covariance UBM is trained. Then, one iteration of the expectation maximization (EM) is run with fixed means and mixture weights, to obtain a full-covariance model. For any utterance X, first a small number of Gaussians are selected on the basis of diagonal model, and then posteriors are computed using the full version of Gaussians with the same means. The performance of this GMM-UBM based system is compared with the proposed DNN-UBM based approach. The system uses same dataset for training I-vector extractor and logistic regression model which are used for DNN-UBM based approach.

Experimental Set up, Results and Analysis
All experiments are performed in the Kaldi Speech Recognition Toolkit [15]. The scripts for the DNN-UBM based LR framework and various experiments reported here are now available as the LRE07/v2 example of the Kaldi code repository (https://github.com/kaldiasr/kaldi/blob/master/egs/lre07/v2/run.sh). Figure 1 shows the process logic of the system. The experimiental results are reported in terms of classification Error Rate (ER) and Average Cost (Cavg) [16].

Database
The DNN is trained using the 1800 hours of Fisher English.

Acoustic feature extraction and speech activity detection
The acoustic features are collected from 20 ms window with a overlap of 10 ms. Low energy dither is added to the windowed signal to avoid runs of digital zeros. Per-file vocal tract length normalization (VTLN) warps are applied. SDC features are extracted using the conventional 7-1-3-7 scheme and the 7 static cepstra are appended producing a 56-dimensional feature vector. This is followed by non-speech frame filtering using energy based speech activity detection and cepstral mean normalization of 3 sec window. For experimental purpose, 19 MFCC aug-

GMM-UBM training
Initially, one diagonal UBM is trained with a small subset of training data (5000 utterances) with 2048 mixture components using the variance equal to the global variance of the features and the mean equal to the mean of distinct and randomly chosen frames. This UBM is used to train a full covariance UBM with same 2048 mixture components with another subset of development data (10000 utterances) which is again used to initialize the final full covariance UBM using the entire training data. I-vector extractor is trained utilizing both full covariance and diagonal covariance in a computationally efficient manner as described in Section 3.

Replacement of GMM-UBM with DNN-UBM
In order to replace the GMM-UBM with a DNN-UBM another set of acoustic features are required. To compute posteriors from the DNN, it uses a special set of high-resolution MFCC features. This is formed by all 40 filterbank coefficients without cepstral truncation extracted from frame-length of 25ms with 10 ms shift. The intention is to provide raw speech features of spectrogram to the DNN, since the deep architectures have inherent capability of learning features from the input signal. Cepstral mean subtraction is performed over a window of 6 seconds. The non speech frames are not filtered beforehand. The posteriors are computed from all frames to maintain actual temporal context and posteriors related to non speech frames are later filtered out. However, to train the T-matrix of I-vector system it uses the standard 56-dimensional SDC features.

I-vector extractor training
Two separate I-vector extractors are trained. First using the full covariance GMM-UBM, which provides the GMM-UBM Ivector based system for comparison. Secondly, the DNN-UBM based I-vector extractor is trained using posteriors computed from DNN. This I-vector extractor is initialized using a separately trained supervised GMM [6]. This GMM is supervised because DNN posteriors are used for training. However supervised GMM is used only to initiate the I-vector extractor, but the training of the T-matrix uses the DNN posteriors. The 40 filterbank features are used to compute the posteriors from the DNN. But SDC features are used for T-matrix training. The posteri- ors related to non speech frames are later filtered out using the speech activity detection information used for the LR features. All I-vectors are 600 dimensional. Two sets of I-vectors for all the testing and training data are computed using both this GMM based I-vector extractor and DNN-based I-vector extractor.

Logistic regression based classification
The I-vectors of the 14 languages namely Arabic, Bengali, Chinese, English, Farsi, German, Hindustani, Japanese, Korean, Russian, Spanish, Tamil, Thai and Vietnamese are used for training the logistic regression model. This logistic regression model is finally used for LR evaluation. Posterior probabilities for the I-vectors of the LRE 2007 evaluation datasets are computed and these posteriors are used for scoring.

Evaluation of the System and Discussion
The system is evaluated using the 14 language targets as outlined in Section 4.6, which are specified in NIST LRE 2007 evaluation. All target languages also serve as the non-target (alternative hypothesis) languages, which forms the target/nontarget language pairs for evaluation. The speech segments are extracted from one side of telephone conversation at standard 8-bit, 8-kHz, µ-law format. Basic pair-wise LR performance are computed for all target/non-target language pairs at 3 sec, 10 sec and 30 sec testing conditions and finally average cost performance Cavg [16] is computed.
The DNN-UBM based I-vector extractor is compared with GMM-UBM based I-vector extractor. Part (a) and part (b) of Table 1 represents the ER and Cavg for GMM-UBM and DNN-UBM systems. The GMM-UBM based system achieves Cavg of %, % and % for 3 sec, 10 sec and 15 sec conditions and these numbers reduces to 7.73%, 1.61% and 0.24% for the proposed DNN-UBM based system. The results clearly state that use of DNN-UBM to collect SS for I-vector extractor provides a significant improvement in individual Cavg compared to GMM-UBM for all three test conditions. Further it has been observed that the performance of the proposed TDDNN-UBM system is relatively better than the results reported in [7] using SDC features in DNN and I-vector framework (Cavg of 19.5%, 8.21% and 4.00% for 3 sec, 10 sec and 15 sec conditions respectively), using bottleneck features DNN and I-vector framework (Cavg of 18.2%, 7.71% and 3.79% for 3 sec, 10 sec and 15 sec conditions) in LRE 2011 evaluation. The reason for this improvement is perhaps the use of an uniform UBM which contributes all variations to the I-vector, hence the I-vector is more pure with respect to language signal. The current study could not provide a comparison using LRE 2011 database since the database is not yet available publicly. Further, another advantage of the proposed approach is that, its easy to extend the system to a new language even if limited audio is available in a new unknown language, by just doing the final logistic regression step. The work described in [17] and [18] presents results using DNN based feedforward classifier and LSTM-RNN based classifier. However those systems are evaluated choosing 8 unidentified languages out of 23 languages in LRE09. But the present results use the 14 languages of LRE 2007 evaluation and applied no filtering based on the amount of data available or language pair confusability. The current architecture allows easy addition of a new language because only the last layer of ligistic regression can be retrained, and the ivector-extractor retraining can be skipped in most cases.
Part (c) of Table 1 represents the performance difference if we skip the VTLN warping of speech signal. This study also experimented with 60-MFCC features for LR. The I-vector extractor is trained using 60 dimensional MFCC-delta-acceleration features (which are standardized in speaker recognition task) and DNN posteriors. Evaluation results are shown in part (d) of Table 1. Individual comparison reflects that for 30 sec condition MFCC provides similar performance, however for short duration like 3 sec condition SDC performs better.

Conclusion
The work reported here has integrated a TDDNN based UBM in an I-vector based LR system replacing the GMM-UBM to compute posterior distribution of feature vectors. The DNN used here is trained as an acoustic model in a monolingual English ASR. The expermental results have shown significantly low average cost performance compared to the GMM-UBM based system. The current set up uses 14 languages for evaluation. However in order to include a new language to the system, the final Logistsic regression model is the only component which is required to retrain. This architectural flexibility is another major advantage of the proposed sysetm.