{'fraction_non_alphanumeric': 0.03346303501945525, 'fraction_numerical': 0.01556420233463035, 'mean_word_length': 5.084166228300894, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
End-to-End Bengali Speech Recognition

Bengali is a prominent language of the Indian subcontinent. However, while many state-of-the-art acoustic models exist for prominent languages spoken in the region, research and resources for Bengali are few and far between. In this work, we apply CTC based CNN-RNN networks, a prominent deep learning based end-to-end automatic speech recognition technique, to the Bengali ASR task. We also propose and evaluate the applicability and efficacy of small 7x3 and 3x3 convolution kernels which are prominently used in the computer vision domain primarily because of their FLOPs and parameter efficient nature. We propose two CNN blocks, 2-layer Block A and 4-layer Block B, with the first layer comprising of 7x3 kernel and the subsequent layers comprising solely of 3x3 kernels. Using the publicly available Large Bengali ASR Training data set, we benchmark and evaluate the performance of seven deep neural network configurations of varying complexities and depth on the Bengali ASR task. Our best model, with Block B, has a WER of 13.67, having an absolute reduction of 1.39% over comparable model with larger convolution kernels of size 41x11 and 21x11.


Introduction
Bengali, also referred to as Bangla, is the seventh most spoken native language in the world, the second most widely spoken language in India and the official language of Bangladesh. It has around 185 million speakers in the Indian subcontinent. Bengali consists of 29 consonants, 7 vowels and 7 nazalized vowels and shares the phonetic space with other Magadhan languages like Assamese, Odiya etc. However, unlike various prominent languages spoken in the Indian subcontinent, research and resources for automatic speech recognition in Bengali are scarce.
With the emergence of voice as a natural form of humancomputer interaction and the advent of large datasets, development of Automatic Speech Recognition (ASR) has seen major strides over the past few years in terms of robustness and practical applications. Deep Learning, facilitated by Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN), as well as advancements in end-to-end training methodologies such as the Seq2Seq and Connectionist Temporal Classification paradigm have enabled the development of state-ofthe-art end-to-end ASR pipelines.
In this paper, we investigate the performance and applicability of existing deep learning based end-to-end automatic speech recognition paradigms, specifically CNN-RNN based hybrid models trained using Connectionist Temporal Classification (CTC) Loss in an end-to-end manner [1] on the Bengali ASR task. We also perform ablation experiments to investigate the efficacy of smaller convolution kernels widely used in com-puter vision ([2], [3]) which pose various benefits such as optimal number of parameters [2] and lower memory addressing cost [4], over the larger filters prevalent in the speech recognition domain [1]. To this end, we benchmark and evaluate 7 different CNN-RNN based deep neural network architectures with various complexities and depth, on the Large Bengali ASR Training data set, a large scale publically available Bengali corpus with around 200,000 utterances.

Deep learning and end-to-end ASR: Primer
Traditional automatic speech recognition systems spearheaded by HMM-GMM based acoustic models, where HMMs were used for alignment and GMM mapped these aligned tri-phones to output characters, have dominated the field for quite some time, and only recently were surpassed by HMM-DNN systems, with deep neural networks replacing GMMs.
The advent of sequence-to-sequence models (Seq2Seq) paired with the development of Connectionist Temporal Classification (CTC) [5] revolutionized the end-to-end approach for training speech recognition systems by training a single network that directly mapped audio sequence to the output text, with recurrent neural network ( [5], [6], [1]) and attention ( [7], [8]) based end-to-end models setting state-of-the-art results on several benchmark datasets.
Most recent deep learning based end-to-end systems utilize log-spectrogram or MFCC features as inputs to the network. Given the innate capability of deep neural networks to learn features from raw inputs, multiple recent works have switched to learning features from raw speech signal. To this end, [9] recently proposed time-delay convolution based trainable filterbanks, observing comparable performance with melfilterbank based systems on the WSJ dataset.

Speech Recognition and Indian Languages
Languages spoken in the Indian subcontinent are a topic of interest primarily due to the sheer number of speakers, but the lack of large-scale, publicly available datasets pose a challenge in the development and evaluation of deep learning based systems.
Most recently, a multilingual phone recognition system was proposed by [10], covering four Indian langauges, viz. Kannada, Telugu, Bengali and Odia. They proposed a traditional HMM-DNN based phone recognition system leveraging Articulatory Features (AFs). Their DNN subsystem utilized tanh [] activation function at the hidden layers, and was trained in a greedy layer-by-layer supervised setting. However, unlike [10], which proposed a phoneme recognition system evaluated on a very small dataset (19.2 hours total), we evaluate and propose CNN-RNN based end-to-end systems evaluated on a very large dataset.
Addressing the challenges involved in building robust speech recognition systems for Indian languages, [11] proposed training phoneme recognition systems using articulatory movements captured by an electromagnetic articulograph. They evaluated multiple phoneme recognition models, similar to [10], with the addition of HMM-RNN based acoustic model. However, they collected a small, 2 male speaker dataset for each language covered, which is an inconclusive sample size to comment on the genericity and applicability of the approach.

Datasets
Large Bengali ASR Training data set (http://openslr.org/53/) is used for evaluating models on the Bengali ASR task. Since dev/test splits were not provided for the Bengali data set, we propose our own evaluation procedure, splitting the data set into train, validation and test sets (table. 1), available upon request.

Proposed Approach
In the following subsections, we shed light on the evaluated CNN-RNN based end-to-end speech recognition networks. We also discuss the proposed changes to the original convolution as proposed in  [6]). The convolution block represents n sized stack of convolution layers. Fig. 2 shows the original 2-layer convolution stack as proposed in [1], where each convolution layer is followed by a Batch Normalization layer [12] and Hard-Tanh non-linearity.

Proposed Convolution Blocks
The initial layers of a CNN based deep neural network are of significant importance in terms of overall network complexity as well as performance. The initial layers consume the most memory and comprise of the most floating point operations [2] due to larger spatial dimensions of the input which is gradually downsampled with the depth of the network. Convolutional neural networks comprising solely of kernels with very small receptive fields (3x3 [2], 7x7 [3]) have significantly pushed the state-of-the-art on various computer vision tasks and challenges   [4], [13], [14], [15], [16], [17], [18]), and have various benefits over larger filters, viz, 1. Smaller kernels are more parameter efficient. [2] 2. Multiple smaller kernels stacked over each other effectively increase the discriminative power of the decision function [2], by increasing the total number of nonlinearities.
3. Larger convolution filters operated at a stride on the very first layer greatly reduce spatial resolution of filter maps, which might be too drastic and might hurt the performance of the network.
We propose 2 convolution blocks with 2 and 4 convolution layers, henceforth referred to as Block A and B, respectively (Fig. 3). Instead of using large rectangular filters similar to [1], we utilize smaller 3x3 filters for all but the first layer (which uses 7x3 filters) in all our layers. Table.2 depicts the total number of floating point operations (FLOPs) in millions and the number of parameters for each proposed convolution block as well as the 2-layer convolution block as proposed in DeepSpeech 2. As evident from the aforementioned table, the proposed convolution blocks have significantly lower FLOPs and parameters.

Proposed Networks
We propose and evaluate 7 network configurations of varying complexities as listed in Table 3. 3 distinct RNN stacks are evaluated with 3, 4 and 5 bidirectional GRU layers respectively.

Implementation and Training Details
Following [1], we use normalized log-spectrogram calculated using a sliding window of width 20ms and stride of 10ms, followed by a 160 point FFT as inputs to the network. The network is trained end-to-end using the CTC loss function [5], which facilitates prediction of character sequences directly from input.
Training was performed on a single GTX 1080 Ti system with a batch size of 20 using Stochastic Gradient Descent, with an exponential learning rate decay schedule and an initial learning rate of 3e − 4. Early stopping [19] with a patience of 3 was used for regularization, and all the networks were trained for a maximum of 30 epochs.
Inference: At inference time, we pair the proposed models with a 4-gram Kneser-Ney based language model with a beam size of 100, trained on a large corpus of Bengali text collected from Bengali news sites and blogs, using the Kenlm toolkit [20]. Results without the language model are also provided for the test set. Table 4 depict the word error rate (WER) for the Bengali ASR task on various networks on the val and test set. WER for validation is calculated without a language model, whereas results with and without language model are shown for the test set.

Experiments and Results
Model B-5GRU-Large is the best performing configuration, achieving a WER of 13.67%, followed closely by B-5GRU. It is worth noting that B-5GRU, despite having fewer hidden units  per GRU layer as compared to 2CNN-5GRU (512 v/s 800) outperforms the latter by a significant margin: an almost 2% absolute reduction in WER (without a language model), while at the same time consisting of a convolution block with lower number of parameters and FLOPs, hence validating the applicability of smaller convolution kernels for ASR task.

Conclusions
In this work, we investigated the performance and applicability of prominent end-to-end automatic speech recognition pipeline comprising of CNN-RNN based deep neural network trained using the CTC loss function for the Bengali language, evaluating and benchmarking various network configurations of different complexities. Our studies also validate the applicability of smaller convolution kernels widely used in the computer vision domain for ASR tasks, with the proposed convolution block A performing on par with convolution block comprising of larger 41x11 kernels keeping in consideration the drastically lower number of parameters for the former, whereas convolution block B outperforming the convolution block comprising of larger convolution kernels. The proposed network configurations can be applied to other Magadhan languages and comparable results can be expected as they share the phonetic space with Bengali. [2] K. Simonyan and A. Zisserman, "Very deep convolutional networks for large-scale image recognition," arXiv preprint arXiv:1409.1556, 2014.