{'fraction_non_alphanumeric': 0.030603243536676393, 'fraction_numerical': 0.010789170116034471, 'mean_word_length': 5.140833333333333, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Multilingual Transformer Language Model for Speech Recognition in Low-resource Languages

It is challenging to train and deploy Transformer LMs for hybrid speech recognition 2nd pass re-ranking in low-resource languages due to (1) data scarcity in low-resource languages, (2) expensive computing costs for training and refreshing 100+ monolingual models, and (3) hosting inefficiency considering sparse traffic. In this study, we present a new way to group multiple low-resource locales together and optimize the performance of Multilingual Transformer LMs in ASR. Our Locale-group Multilingual Transformer LMs outperform traditional multilingual LMs along with reducing maintenance costs and operating expenses. Further, for low-resource but high-traffic locales where deploying monolingual models is feasible, we show that fine-tuning our locale-group multilingual LMs produces better monolingual LM candidates than baseline monolingual LMs.


Introduction
Automatic Speech Recognition (ASR) usually involves two passes. The first-pass acoustic models and n-gram language models generate n-best hypotheses from the global search space [1]. In the second pass, Neural Network Language Models (NNLM) are widely used to re-rank the n-best hypotheses [2]. It has been demonstrated that re-ranking using NNLM is effective at reducing WER (Word Error Rate) [3], with Transformer language models producing state-of-the-art results in re-ranking [4]. Today our ASR system supports 100+ locales, but reranking is only applied to a few high-resource locales, even though we have proven the higher benefits of re-ranking for low-resource locales like Slovenian. The key challenges today are: (1) the low-resource locales' training data is scarce, which limits our capacity to train the NNLM, (2) it is computationally expensive to train and regularly refresh 100+ monolingual re-ranking models, one for each locale, (3) it is prohibitively expensive and inefficient to host these monolingual models in production, as traffic can be sparse, but each model ends up taking memory and compute to host across all our Speech clusters.
Multilingual Transformer language models [5] [6] [7] provide a very effective general solution to support ASR with pretrained components and data sources that can be shared across multiple languages. When applied blindly, however, multilingual transformer language models may not always match or beat the monolingual models. We found that grouping multiple akin locales can optimize performance, especially when dealing with low-resource locales. As a result, our Locale-group LMs outperform the general multilingual solutions.
This key insight has helped us tackle the above-listed challenges. Locales with limited resources (scarce data) can benefit from all the data available for their locale group. We would only need to train and maintain a few locale-group Transformer LMs and still can attain locale coverage of 100+ locales for reranking, and fewer overall 2 nd pass LMs result in hosting and scaling efficiencies across our clusters in production.
In addition, our key finding regarding grouping lowresource locales has been found to work in other related domains, such as improving capitalization and punctuation in recognition outputs, with potential future applications beyond speech.

Related Work
Multilingual/Cross-lingual. The effectiveness of sentence encoders' generative pre-training was first demonstrated for English natural language processing [8] [9] [10]. Multiple approaches have since been proposed to extend it to multilingual/cross-lingual pretraining and show the success in transfer learning, such as mBERT [9], XLM [5], XLM-R [6], Unicoder [11], etc. Large amounts of unlabeled data from multiple languages are used to train these models, with the goal that low-resource languages can benefit from high-resource languages from shared vocabularies and underlying linguistic similarities. mBert trains a BERT model using Wikipedia corpora in 104 languages. XLM introduced a translation language model (TLM) in addition to masked language model (MLM), in which bilingual sentences are concatenated as inputs. To further improve the performance, Unicoder presents three new crosslingual pre-training tasks, including cross-lingual word recovery, cross-lingual paraphrase classification and cross-lingual masked language model. XLM-R trains exclusively with MLM objective on a huge multilingual dataset at an enormous scale.
Multilingual is also explored in ASR, primarily from an acoustic perspective. In [12], a massive multilingual acoustic model trained with more than 50 languages and more than 16,000 hours of audio is proven to improve recognition performance especially in low-resource languages. XLSR [13], a cross-lingual speech representation learning method is proposed by pre-training a single model from 56,000 hours raw waveform of speech in 53 languages.
Our work mainly focuses on Multilingual Language Model in ASR 2 nd pass re-ranking, where a language model score is interpolated with 1 st pass LM and AM to select the 1-best recognition candidate.

Locale-group Transformer LM
Our proposed approach involves two steps. The first is to identify the underlying language group of the low-resource locale using our data-driven method, and the second is to process the low-resource locales' data with shareable Byte Pair Encoding (BPE) tokens [14] and train the large-scale Locale-group Trans- former Language Model. Whenever we lack enough resources and hardware to support individual model development and deployment, we can choose to deploy the group-based multilingual Transformer Language Model, which provides significant Speech Recognition accuracy improvement, maintenance and cost reduction.

Language Group Identification
As one of three organizations selected to potentially partner with the European Parliament in 2020, Microsoft developed a real-time AI-based tool for live transcription and translation of debates. To identify the underlying language groups for 26 European languages, we proposed a two-step data-driven mechanism.
Firstly, we computed bi-lingual similarity score, which can be a measure of the number of phonemes, words, phrases, or the like that are present in both locales. The Figure 1 shows an example of bi-lingual lexical similarity scores for 26 European languages, where higher score indicates closer linguistic relations between the languages. We observed codeswitching/loanwords to be common, especially in English. In contrast, per our experiments Bulgarian (bg-BG) does not appear to be close to any other languages based on our collected data. We suspect that some of the data skew was caused by source data filtering.
Secondly, we applied vector-based clustering techniques to categorize similar languages together based on similarity score vectors. As shown in Table 1, this mechanism successfully identifies language family like Balto-Slavic, the group 2, which contains Slovenian, Croatian, Slovak and Czech. Group 3 consists of most high-resource Germanic languages such as English and German, and Latin (Romance) languages like Italian, Spanish and French. In Group 4, Greek is supposed to have its own alphabet -the similarity with other languages is mainly due to code-switching/loanwords.

Shareable BPE Tokens
In our proposed approach, we process all languages with the same shared vocabulary created through Byte Pair Encoding

Data Balance
We compile a language group training dataset with a balancing mechanism and train the Locale-group Multilingual Transformer Language Model. To ensure balanced data coverage for multiple regions within the same family, we sample sentences with multinomially distributed probabilities {q}i=1...N , similar as how sentences are sampled in [5],

Locale-group Model Training
Using balanced language family data, we train the Locale-group Multi-lingual Transformer Language Model, which is similar to the structure used in [15]. During training, we record the valid loss and perplexity of individual locales, and of the language family. According to our findings, the average loss minimum is within the range of the individual locale's loss minimum, which indicates that the Locale-group Multilingual Transformer Language Model converges for all locales within the identified language group.

General & Masked Fine-tuning
In natural language processing (NLP) and machine translation (MT), fine-tuning a pre-trained language model has become the de facto standard for transfer learning [10] [16]. In our work, our ability to serve efficiently and reduce computation costs will be compromised if we fine-tune the multilingual model towards a target language. However, the possibility of fine-tuning is also worth exploring, as (1) knowledge transfer will be verified if we can achieve better performance with fine-tuned monolingual model than the one trained from scratch, along with a unified training recipe, and (2) certain high-traffic locales will separate themselves from low-resource groups as more data is collected and more traffic is received over time, which allows us to support monolingual model deployment with adequate resources.
In general fine-tuning, we reserve the pre-trained multilingual model parameters, feed the target locale's data in, and train for several more epochs until it converges to a new minimum.
Masked Fine-tuning is designed to force the model to tune to a target locale. For BPE tokens that do not exist in the target locale, we freeze the token embedding updates and set the prediction score to a very small negative number, so the token loss will be close to 0, and avoid paying attention to irrelevant tokens. This process is illustrated in Figure 2

Train
We sampled 5B sentences across 26 European locales from our in-house text data corpus, with an average sentence length of 12. All text data are pre-processed into lexical format with our own text-normalization pipeline. Low-resource locales are upsampled per Section 2.3.1. The same data is used to train the shareable 250K BPE tokens.

Test
Test audio data used for word-error-rate reductions (WERR) measurement of each locale consists primarily of dictations and

Model
We experiment with configurations described in Table 4.
We train a baseline 1x1024:512 LSTM LM for each language, where 1 is the number of layers, 1024 is the dimensionality of the LSTM state, 512 is the dimensionality of the embedding and also the output dimensionality of the projection layer.
In addition, we trained one multilingual all-languagestogether, four multilingual locale-group, and 26 monolingual Transformer language models with the same shareable 250K BPE token set, same data distribution and similar model configuration. These Transformer language models consist of 12 transformer layers, where each transformer layer contains 4096 feedforward dimensions with 16 heads. The warm-up is set to increase the learning rate gradually to improve the convergence of Transformer LMs [17].
We also applied general fine-tuning and masked fine-tuning as described in section 3.3.3.
As shown in Table 3, we observe 3.34% average WERR improvement because of architecture upgrade from LSTM to Transformer, and more parameters. However, it is challenging to deploy those monolingual Transformer LMs as we discussed three limitations in the beginning.
On the other hand, the Locale-group Transformer LM provides a good solution considering the deployment restrictions. Firstly, with one Transformer LM to serve multiple locales in the same group, we can achieve 3.12% average WERR gain compared with the LSTM baseline. Secondly, the locale-group   Even more, if we can allocate enough resources to train and deploy dedicated Transformer LM for low-resource locales, we can achieve more with masked fine-tuning. Compared with the monolingual LSTM baseline, our multilingual Locale-group model with masked fine-tuning can provide an additional 4.64% average WERR.

BPE Token Size
We also trained models with 64k shareable BPE tokens to evaluate the impact of BPE token size. WERR on individual locales varies, but compared with 250K, the 64K BPE based Localegroup models generally regress around 2.48%. When we do masked fine-tuning, the gap is reduced to 0.66%. Our hypothesis is that the average text sequence length are elongated by smaller BPE token size, therefore brings in challenges to learn generic sequence patterns under the same number of model pa-rameters.

Hosting Efficiency
One key motivation for exploring this idea was to improve production efficiency. Our speech service needs to be deployed globally to tens of clusters. Monolingual and general multilingual are two extremes when it comes to hosting costs. Deploying over 100 monolingual models everywhere is quite resource heavy and often wasteful as traffic can be sparse for some locales. On the other hand, general multi-lingual models oversimplify this, at the cost of WERR regression for some locales. Natural data imbalance is also harder to tackle at this scale, and often multi-lingual models can beat monolingual only by increasing model complexity (like adding more Transformer layers). This is also undesirable as this affects individual request latency at serving time. Our approach of locale-group multilingual models finds a more optimal point, needing us to deploy only a few locale-group models across our speech clusters, and better WERR than both monolingual and general multi-lingual models in most cases. We keep the option to deploy monolingual or fine-tuned monolingual models for certain high-traffic locales.

Parameter Tuning
We haven't tuned dropout and hyper parameters extensively for all the models. We believe we will achieve more gains with parameter tuning, and plan to explore this in the future work.

Application in other areas
In our hybrid ASR system, there are many other areas where we use monolingual models or general multi-lingual models. We explored the applicability of this locale-grouping technique to Capitalization models. We achieved similar results with locale-group capitalization models outperforming monolingual and general multi-lingual capitalization models.

Conclusion
Neural Network Language Model (NNLM) is an essential module in hybrid ASR to deliver the optimal recognition accuracy. In this work, we proposed a general and scalable approach to train and deploy large-scale Locale-group Transformer NNLM to support ASR in low-resource languages, where we observed significant accuracy improvement and reduction in model development and maintenance. Further, our fine-tuning experiments show that this locale-grouping helps create better monolingual models for low-resource languages.