{'fraction_non_alphanumeric': 0.033391915641476276, 'fraction_numerical': 0.014091707940565586, 'mean_word_length': 4.849719626168224, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
DICT-MLM: Improved Multilingual Pre-Training using Bilingual Dictionaries

Pre-trained multilingual language models such as mBERT have shown immense gains for several natural language processing (NLP) tasks, especially in the zero-shot cross-lingual setting. Most, if not all, of these pre-trained models rely on the masked-language modeling (MLM) objective as the key language learning objective. The principle behind these approaches is that predicting the masked words with the help of the surrounding text helps learn potent contextualized representations. Despite the strong representation learning capability enabled by MLM, we demonstrate an inherent limitation of MLM for multilingual representation learning. In particular, by requiring the model to predict the language-specific token, the MLM objective disincentivizes learning a language-agnostic representation -- which is a key goal of multilingual pre-training. Therefore to encourage better cross-lingual representation learning we propose the DICT-MLM method. DICT-MLM works by incentivizing the model to be able to predict not just the original masked word, but potentially any of its cross-lingual synonyms as well. Our empirical analysis on multiple downstream tasks spanning 30+ languages, demonstrates the efficacy of the proposed approach and its ability to learn better multilingual representations.


Introduction
Masked Language Modeling (MLM), introduced by Devlin et al. (2019), Taylor (1953), in conjunction with transformers (Vaswani et al., 2017), has spurred a rapid succession of massive pre-trained models -RoBERTa (Liu et al., 2019a), ALBERT (Lan et al., 2019), ELECTRA (Clark et al., 2020), etc. These new models have shown significant advances in several natural language processing * Work done as part of Google Research Internship.
(NLP) tasks and set new benchmarks. The success of MLM is attributed to the model learning better contextualized representations by virtue of using the bi-directional context to predict randomly masked tokens or sub-words in a sequence.
MLM and its variants have also been applied successfully to the multilingual setting (mBERT 1 , XLM; Lample and Conneau (2019)) wherein a single model is pre-trained on the concatenated monolingual corpora of multiple languages. This has greatly improved the cross-lingual generalizability of the pre-trained model resulting in significant improvements, especially in the zero-shot setting where the pre-trained model is fine-tuned for the task in question on one language and directly applied to another language.
The key to successful cross-lingual transfer learning (especially in the zero-shot setting) is the ability to learn semantically rich language-agnostic representations (Ruder et al., 2019). The more language-agnostic a representation, the better its chances of carrying over knowledge seamlessly across languages and overcome the lack of training data in other languages (Upadhyay et al., 2016;Ruder et al., 2019).
However, the design of MLM is not conducive for learning language-agnostic representations. This is because MLM requires the model to predict the masked token in the specific language, and not any of its synonyms -either within or across languages. For example, in the following sentence Food and water are neccessities of life, with the word life masked, the MLM objective drives the model to learn this exact word (life) in the given context. This thus forces the learned representation of the word to be different from its synonyms in other languages (e.g. vida, Leben, jeevan, . . . ). In other words the learned representations of the model are forced to be language-specific to help identify the exact language of the sentence. This is partially the reason behind mBERT being very good at the Language-Identification task 2 (Pires et al., 2019).
This behavior is also seen when we evaluate the language-agnosticity of the learnt mBERT representations across the multiple layers of the model. As seen in Figure 3, the 8th layer of mBERT is 25+% better at cross-lingual semantic retrieval compared to the final layer of BERT. This behavior has also been observed in prior work (Hu et al., 2020;Pires et al., 2019). One approach proposed to improve this language-agnosticity is translation language modeling -TLM - (Lample and Conneau, 2019). However this relies on the use of expensive, sentence-aligned parallel corpora, while still retaining the same MLM-incentive issues.
To address these limitations, we propose DICT-MLM which facilitates cross-lingual alignment more directly. DICT-MLM achieves this by incentivizing the model to be able to predict not only the masked token in its original language, but also its synonyms in other languages. While some existing approaches rely on millions of expensive translations, we show that even with only thousands of cross-lingual word pairs (from bilingual dictionaries), DICT-MLM is able to learn a more potent language-agnostic representation than mBERT.
We demonstrate this empirically on multiple downstream tasks including sequence labeling (NER, POS), classification (MLDOC-Headline, PAWS-X), textual entailment (XNLI) and crosslingual sentence retrieval (TATOEBA) -leveraging a data setup similar to the XTREME (Hu et al., 2020) benchmark which covers 40 typologically diverse languages. Our contributions can be summarized as follows: • We propose a new pre-training method DICT-MLM to enforce cross-lingual alignment more directly. We find that DICT-MLM is especially effective for sequence-labeling tasks (NER, POS) since these tasks rely more on token-level representations rather than the full sequence representation which is more conducive to our cross-lingual word alignment objective (Table 1).
• Overall, we observe that our model outper-2 In fact, prior work (Conneau et al., 2017) has explicitly tried to learn multilingual representations by adversarially training the model to not be able to distinguish language. forms mBERT for all tasks with improvements of +2.4 F1 for NER, +3.7 accuracy for POS, +0.5 accuracy for XNLI, +2.5 accuracy for MLDOC-Headline and +5.3 accuracy for PAWS-X, averaged across all languages. Furthermore, our model also outperforms existing work that leverage parallel text which is a more expensive resource than bilingual dictionaries (Table 1 and Table 2).
• Our learned representations are empirically shown to be far more cross-lingual, with 20+% improvements observed on TATOEBA. (Fig 3).
• Qualitative analysis reveals that even languages which do not have publicly available bilingual dictionaries benefit from DICT-MLM (Table 3).

Methodology
Our Focus: While there exists different (often complementary) techniques to improve crosslingual transfer learning -including use of larger, richer pretraining datasets (Conneau et al., 2020) or finetuning using data augmentation (Fang et al., 2020) -our focus is specifically on the improving the pretraining objective that is common to all of these approaches, namely MLM. In order to address the aforementioned limitations of MLM, we propose DICT-MLM which leverages bilingual dictionaries during pre-training to explicitly facilitate cross-lingual word alignment. In Section §2.2 we describe the training data creation for DICT-MLM followed by the model and training regimen in Section §2.3.

Preliminaries
Given an input text stream W = w 1 , w 2 · · · w n , MLM uses the past and future tokens W \t := (w 1 , · · · , w t−1 , w t+1 , · · · , w n ) to predict the masked token w t . Devlin et al. (2019) apply this objective on the concatenated monolingual corpora of 104 languages to get a pre-trained multilingual model (mBERT). 3

Preparing the Training Data
To facilitate cross-lingual word alignment during pre-training, DICT-MLM is trained to predict the cross-lingual synonym of the masked token. To train this model, we first create multilingual code-switched sentences from the monolingual corpora of L languages by leveraging bilingual dictionaries. Consider the monolingual corpus D l = {x l 1 , x l 2 , · · · , x l n } for language l. For each sentence x l i ∈ D l , we randomly select (and mask) 15% of whole words which have at least one dictionary entry in any bilingual lexicon. For each such masked token (w l ∈ x l i ), we first retrieve all its synonyms from the bilingual dictionaries 4 . From the retrieved set of candidate (cross-lingual) synonyms we then randomly select one synonym given by s l d w where l d refers to the language of the selected synonym. An example of such a generated sentence can be seen in Figure 1. Note that since synonyms for masked tokens are sampled independently, the resulting sentences could contain tokens from multiple different languages (as seen in above example). This makes the masking task a lot more challenging, as the model not only needs to learn the semantics of the word in the masked slot, but also needs to predict the word in the desired language (which may not be the same as the surrounding context). Our hypothesis is that this dual-challenge now forces the learned representations to be both semantically and cross-lingually rich. By repeating this process for all monolingual corpora of the L different languages, we get the desired training dataset 5 .
After generating this multilingual code-switched data we perform WordPiece (Wu et al., 2016) tokenization following BERT (Devlin et al., 2019). In the original implementation of BERT (Devlin et al., 2019), the authors mask individual wordpieces within a sentence which however causes only some word-pieces within a word to be masked making MLM a simpler task. The authors later proposed to perform whole-word masking (WWM) wherein all the wordpieces of a word are masked while keeping the percentage of masked tokens as before. They empirically find WWM to perform better hence we also perform WWM for DICT-MLM. We further use a dynamic masking strategy, as proposed by RoBERTa (Liu et al., 2019a), which duplicates the data n times causing different tokens to be selected for masking each time. For DICT-MLM, this amplifies the randomness since during the dictionary-replacement step the cross-lingual synonym is also selected randomly.

Model and Training Regimen
We next describe the training setup used for pretraining the different models. We base our model on mBERT (Devlin et al., 2019) which uses a transformer stack (Vaswani et al., 2017) and is trained on the Wikipedia monolingual corpora of 104 languages. For the DICT-MLM methods, we did not use the next sentence prediction (NSP) objective as prior work (Liu et al., 2019a) has shown that to add very little gains.
As in the original mBERT implementation, masked tokens are replaced with the [MASK] token 80% of the time, 10% of the time with the original token and the remaining 10% with a random token. Given that downstream tasks/datasets are typically monolingual in nature, we combine our DICT-MLM objective with the regular MLM objective to ensure the model learns from monolingual contexts just as well. In particular, we multitask these two objectives by setting the label of the masked token to be the cross-lingual synonym t% of the time. For the remaining (100-t%) time the original token is used as the label. We show the effect of the value of t in our experimental section.
At this point, an astute reader may wonder how a model can learn to predict masked tokens in different languages without being given any cue about the target token's language. To remedy this we make two architectural changes to the mBERT  model. First, we add a language embedding layer to the input layer similar to the position and segment embeddings layer. Furthermore, since we want to provide the MLM prediction task with additional language context to help it predict tokens in the appropriate language, we make a second change as well. Specifically we modify the dense transformation that happens as part of the MLM prediction. Rather than providing this dense block only the token embedding of the final transformer layer as in mBERT, we additionally concatenate this token representation with an embedding of the associated language. As in MLM, the output of this dense transformation is used to make a prediction over the shared wordpiece vocabulary via softmax. These modifications are illustrated in Figure 2. It is worth noting that the latter change -of the additional conditioning before the MLM softmaxonly affects the pretraining and not the finetuning.
Furthermore since the number of languages is much smaller than the vocabulary size, these changes add very few parameters to the overall model, and do not affect inference / training speed. While both of the above mentioned modifications require the use of language embeddings, which could be independently learned, in practice we found that coupling the two help improve performance slightly and is thus the setup we use in our experimentation. We also looked to understand the effect of these architecture changes in our empirical analyses.
DICT-TLM Inspired by the translation language modeling (TLM) objective (Lample and Conneau, 2019), we experiment with a similar TLM variant of DICT-MLM which uses bilingual dictionaries instead of parallel data. Specifically, rather than rely on parallel corpora for sentence-aligned translations, we instead simulate translations using the multilingual code-switched sentences as described in Section §2.3. As in TLM, this (synthetic) translation is concatenated with the original sentence, with 15% of the tokens of the concatenated sentence being masked (WWM) and predictions made using the MLM objective. We refer to this model variant as DICT-TLM.

Experiments
In this section, we empirically demonstrate the efficacy of DICT-MLM on several downstream tasks including textual entailment, sequence labeling, sentence classification and sentence retrieval.
Data: We use the Wikipedia monolingual corpora for 104 languages. Owing to wide differences in the monolingual data availability of the different languages we employ a temperature based data sampling policy following . We use the publicly released MUSE (Conneau et Table 1: Average scores for each task across the respective languages reveal that our proposed methods outperform existing baselines. The models/tasks for which we were able to train multiple runs, we report the mean and std. deviation across two runs, wherever applicable. We refer to XTREME Hu et al. (2020) for the MBERT (PUBLIC) and XLM results. For TATOEBA, we report the accuracy averaged across last four layers while Hu et al. (2020) report results by concatenating the representations from those last four layers.
We pre-process all 110 dictionaries to create a single dictionary aggregating all synonyms across multiple languages as seen in Table 4. While 40 of the 45 languages have dictionaries available only to/from English, five languages (German, Spanish, Italian, French, Portuguese) have additional dictionaries between them. As mentioned in Section §2.3, we randomly sample one synonym for each (DICT-MLM) masked token ensuring that synonyms from all languages have equal probability.

Experimental setup
We follow the same hyperparameter settings as mBERT (Devlin et al., 2019) using 12 layers of the transformer stack (Vaswani et al., 2017) with 768 hidden units. We use a learning rate of 0.0016 with the LAMB optimizer (You et al., 2020) and a batch size of 8192. We train all our models including the mBERT baseline with the above hyperparameters on 128 TPU v3 chips for 500K steps. We use language embeddings of 768 hidden size. We experiment with three different DICT-MLM models by varying the percentage of regular MLM tokens denoted by t = [50%, 70%, 90%], as outlined in Section §2.3. To account for model variance we pre-train the DICT-MLM model twice and report the averaged results (along with the variance). We believe this provides a more fair assessment of the efficacy of these deep models given their inherent variance across runs.
During fine-tuning we use a learning rate of 3e− 5 with the AdamW optimizer and train for 8 epochs. In order to be consistent with the pre-training setup, for all DICT-MLM fine-tuning runs we add language embeddings with the token embeddings as input to the transformer stack. We evaluate our model in the zero-shot setting where we fine-tune the models only on the English portion of the tasks and directly apply the model on the test portion of the different languages.

Tasks
Textual Entailment We use cross-lingual natural language inference XNLI (Conneau et al., 2018) dataset which covers 15 languages. The model takes in two input sentences and is required to classify into one of the three labels: entailment, contradiction, neutral.
Sequence Labeling We use named entity recognition NER (Pan et al., 2017) and part-of-speech tagging POS (Nivre et al., 2018) datasets which cover 38 languages and 33 languages respectively. We use the same set of languages as the XTREME benchmark (Hu et al., 2020). For NER, the model takes in an input sequence and is required to identify entities of the following three types: PER, ORG, LOC. For POS tagging, the model is required to tag each token in the input sequence with its universal part-of-speech (UPOS) tag.

Sentence classification
We use the PAWS-X (Yang et al., 2019) dataset which takes two input sequences and classifies whether these sequences are paraphrases of each other. This is available for 7 languages. We also use the MLDOC (Schwenk and Li, 2018) dataset which performs document classification (into 4 categories) given an input headline and is available for 8 languages.
Sentence Retrieval We use the TATOEBA (Artetxe and Schwenk, 2019) dataset which contains upto 1000 English-aligned sentences across 122 languages. We use the same set of languages as the XTREME benchmark (Hu et al., 2020). We find the nearest neighbor using cosine similarity of the sentence embeddings. Sentence embeddings are derived by mean-pooling of the token embeddings. 6

Results
In Table 1 we report results for the six tasks, averaged across all languages and runs. 7 We find that our proposed models significantly outperform the vanilla mBERT baselines. Furthermore, the DICT-MLM models even outperform existing works that leverage (millions of) parallel sentence pairs, specifically the XLM model (Lample and Conneau, 2019). Table 2 directly compares the performance of our approach versus the recently proposed contrastive alignment method (Wu and Dredze, 2020b). In particular, Wu and Dredze (2020b) -which proposes several alignment methods that use parallel data to improve the learned multilingual representations 8 -only report results on NER, POS and XNLI for 9 languages. Table 2 reports performance for those languages and tasks with significant gains observed for NER and XNLI tasks. Interestingly on POS tagging, Wu and Dredze (2020b) report significantly higher results for their mBERT baseline itself than either we could reproduce or that previous works reported (Hu et al., 2020). We still observed DICT-MLM outperform (our implementation of) mBERT and leave resolving this disparity to future work. A further discussion on a possible explanation for DICT-MLM's reduced performance on POS-tagging can be found in Sec. §3.4. Does DICT-MLM help learn more languageagnostic representations? The primary motivation behind DICT-MLM was overcoming the limitations of MLM when it comes to learning language-agnostic representations. Thus to understand the cross-linguality of the learned representations of the DICT-MLM models, we use the Tatoeba cross-lingual nearest-neighbor retrieval task. This dataset directly measures and rewards the languageagnosticity of the learned representations. As seen in Figure 3, we find that our proposed methods 6 The results and trends are similar for other distance functions and pooling strategies but we leave out these variants for the sake of brevity. 7 We use the same set of test languages as used by the XTREME (Hu et al., 2020) benchmark. Due to time constraints we only pre-train the DICT-MLM-X models twice. 8 We compare with the Strong Align variant from Wu and Dredze (2020b) which improves upon the mBERT model in most cases. are able to learn much better cross-lingual representations across *all* layers relative to mBERT. Relative to the steep drop-off in performance between middle and final layers observed in mBERT models, the DICT-MLM models are able to retain significantly more multilingual representations in the final layer representations. One hypothesis for the observed drop in the DICT-MLM models' final layer is the reliance on the language embedding added to the input. We leave further investigation of this phenomena to future work.

Analysis
In this section, we aim to understand the behavior and performance of the proposed DICT-MLM models in more detail.
Is there a relation between dictionary availability and the downstream performance? As mentioned earlier, the MUSE bilingual dictionaries cover only 45 of the 104 languages used for pretraining. Specifically, across these 45 languages, we find 34.15% of the total Wikipedia tokens have at least one synonym available. Naturally a question arises whether only languages having dictionaries are benefited by this approach? In Table 3 we therefore compare the downstream performance on languages which have dictionaries available versus those which do not. We report the difference in the (average) scores between the DICT-MLM-50 model and the mBERT model for NER, POS and TATOEBA. 9 Needless to say, languages which have dictionaries available show more improvements however, we do observe significant gains  Table 2: Average scores for each task across the 9 languages (as reported in Wu and Dredze (2020b)) reveal that our proposed dictionary-based methods outperform the existing baseline which uses parallel data. MBERT (THEIRS)* refers to the results reported for mBERT by Wu and Dredze (2020b) and MBERT (OURS) refer to the scores from our re-trained model. For our trained models we report the mean and std. deviation across two runs, wherever applicable.  Do some tasks/languages benefit more from a particular DICT* variant? As mentioned in Section §2.2, we multi-task between the regular MLM objective and the DICT-MLM objective for the three model variants. This leads to having different amounts of cross-lingual masked token labels in the pre-training data for the variants. In particular, 34% of the labels of the masked tokens of the DICT-MLM-50 dataset, belong to a different language than the sentence. Likewise this fraction is 55% and 82% for DICT-MLM-70 and DICT-MLM-90 re-spectively. 10 Based on the results, such as those in Table 1, we find that the DICT-MLM-50 and DICT-MLM-70 consistently score higher than the DICT-MLM-90 variant across all tasks. This confirms our intuition for the need to multi-task with regular MLM. By relying so heavily on predicting cross-lingual synonyms largely (82% of masked tokens), DICT-MLM-90 does worse than the other variants on the downstream tasks which are all monolingual texts -something it sees only 10% of the time. This pretraining-downstream mismatch coupled with not learning from enough monolingual examples, is likely why it trails the other two.

TASK
We also find the DICT-TLM variant to be fairly competitive in some tasks, most notably the POStagging ones. We believe this is due to a notable downside of our setup affecting the DICT-MLM variants. Among all of the evaluated tasks, POS tagging is the most reliant on lexical cues. If is often the case that the internal structure (morphology) of the individual tokens is sufficient to correctly predict its POS value. However as part of our cross-lingual synonym prediction task in the DICT-MLM variants, we sample synonyms randomly i.e., we do not consider the morphology of the word or the surrounding context. Since most publicly released dictionaries only have a single morphological form annotated, this can lead to grammatically incorrect substitutions or a loss of morphological information. For instance, in Table 4 we find that SOURCE TARGET pt: andar it:camminare, es:piso, en:walking, en:walk no: vokal en:vowels, en:vowel ms: cubaan en:attempt, en:attempting, en:attempted, en:testing, en:attempts Table 4: Example dictionary entries show that a given source word has multiple synonyms across different languages. The same source word also has synonyms across different morphological forms of which one form is randomly selected. The language codes are -pt: Portuguese, no: Norwegian, ms: Malay, it: Italian, es: Spanish, en: English. ) performance across the different pre-trained models grouped by Wikipedia size for a) XNLI and b) NER task. Note higher scores are better. For XNLI we plot difference in the accuracy and for NER difference in the F1 score with respect to mBERT on y-axis.
the Norwegian word for 'vokal' (vowel) is mapped to both singular and plural form of the respective English word which is incorrect since Norwegian has a different word 'vokaler' for the plural form (vowels). We posit that cleaning up the lexicons and a more careful selection of synonyms can help alleviate this issue. We further compare the performance of the model variants with respect to the pre-training corpus size (wikipedia-size) of the different languages. We group languages by their Wikipedia size and use the same grouping as Wu and Dredze (2020a). In Figure 4 we present results for two tasks, NER and XNLI. 11 First, we observe that the difference between the model variants is more apparent for NER than for XNLI. This is because NER being a sequence labeling task, the fine-tuned model for NER relies heavily on token-level representations as compared to XNLI which uses the full sequence representation ([CLS] token) for final prediction. This probably leads the sequence labeling models to be more sensitive to the different variants.
Specifically for NER and POS, we observe that languages with small wikipedia sizes are bene-11 Plots for other tasks can be found in the Appendix. fited more which suggests that the additional crosslingual supervision in the form of bilingual dictionary is particularly helpful for low to mid-resource languages. Further, we find that for a given task, the proposed methods perform consistently across all language families i.e. in most cases the different model variants have a similar rank order with respect to their performance across the language families. This is promising as researchers do not need to pre-train multiple model variants for a given task saving much time and computation effort.
Is the language conditioning layer in DICT-MLM* necessary? In Section §2.2, we discussed two architectural changes to the mBERT model where one of them was adding a language conditioning dense layer for the (dict) MLM prediction task. Specifically, we concatenated the associated language's embedding with the token's embedding from the final transformer layer to aid in the crosslingual synonym prediction. To evaluate the effect of the conditioning layer, we conduct an ablation study by removing the conditioning layer and report the results on five downstream tasks for the DICT-MLM-50 model variant in Table 5. We  find that for NER, removing the conditioning layer causes a significant drop in the performance as compared to XNLI and MLDOC-H. For POS and PAWS-X we observe a slight increase in performance upon removing the language conditioning layer. We do note that  is the best performing model variant for NER which could suggest that the conditioning layer is indeed providing additional performance gain. However, we leave this for future work to further investigate this observation.

Related Work
There has been a growing trend towards using explicit alignment objectives to encourage crosslinguality in word representations. Upadhyay et al. (2016) show that training word representations with different levels of cross-lingual alignment supervision improves downstream performance on both semantic and syntactic tasks. They use supervision in the form of parallel text with word alignments, an expensive cross-lingual resource, to predict words cross-lingually. They also find that for syntactic tasks such as dependency parsing, a weaker supervision signal in the form of bilingual dictionaries is competitive with other alignment objectives which require parallel text. Recently, such alignment objectives have also been incorporated with pre-trained models. The most recent work is perhaps by Wu and Dredze (2020b) which proposes various methods to align the source and target word representations using parallel text and observe slight improvements over mBERT. However, for a larger capacity model such as XLM-R they do not observe significant gains under an extensive evaluation setup. XLM (Lample and Conneau, 2019) also uses parallel data with the MLM objective to implicitly encourage cross-lingual alignment. In contrast, our work demonstrates significant improvements over existing works while using only bilingual dictionaries which are more easily available than parallel text.
Bilingual dictionaries have been used for training task-specific models such as in Liu et al. (2019b) where they use an attention matrix to select the words for dictionary replacement and create code-mixed sentences. Parallel to our work Qin et al. (2020) also propose to use bilingual dictionaries to construct code-switched sentences for training downstream models. Qin et al. (2020) follow a similar approach as ours of random sampling both word and their synonyms. Our work differs from Qin et al. (2020); Liu et al. (2019b) in two aspects: 1) we pre-train our model from scratch whereas Qin et al. (2020) fine-tune the mBERT model on the multilingual code-switched sentences, and 2) we propose two architectural changes to the pre-trained model to provide additional language signals wherein we add language embeddings to the input layer and a language conditioning dense layer after the transformer stack.

Conclusions and Future Work
In this work, we presented pre-training methods targeted towards improving the cross-lingual representation learning of BERT-based models. We propose using bilingual dictionaries for this purpose since they are a relatively cheap cross-lingual resource and easily available for several languages. We find that our proposed methods outperform existing work which use stronger cross-lingual supervision in the form of parallel text, on six tasks in the difficult zero-shot setting. Furthermore, we find that languages which do not have dictionary resources available are also benefited by our proposed pre-training methods making this method applicable to even endangered languages which might not have dictionaries easily available.
One limitation of our work is that during the training data generation, the cross-lingual synonyms are selected randomly and out of context which often results in ungrammatical sentences be-ing generated. We plan to address this problem in our future work. We also plan to expand our methodology to leverage alternative resources such as multilingual word embeddings to further improve the learned representations.