{'fraction_non_alphanumeric': 0.02956120092378753, 'fraction_numerical': 0.01257010887495876, 'mean_word_length': 5.070698978569998, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Combating the Curse of Multilinguality in Cross-Lingual WSD by Aligning Sparse Contextualized Word Representations

In this paper, we advocate for using large pre-trained monolingual language models in cross lingual zero-shot word sense disambiguation (WSD) coupled with a contextualized mapping mechanism. We also report rigorous experiments that illustrate the effectiveness of employing sparse contextualized word representations obtained via a dictionary learning procedure. Our experimental results demonstrate that the above modifications yield a significant improvement of nearly 6.5 points of increase in the average F-score (from 62.0 to 68.5) over a collection of 17 typologically diverse set of target languages. We release our source code for replicating our experiments at https://github.com/begab/sparsity_makes_sense.


Introduction
Word sense disambiguation (WSD) is a longstanding and fundamental problem of Natural Language Processing, known to be affected by the knowledge acquisition bottleneck (Gale et al., 1992). Large pre-trained neural language models are known to effectively mitigate the problems related to the paucity of high quality, large-coverage sense annotated training data for WSD (Loureiro and Jorge, 2019;Loureiro et al., 2021b;inter alia).
Most recently, the knowledge acquisition bottleneck has been identified as an immense problem in the cross-lingual setting as well (Pasini, 2020). A straightforward solution for handling this problem is to apply large multilingual pre-trained language models in a zero-shot setting, however, this approach has a potential limitation owing to the curse of multilinguality (Conneau et al., 2020a), i.e., the inability of such models to handle the large number of languages involved during training such models to an equally good quality.
With the prevalence of large monolingual pretrained models, the important research question arises if their language-specific nature can be successfully exploited during zero-shot learning. Our research provides a thorough comparison of the application of large multilingual and monolingual pre-trained language models for zero-shot WSD.
Another crucial aspect that we carefully investigate in this paper is the integration of sparse contextualized word representations into cross-lingual zero-shot WSD. Sparse word representations have a demonstrated ability to align with word senses (Balogh et al., 2020;Yun et al., 2021). While the benefits of employing sparsity has been shown for WSD in English (Berend, 2020a), its viability in the cross-lingual setting has not yet been verified.
In order to conduct such an analysis, we propose an algorithm for obtaining cross-lingual sparse contextualized word representations from independently trained monolingual language models.

Related work
The analysis and the investigation of the transfer capabilities of large pre-trained language models (such as mBERT or XLM) across languages has spurred significant research interest (Pires et al., 2019;Dredze, 2019, 2020;K et al., 2020). In contrast to the availability of multilingual neural language models, a series of recent papers have argued for the creation of dedicated neural language models for different languages (see e.g. Table 1). While monolingual neural language models can more accurately model the distinct languages, models that are trained in isolation of other languages cannot directly benefit from downstream application-specific annotated training data available in different languages. Artetxe et al. (2020) proposed an approach for making monolingual models compatible with each other by first pre-training a masked language model on a source language, then freezing its parameters apart from its embedding layer that get replaced and trained for additional target languages using a standard masked language modeling objective. Note that this approach is complementary and strictly more resource intensive to ours, as it involves the pre-training of a (freezed) transformer model with respect its embedding layer for a target language. In contrast, our approach can operate on monolingual language models fully pre-trained in total isolation from the source language encoder. Also, our approach learns substantially fewer parameters in the form of an alignment matrix between the hidden representations of the contextualized target and source language spaces. Conneau et al. (2020b) analyzed the multilingual patterns emerging in large pre-trained language models. The authors found that "language universal representations emerge in pre-trained models without the requirement of any shared vocabulary or domain similarity". That work have demonstrated that monolingual BERT models can be effectively mapped for performing zero-shot crosslingual named entity recognition and syntactic parsing. Similarly, Wang et al. (2019); Schuster et al. (2019) also illustrated the efficacy of linear transformations for using BERT-derived representations in cross-lingual dependency parsing.
WSD has been a fundamental and challenging problem in NLP for many decades, dating back to (Weaver, 1949(Weaver, /1955. The utilization of contextualized word representations was first advocated by Peters et al. (2018), later popularized by (Loureiro and Jorge, 2019;Loureiro et al., 2021a). Bevilacqua et al. (2021) offers a survey of the recent approaches.
Most recently, Rezaee et al. (2021) have explored the usage of multilingual language models (XLM) in zero-shot WSD. While the experiments in (Rezaee et al., 2021) cover four related target languages (German, Spanish, French and Italian), our investigation involves a typologically diverse set of 17 target languages (beyond English) from (Pasini et al., 2021). Our work also extends that line of research in important aspects, as we show that the application of monolingual neural language models can vastly improve the performance of crosslingual zero-shot WSD. Additionally, we also provide a careful evaluation of sparse contextualized word representations in zero-shot WSD.
Berend (2020a) introduced sparse contextualized word representations via the application of dictionary learning, and showed that sense representations that are obtained from the co-occurrence statistics of the sparsity structure of the contextualized word representations and their sense annotations can provide significant improvement in monolingual WSD. Our work relates to that line of research by providing a mapping-based procedure, which enables the usage of such sense representations created in some source language to be applied in other target languages as well. The kind of mapping we employ can be viewed as a generalization of the approach introduced in (Berend, 2020b) with the notable exception that in this work, we obtain sparse word representations for contextualized models as opposed to static word embeddings.

Methodology
In order to allow for zero-shot transfer between monolingual language models pre-trained in isolation from each other, we need to determine a mapping between their hidden representations. We first introduce our methodology for doing so, then we integrate this to the creation of sparse contextualized word representations.

Mapping hidden representations
The alignment of word representations between independently constructed semantic spaces can be conveniently and efficiently performed via linear transformations. This has been a standard approach for non-contextualized word embeddings (Mikolov et al., 2013;Xing et al., 2015;Smith et al., 2017), but it has been shown to be useful in the contextualized case as well (Conneau et al., 2020b).
The standard approach is to obtain a collection of pairs of anchor points {x i , y i } n i=1 with x i and y i denoting the representation of semantically equivalent words in the target and source languages, respectively. The mapping W is then obtained as As we deal with contextualized models, we can obtain various representations for a word even in the same context, by considering the hidden representations from different layers of the neural language models employed. Additionally, as constraining the mapping matrix to be an isometric one have proven to be a useful requirement, we define our learning task to be of the form with I denoting the identity matrix, x (lt) i and y (ls) i denoting the hidden representations obtained from the l t th and l s th layers of the target and source language neural language models, respectively.
Finding the optimal isometric W can be viewed as an instance of the orthogonal Procrustes problem (Schönemann, 1966) which can be solved by W ⊥ = U V , with U and V originating from the singular value decomposition of the matrix product Y ⊺ X, where X and Y include the stacked target and source language contextual representations of pairs of semantically equivalent words.
As words of the input sequences to the neural language models can be split into multiple subtokens, we followed the common practice of obtaining word-level neural representations by performing mean pooling of the subword representations. Throughout our experiments, we also relied on the RCSLS criterion (Joulin et al., 2018), which offers a retrieval-based alternative of obtaining a mapping from the target to the source language representations.

Cross-lingual sparse contextualized word representations
Our approach extends the information theoretic algorithm introduced in (Berend, 2020a) for its application in the cross-lingual zero-shot WSD setting. In order to obtain sparse contextualized representations for the source language, we first populate Y ∈ R d×N with d-dimensional contextualized representations of words determined for texts in the source language, and minimize the objective where C denotes the convex set of d × k matrices with column norm at most 1, λ is a regularization coefficient and the sparse coefficients in α are required to be non-negative. We used the SPAMS library (Mairal et al., 2009) for calculating D and α.
Having obtained D for the source language, we determine a sparse contextualized word representation for a target language word with dense contextualized representation x i as where W is the alignment transformation as described earlier in Section 3.1. Eq. (4) reveals that the cross-lingual applicability of the sparse codes are assured by the mapping transformation W and the fact that the sparse target language representations are also using the same D that was determined for the source language, which also ensures the efficient calculation of sparse representations during inference time. Apart from these crucial extensions we made for providing the use of contextualized sparse representations in the cross-lingual setting, the way we utilized them for the determination of sense representation and inference is identical to (Berend, 2020a). That is, for all sense-annotated words in the training corpus, we calculated a weighted cooccurrence statistics between a word pertaining to a specific semantic category and having non-zero coordinates along a specific dimension in their sparse contextualied word representations. These statistics are then transformed into pointwise mutual information (PMI) scores, resulting in a sense representation for all the senses in the training sense inventory.
Sense representations obtained that way measure the strength of the relation of the senses to the different (sparse) coordinates. Inference for a word with sparse representation α is simply taken as arg max s Φα ⊺ , where Φ is the previously defined matrix of PMI values and s corresponds to the sense at which position the above matrix-vector products takes its largest value.

Experimental results
All the neural language models that we relied on during our experiments were obtained from the transformers library (Wolf et al., 2020). We used four NVIDIA Titan 2080 GPUs for our experiments.
As the multilingual language model, we used the 24-layer transformer architecture, XLM-RoBERTa (XLM-R for short) (Conneau et al., 2020a). We chose the cased BERT (Devlin et al., 2019) large model as the monolingual model for encoding English text. As for the rest of the monolingual language models involved in our experiments, we relied on the models listed in Table 1. These monolingual models have the same size as the BERT-base model, i.e., they consist of 12 transformer blocks and employ hidden representations of 768 dimensions.
For evaluation purposes, we used the extra-large cross-lingual evaluation benchmark XL-WSD, recently proposed in (Pasini et al., 2021). The database contains a high-quality sense annotated corpus for English as the concatenation of the Sem-Cor dataset (Miller et al., 1994) and the sense definitions and example sentences from WordNet (Fellbaum, 1998). XL-WSD uses the unified crosslingual sense inventory of BabelNet (Navigli and Ponzetto, 2012).
The dataset contains 17 additional typologically diverse languages besides English (that we listed in Table 1). The authors also released machine translated silver standard sense annotated training corpora for all the languages, which makes the language-specific fine-tuning of monolingual models possible, however, as shown in (Pasini et al., 2021), that approach resulted in inferior results compared to the application of multilingual models in the zero-shot setting.
Throughout the application of sparse contextualized representations, we employ the same set of hyperparameters that were used in (Berend, 2020a), i.e., we set the number of the regularization coefficient to λ = 0.05 and the number of (sparse) coordinates to k = 3000. There made one optional change, i.e., we decided whether to use the normalization of PMI values (Bouma, 2009) during the calculation of the sense representation matrix Φ on a per language basis based on development set performances. An ablation study related to the (optional) normalization of PMI scores is reported in Table 5, Appendix B.
When we do not employ the sparsification of the contextualized word representations for determining the sense representations, we follow the approach introduced in (Loureiro and Jorge, 2019). That is, we take the centroid of word vectors belonging to a particular sense as the representation of that sense, and perform a nearest neighbor search during inference.

Alignment of contextualized representations
As the different layers of neural language models have been shown to provide different levels of utility towards different tasks, we experimented with mappings between different combinations of layers from the target and source language neural language models. Since the last few layers of the neural models are generally agreed to be the most useful for semantics-related tasks (Peters et al., 2018;Tenney et al., 2019;Reif et al., 2019), we decided to learn mappings between the hidden representations of any of the last four layers of the target and source language encoders. We used BERT as the language specific encoder for the source language texts in English, but we also investigated the application of XLM-R, so that we can see the effects of replacing it by an encoder especially tailored for English. As for the target languages, we used the respective models for each language as listed in Table 1. Similar to the source language, we also investigated the case when target languages were encoded by the multilingual model.
In what follows, we label the different experimental settings according to the followings: • multi→multi means that we map the target language representations obtained by the multilingual (XLM-R) model to the representation space of the source language also obtained by the multilingual (XLM-R) encoder, • multi→mono, means that we map the target language representations obtained by the multilingual (XLM-R) model to the representation space of the source language obtained by the monolingual (English BERT) encoder, • mono→multi, means that we map the target language representations obtained by their respective monolingual language model to the representation space of the source language obtained by the multilingual (XLM-R) encoder, • mono→mono, means that we map the target language representations obtained by their respective monolingual language model to the representation space of the source language obtained by the monolingual (English BERT) encoder.
In order to obtain the cross-representational mappings, we accessed the Tatoeba corpus (Tiedemann, 2012) through the datasets library (Lhoest et al., 2021). The Tatoeba corpus contains translated sentence pairs for several hundreds of languages which we used for obtaining the pivot word mention pairs together with their contexts.
In addition to the Tatoeba corpus, we used the word2word library (Choe et al., 2020) containing word translation pairs between more than 3,500 language pairs. By denoting (S s i , S t i ) the i th translated sentence pair from the Tatoeba corpus, we treated those (w s ∈ S s i , w t ∈ S t i ) word occurrences as being semantically equivalent, for which the w t ∈ T ranslationOf (w s ) and the w s ∈ T ranslationOf (w t ) relations simultaneously held according to the translation list provided by word2word.
As an example, given the German-English translation pair from Tatoeba, {'de:' 'Es steht ein Glas auf dem Tisch.', 'en': 'There is a glass on the table.}, underlined pairs of words with the same color would be treated as contextualized translation pairs of each other.
One benefit of our approach for determining contextual alignment of word pairs is that it does not require word level alignment of the parallel sentences, hence it suits such lower resource scenarios better, when only parallel sentences (without word level alignments) and a list of word translation pairs are provided. Naturally, different contextual alignment approaches could be integrated into our approach at this point, and this is something that we regard as potential future extension of our work.
We evaluated the quality of the mapping learned between the target and the source language representations by defining a contextualized translation retrieval task and evaluating it on its accuracy@1  metric, i.e., for what fraction of the contextualized translation pairs -not seen during the determination of the mapping between the two representation spaces -are we able to rank the original translated context as the highest.
In the multi→multi case, i.e., when both the target and source languages are encoded by the same multilingual model (XLM-R), it also makes sense to use the identity matrix as the mapping operator for mapping the target language contextual text representations to the semantic space of the source language (as long as the target and source language texts are obtained from the same layer of the multilingual encoder). We also evaluated the quality of this approach in our experiments that we refer to as the identity approach.
We list the statistics of the Tatoeba corpus and the size of the training and test contextualized translation pairs in Table 2. Our results on the top-1 contextualized translation retrieval accuracies along the different languages and combination of target and source encoder usage are reported in Figure 1. The quality of the combination which uses monolingual encoders for both the target and source languages (mono→mono) performed the best.

Monolingual evaluation
We first conducted evaluations in the monolingual setting, i.e., we used the sense annotated training data to train and evaluate WSD models in English.
The results of these experiments -depending on the encoder architecture used (BERT/XLM-R), the layer of the encoder utilized ({21,. . . ,24}), and whether the sparsification of the contextualized representations took place (Dense/Sparse) -are included in Table 3. Unsurprisingly, the application of the languagespecific BERT model achieved better scores compared to that of XLM-R. An interesting observation though, is that the drop in performance is much more subtle for those cases when the contextualized representations are enhanced via sparsification, i.e., the typical loss in performance across the layers is only 3 points (apart from the final layer), opposed to the typical loss of 4-7 points in the dense case. Table 4 includes the zero-shot cross-lingual WSD results for a collection of baseline approaches (Table 4a) from (Pasini et al., 2021), followed by our models not utilizing the sparsification of the contextualized embeddings (Table 4b) and the ones that additionally benefit from sparsification as well (Table 4c). It is useful to note that the mono→* approaches are strictly more resource efficient during inference as they are based on 12-layer encoders instead of the 24 layers of the multilingual XLM-R model.

Cross-lingual zero-shot evaluation
At this point, we separate the multi→multi results into two, i.e., 1) those obtained when relying on the hidden representations from the same layer of XLM-R without mapping (or equivalently, with the identity mapping from the target to source representations); and 2) those obtained when the target and source language contextual representations could originate from different layers of the XLM-R encoder, and a non-identity (either isometric or RC-SLS) mapping was employed. We keep referring to the latter as multi→multi, and denote the former type of experiments as multi (without the →multi suffix as there were no real mappings performed in these cases). Inspecting the first two rows of Table 4b and Table 4c reveals that enhancing the multilingual encoder towards the treatment of a particular pair of languages by providing it a language pair specific mapping has a larger positive effect when using dense vectors. In fact, it increased the micro-averaged F-score over the 17 languages by 1.72 and 0.11 points for the dense and the sparse cases, respectively.
Overall, the micro-averaged F-score of our final approach managed to improve nearly 6.5 points (cf. the first row of  to the replacement of the XLM-R encoder for both the source language during training and target languages for inference (cf. the first and last row of Table 4b) and an additional 1.5 points of improvement was an effect of our sparsification in the crosslingual setting. The inspection of the third and fourth rows in both Table 4b and Table 4c reveals that using a monolingual encoder during inference helps more compared to the application of a monolingual encoder for encoding the source language during training.
We conducted the McNemar test between our system outputs when a non-identity mapping was used between a pair of languages. Our investigation revealed that all such 8 2 pairs of system outputs from Table 4b and Table 4c differ significantly from each other with p < 0.0007, with only four exceptions, i.e, 1) multi→multi and multi→mono from Table 4b; 2) multi→multi and multi→mono from Table 4c; 3) mono→multi from Table 4c and  mono→mono from Table 4b; 4) multi→mono from Table 4c and mono→multi from Table 4b. Figure 2 summarizes the results of all the possible runs conducted. When using the multilingual encoder for both the target and source languages without a mapping step between the two (multi), we ran 4 different experiments per each language based on the hidden representations obtained from in 2 × 32 experiments all together. Our ablation study in Table 5 illustrates that this extra factor of 2 for the sparse experiments did not provided us an unfair advantage, i.e., when fixing the value of normalization in any way, the overall results did not differ substantially. The difference in the average performance of our approach transforming sparse contextualized representations obtained by monolingual models is significant (using unpaired t-test 2 , p < 0.005) compared to any other configuration. This suggests that the mono→mono approach has a robust advantage over alternative variants, and the improvements seen in Table 4 are not an effect of careful hyperparameter selection, but they generalize over a wide range of choices.
This effect is further corroborated in Figure 3, which offers a comparison between the two systems with the best average performance, i.e., mono→mono that operates with the dense vectors (results are along the x-axis) and the same model but with the enhancement of sparsification (results are along the y-axis). Each data point corresponds to a setting with the same hyperparameter choices, and points above the diagonal line with slope one demonstrate the benefits of sparsification.
We have demonstrated the improved utility of mapping language-specific sparse contextualized representations for conducting zero-shot WSD, requiring large pre-trained language-specific text encoders for the target languages. While such models are available for all languages in XL-WSD, a vari-ety of the existing languages lack their dedicated language-specific pre-trained language model. As such, an important question emerges whether it is possible to enjoy the benefits of mapping sparse contextualized representations for zero-shot WSD in the absence of a large pre-trained language model dedicated to the target language. To this end, we shall inspect the results of our multi→mono approach in Table 4, a series of mapping-based experiments in which we acted as if the monolingual language models (other than the one for English) did not exist. In these experiments, the sense embeddings were obtained with bert-large-cased (being specialized to English), and the mapping to the non-English target languages were performed towards their XLM-R representations during the evaluation. This way, we could simulate the effects of the absence of language-specific models.
The multi→mono approach provided a substantially better average performance compared to the mere utilization of a multilingual encoder in the case of dense contextualized representations as it can be seen in Table 4b. The average results of multi→mono are slightly inferior (albeit statistically insignificantly) to that of the multi approach for the application of sparse contextualized representations. However, when comparing the multi→multi results with that of multi→mono, we can see that by relying on a multilingual encoder alone, and allowing a mapping to be employed between its hidden representations pertaining to different languages, one can obtain the same (or even slightly better) performance as with the multi→mono approach. This highlights the importance of monolingual encoders for the target language, which seems to be more important than having access to a monolingual encoder for the source language.

Conclusions
In this paper we provided a systematic investigation of the benefits of using large monolingual pretrained language models in place of multilingual language models, such as XLM-R. We have shown that since monolingual neural language models are specifically tailored for a single (or at most a few related) languages, they can effectively mitigate the curse of multilinguality typical of multilingual models, and their application can significantly improve the F-scores in zero-shot WSD. We additionally showed that the benefits of sparse con-textualized word representations, obtained via a dictionary learning procedure, also convey to the cross-lingual setting, and that it provides complementary improvements to the usage of monolingual neural language models.

A Analysis of the language models
We compare some of the basic properties of the pretrained language models that we employed in Figure 4 and Figure 5. This can be useful as the monolingual quality of the language models we used could influence and account for their utility when used in conjunction with our mapping-based algorithm.
Figure 4 includes quantitative scores over the different languages related to the subword tokenizers employed by the various language models. Fertility in Figure 4a refers to the average number of subtokens a single token gets separated into by the tokenizer of the given language model. Multi-token ratio (MTR) in Figure 4b indicates the fraction of tokens that gets split into more than one piece upon tokenization (Ács, 2019;Rust et al., 2021). Smaller values of MTR mean a better adaptation of the tokenizer to the peculiarities of the given language. It can be seen that the monolingual models do a much better job compared to XLM-R, which can be part of the reason why mapping independently trained monolingual .
In Figure 5a, we refer to the last four layers of the investigated models as {-4,-3,-2,-1} as the English BERT is a 24-layer model, whereas the rest of the monolingual models consist of 12 layers. This means that layer -1 refers to layer 24 for English and layer 12 for some non-English model. Even though Figure 5a shows pathological masked language modeling (MLM) losses for certain monolingual models (e.g. Bulgarian or Basque) when measured on the XL-WSD database, their mappingbased utilization in zero-shot WSD was still possible as indicated by our main results (see Table 4). A further interesting phenomenon is that the performance of XLM-R exceeds that of the bert-largecased model in terms of MLM for English. These results suggest that the masked language modeling performance of pretrained language models and their utility in WSD are not strongly related with each other.

B Analysis on using the normalization of PMI scores
Upon the calculation of the sense representation matrix Φ, involving the calculation of PMI scores between the various senses from the sense inventory and the coordinates of a sparse contextual representation being non-zero, Berend (2020a) suggested the use of normalized PMI scores (Bouma, 2009). Our preliminary results suggested that the normalization of PMI scores can have a mixed effect over the different languages. Table 5 includes a detailed breakdown on this effect for the individual languages.  (c) Our results based on sparse sense vectors when the normalization of PMI scores was optional and based on the development set for each language. Table 5: The effects of making the normalization of PMI scores (Bouma, 2009) (a) mandatory, (b) prohibited, (c) optional to use (based on development set results) during the creation of the sparse sense representations.