{'fraction_non_alphanumeric': 0.04742843978575131, 'fraction_numerical': 0.015435120658872314, 'mean_word_length': 4.950479780671693, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Probing Cross-Lingual Lexical Knowledge from Multilingual Sentence Encoders

Pretrained multilingual language models (LMs) can be successfully transformed into multilingual sentence encoders (SEs; e.g., LaBSE, xMPNet) via additional fine-tuning or model distillation with parallel data. However, it remains unclear how to best leverage them to represent sub-sentence lexical items (i.e., words and phrases) in cross-lingual lexical tasks. In this work, we probe SEs for the amount of cross-lingual lexical knowledge stored in their parameters, and compare them against the original multilingual LMs. We also devise a simple yet efficient method for exposing the cross-lingual lexical knowledge by means of additional fine-tuning through inexpensive contrastive learning that requires only a small amount of word translation pairs. Using bilingual lexical induction (BLI), cross-lingual lexical semantic similarity, and cross-lingual entity linking as lexical probing tasks, we report substantial gains on standard benchmarks (e.g., +10 Precision@1 points in BLI). The results indicate that the SEs such as LaBSE can be 'rewired' into effective cross-lingual lexical encoders via the contrastive learning procedure, and that they contain more cross-lingual lexical knowledge than what 'meets the eye' when they are used as off-the-shelf SEs. This way, we also provide an effective tool for harnessing 'covert' multilingual lexical knowledge hidden in multilingual sentence encoders.

Following the procedures from monolingual setups, another line of research has been transforming multilingual LMs into multilingual SEs (Feng et al., 2022;Reimers and Gurevych, 2020), which enable effective sentence matching and ranking in multiple languages as well as cross-lingually (Litschko et al., 2022). The transformation is typically done by coupling 1) LM objectives on monolingual data available in multiple languages with 2) cross-lingual objectives such as Translation Language Modeling (TLM) (Conneau and Lample, 2019) and/or cross-lingual contrastive ranking (Yang et al., 2020). Such multilingual SEs consume a large number of parallel sentences for the latter objectives. Consequently, they outperform multilingual off-the-shelf LMs in cross-lingual sentence similarity and ranking applications (Liu et al., 2021d;Litschko et al., 2022). However, as we show in this work, such multilingual SEs may still lag behind traditional static cross-lingual word embeddings (CLWEs) when encoding sub-sentence lexical items (e.g., words or phrases) (Liu et al., 2021c) for cross-lingual lexical tasks (e.g., BLI).
In this work, we probe multilingual SEs for cross-lingual lexical knowledge. We demonstrate that, due to their fine-tuning on multilingual and parallel data, they indeed store a wealth of such knowledge, much more than what 'meets the eye' when they are used 'off the shelf'. However, this lexical knowledge needs to be exposed from the original multilingual SEs, (again) through additional fine-tuning. In other words, we show that multilingual SEs can be 'rewired' into effective cross-lingual lexical encoders, as illustrated in Figure 1. This rewiring is again done via a quick and inexpensive contrastive learning procedure: with merely 1k-5k word translation pairs, we successfully convert multilingual SEs into state-of-the-art bilingual lexical encoders for any language pair. 1 We probe the original LMs and SEs as well as demonstrate the usefulness of the proposed contrastive procedure for 'exposing' cross-lingual lexical knowledge on three standard lexical crosslingual tasks using standard evaluation data and protocols: BLI, cross-lingual lexical semantic similarity (XLSIM), and cross-lingual entity linking (XL-EL). We show that the 'exposure' procedure is highly effective for both vanilla multilingual LMs (mBERT and XLM-R) and multilingual SEs (LABSE and XMPNET): e.g., we observe ≈+10 Precision@1 points gains on standard BLI benchmarks . Multilingual SEs offer substantially better cross-lingual lexical performance than vanilla LMs, both before and after being subjected to contrastive cross-lingual lexical fine-tuning (see Figure 1). This indicates that multilingual SEs have more cross-lingual lexical knowledge than their vanilla LM counterparts, likely owing to their additional exposure to parallel data.
Finally, inspired by Li et al. (2022), we validate that word vectors produced by cross-lingual lexical encoders (i.e., after contrastive cross-lingual lexical 'exposure') can be effectively interpolated with static CLWEs (Artetxe et al., 2018) and offer even stronger performance in cross-lingual lexical tasks. Encouragingly, our cross-lingual lexical specialization of multilingual SEs (as well as the further interpolation with static CLWEs), yields particularly massive performance gains for pairs of low-resource languages, as demonstrated on the low-resource BLI benchmark .

From Multilingual Sentence Encoders to Cross-Lingual Lexical Encoders
Motivation. The motivation for this work largely stems from the research on probing and analyzing 1 Note that this is a typical requirement of standard mapping-based approaches for learning static cross-lingual word embeddings which excel in the BLI task (Mikolov et al., 2013;Conneau et al., 2018;Glavaš and Vulić, 2020).  Figure 1: Illustration of the pipeline of exposing crosslingual lexical knowledge from multilingual language models (LMs) and sentence encoders (SEs) ( §2). Multilingual LMs (x) can be transformed into multilingual SEs (y) as done in previous work (Reimers and Gurevych, 2020;Feng et al., 2022). A contrastive crosslingual lexical fine-tuning procedure (z) (requiring an external bilingual dictionary) can be applied on both x and y, yielding a fine-tuned cross-lingual lexical encoder ({). At inference, a word/phrase is encoded by the lexical encoder (|). In addition, its encoding can be interpolated with the corresponding static (crosslingual) word embedding (}), producing the final embedding of the word/phrase (~). Before the interpolation, static CLWEs must be mapped into the vector space of the lexical encoder ({): to this end, we learn the standard orthogonal (Procrustes) projection matrix.
pretrained LMs for various types of knowledge they (implicitly) store in their parameters (Ethayarajh, 2019;Jawahar et al., 2019;Rogers et al., 2020). In this paper, we focus on a particular knowledge type: cross-lingual lexical knowledge, and its extraction from multilingual LMs and SEs. Previous work tried to prompt multilingual LMs for word translations via masked natural language templates (Gonen et al., 2020) and extract typelevel word embeddings from LMs (i) directly without context (Vulić et al., 2020a(Vulić et al., , 2021a or (ii) by averaging contextual embeddings over a large auxiliary corpus in the target language (Bommasani et al., 2020;Litschko et al., 2022). This existing body of work 1) demonstrated that even sophisticated templates and extraction strategies cannot outperform cross-lingual word embedding spaces (e.g., induced from monolingual fastText vectors) in cross-lingual lexical tasks such as BLI (Vulić et al., 2020b) and 2) did not attempt to expose crosslingual lexical knowledge from multilingual SEs and compare it against the (same type of) knowl-edge extracted from vanilla multilingual LMs.
Multilingual Sentence Encoders. Off-the-shelf LMs contextualize (sub)word representations, but are unable to encode the precise meaning of input text out of the box. SEs -LMs fine-tuned via sentence-level objectives -in contrast, directly produce a precise semantic encoding of input text. A large body of work focuses on inducing multilingual encoders that capture sentence meaning across languages (Artetxe and Schwenk, 2019;Feng et al., 2022;Yang et al., 2020, inter alia).
The most popular approach obtains multilingual SEs (Reimers and Gurevych, 2020) by distilling the knowledge from the monolingual English SE teacher (trained on English semantic similarity and NLI data) into multilingual LM student (e.g., mBERT), using parallel sentences to guide the distillation process. SEs, being specialized for sentence similarity, encode sentence meaning more accurately and are useful in various (unsupervised) text similarity and ranking tasks, monolingually and across languages.
While SEs' primary purpose is sentence encoding, they can, in principle, be applied to subsentential text: words and phrases. In this work, we show that multilingual SEs can be turned into effective cross-lingual lexical encoders. We achieve this through additional cross-lingual lexical fine-tuning (Vulić et al., 2021a), requiring as supervision only a small set of word translation pairs.

Cross-Lingual Lexical Fine-Tuning
For a given language pair L s -L t , our contrastive cross-lingual lexical fine-tuning of multilingual encoders (LMs and SEs alike) requires a dictionary spanning N (typically N ≤ 5,000) word translation pairs, We consider the translation pairs from D to be positive examples for the contrastive fine-tuning procedure. For each of the N source language words in the dictionary (w i,s ), we precompute a set of N n hard negative samples: these are the L t words that are the closest to w i,s in the representation space of the multilingual encoder, but not its direct translation v i,t . Let f θ (·), be the encoding function of the multilingual LM/SE, with θ as parameters, and let S(·, ·) be a function of similarity between two vectors. For a source word w i,s , we select as hard 2 Note that such bilingual dictionaries are one of the most widespread and cheapest-to-obtain resources in multilingual NLP . negatives words v t from L t that have the highest S(f θ 0 (w i,s ), f θ 0 (v t )) score, with θ 0 as the original encoder's parameters, before fine-tuning.
We encode all training words -those from the seed dictionary D and (at most) N · N n precomputed hard L t negatives -independently and in isolation. Concretely as encoder's sequence start and end tokens, resp.), and take the average of the transformed representation (from the last Transformer layer) of the w's subword tokens as the w's encoding f θ (w). 3 Following common practice in contrastive learning (Henderson et al., 2019;Vulić et al., 2021a), we instantiate S as the scaled cosine similarity: , with C as the scaling constant. We then train in batches of B translation pairs, with the variant of the widely used multiple negatives ranking loss (MNEG) (Cer et al., 2018;Henderson et al., 2019Henderson et al., , 2020 as the fine-tuning objective: where v k,i denotes the k-th hard negative from the language L t for the L s word w i . MNEG combines the N n hard negatives per each positive example with B-1 in-batch negatives (i.e., for a source language word w i,k , each target language word v j,t , j = i from B is used as an in-batch negative of w i,k ). MNEG aims to reshape the representation space of the encoder by simultaneously (a) maximising the similarity for positive pairs -i.e., bringing closer together ('attracting') the words from the positive pairs and (b) minimising the similarity for (both in-batch and hard) negative pairs -i.e., pushing ('repelling') the words from negative pairs further away from each other). 4 Li et al. (2022) recently showed that further performance gains in the BLI task might be achieved by combining the type-level output of the encoding function f with static CLWEs, but they experimented only with multilingual LMs. Static CLWEs and multilingual encoder-based representations of the same set of words can be seen as two different views of the same data point. Following Li et al. (2022), we learn an additional linear orthogonal mapping from the static cross-lingual WE space -e.g., a CLWE space induced from monolingual fastText embeddings (Bojanowski et al., 2017) using VECMAP (Artetxe et al., 2018) -into the cross-lingual space spanned by the multilingual encoder. The mapping transforms 2 -normed d 1 -dimensional static CLWEs into d 2 -dimensional cross-lingual WEs obtained through the multilingual encoder (fine-tuned f θ or original f θ 0 ). Learning the linear map W ∈R d 1 ×d 2 , when d 1 < d 2 , 5 is formulated as a Generalized Procrustes problem (Schönemann, 1966;Viklands, 2006). It operates on all (i.e., both L s and L t ) words from the seed translation dictionary D. 6 Unless noted otherwise, a final representation of an input word w is then computed as follows:

Interpolation with Static CLWEs
where λ is a tunable interpolation hyper-parameter, clwe(w) denotes the static CLWE of w, and f θ (w) the representation of w obtained with the (contrastively fine-tuned or original) multilingual LM/SE. This simple procedure yields an 'interpolated' shared cross-lingual WE space. 4 In practice, we rely on the implementation of the MNEG loss from the SBERT repository www.sbert.net (Reimers and Gurevych, 2019); the default value C = 20 is used. 5 The assumption d1 < d2 typically holds as fastText WEs are 300-dimensional while the dimensionality of standard multilingual LMs and SEs is d2 = 768 or d2 = 1, 024.
6 To learn the mapping W , for pairs from D we decouple Ls words wi,s from their Lt translations vi,t to create vector pairs (clwe(wi,s), f θ (wi,s)) and (clwe(vi,t), f θ (vi,t)) -with clwe(w) as the static CLWE of w (e.g., its VECMAP embedding), and f θ (w) its encoder-based representation -based on which we learn of the orthogonal mapping W (the so-called Procrustes method gives a closed-form solution).

Experimental Setup
Multilingual Sentence Encoders. We probe two widely used multilingual SEs: 1) Languageagnostic BERT Sentence Embedding (LABSE) (Feng et al., 2022) which adapts pretrained multilingual BERT (MBERT) (Devlin et al., 2019) into a multilingual SE; 2) Multilingual XMP-NET is a distillation-based adaptation (Reimers and Gurevych, 2020) of XLM-R (Conneau et al., 2020) as the student model into a multilingual SE, based on the monolingual English MPNet encoder (Song et al., 2020) as the teacher model. LABSE is the current state-of-the-art multilingual SE and supports 109 languages, while XMPNET is the best-performing multilingual SE in the Sentence-BERT repository (Reimers and Gurevych, 2019). 7 Along with LABSE and XMPNET as SEs, we experiment with the original multilingual LMs -mBERT and XLM-R -using the the same training and evaluation protocols (see Figure 1 and §2), aiming to quantify: (i) the extent to which cross-lingual lexical knowledge can be exposed from LMs that have not been specialized for sentence-level semantics, as well as (ii) the increase in quality of lexical knowledge brought about with sentencelevel specialisation (i.e., when multilingual LMs get transformed into multilingual SEs).
Evaluation Tasks. We evaluate on the standard and diverse cross-lingual lexical semantic tasks: Task 1: Bilingual Lexicon Induction (BLI), a standard task to assess the "semantic quality" of static cross-lingual word embeddings (CLWEs) , allows us to 1) directly assess the extent to which cross-lingual word translation knowledge can be exposed from multilingual LMs and SEs and 2) immediately test the ability to transform multilingual sentence encoders into bilingual lexical encoders; We run a series of BLI evaluations on two standard BLI evaluation benchmarks. First, GT-BLI , constructed semiautomatically from Google Translate, comprises 28 language pairs with a good balance of typologically similar and distant languages (Croatian: HR, English: EN, Finnish: FI, French: FR, German: DE, Italian: IT, Russian: RU, Turkish: TR). Second, PanLex-BLI  focuses on BLI evaluation for lower-resource languages, deriving training and test data from PanLex (Kamholz et al., 2014). We evaluate on 10 pairs comprising the following five typologically and etymologically diverse languages: Bulgarian (BG), Catalan (CA), Estonian (ET), Hebrew (HE), and Georgian (KA).
Standard BLI setups and data are adopted: 5k training word pairs are used as seed dictionary D, and another 2k pairs as test data. Note that D is used to (i) contrastively fine-tune multilingual encoders ( §2.1), (ii) learn the (baseline) static VECMAP CLWE space, as well as to (iii) learn the projection between the static CLWE space and the representation spaces of multilingual encoders required to obtain the interpolated representations ( §2.2). The evaluation metric is standard Preci-sion@1 (P@1). 8 For PanLex-BLI, we also run additional experiments using smaller bilingual dictionaries D, spanning 1k translation pairs.
Task 2: Cross-Lingual Lexical Semantic Similarity (XLSIM) is another cross-lingual lexical task: it tests the extent to which lexical representations can capture the (human perception of) fine-grained semantic similarity of words across languages. We use the recent comprehensive XL-SIM benchmark Multi-SimLex (Vulić et al., 2020a), which comprises cross-lingual datasets of 2k-4k scored word pairs over 66 language pairs. We evaluate on a subset of language pairs shared with the GT-BLI dataset: EN, FI, RU, FR. 9 Task 3: Cross-Lingual Entity Linking (XEL) is a standard task in knowledge base (KB) construction (Zhou et al., 2022), where the goal is to link an entity mention in any language to a corresponding entity in an English KB or in a language-agnostic KB. 10 We evaluate on the cross-lingual biomedical entity linking (XL-BEL) benchmark of Liu et al. (2021d): it requires the model to link an entity mention to entries in UMLS (Bodenreider, 2004), a language-agnostic medical knowledge base. We largely follow the XL-BEL experimental setup of Liu et al. (2021d) and probe the encoders first without any additional task-specific fine-tuning on UMLS data, and then with subsequent UMLS fine- 8 We observed very similar performance trends for P@5 and Mean Reciprocal Rank (MRR) as BLI measures. 9 The evaluation metric is the standard Spearman's rank correlation between the average of gold human-elicited XL-SIM scores for word pairs and the cosine similarity between their respective word representations. To avoid any test data leakage, we remove all XLSIM test pairs from the bilingual dictionary D prior to fine-tuning and CLWE mapping. 10 Following prior work (Liu et al., 2021b;Zhou et al., 2022), XEL in this work also refers only to entity mention disambiguation; it does not cover the mention detection subtask. tuning (i) only on the EN UMLS data, (ii) on all the UMLS data in 10 languages of the XL-BEL dataset. 11 Due to a large number of experiments, we again focus on the subset of languages in XL-BEL shared with GT-BLI: EN, DE, FI, RU, TR.
Static CLWEs and Word Vocabularies. As monolingual static WEs, we select CommonCrawl fastText vectors (Bojanowski et al., 2017) of the top 200k most frequent word types in the training data, following prior work on learning static CLWEs (Conneau et al., 2018;Artetxe et al., 2018;Heyman et al., 2019). 12 Static CLWEs are then induced via the standard and popular supervised mapping-based VECMAP method (Artetxe et al., 2018), leveraging the seed dictionary D. These CLWEs are used for interpolation with encoderbased WEs (see §2.2) but also as the baseline approaches for BLI and XLSIM tasks. We compute the type-level WEs from multilingual LMs and SEs for the same 200K most frequent words of each language.
Technical Details and Hyperparameters. The implementation is based on the SBERT framework (Reimers and Gurevych, 2019), using the suggested settings: AdamW (Loshchilov and Hutter, 2018); learning rate of 2e-5; weight decay rate of 0.01. We run contrastive fine-tuning for 5 epochs with all the models, with the batch size of B = 128 positives for MNEG. The number of hard negatives per each positive is set to N n = 10 (see §2.1). 13 Since standard BLI and XLSIM datasets lack a validation portion , we follow prior work  and tune hyperparameters on a single, randomly selected language pair from each dataset, and use those values in all other runs.
Model Configurations. They are labelled as follows: ENC-{noCL,+CL} (λ), where (i) ENC denotes the input multilingual Transformer, which can be a multilingual LM (MBERT, XLM-R), or a multilingual SE (LABSE, XMPNET), (ii) 'noCL' refers to using the input model 'off-the-shelf' without any contrastive lexical fine-tuning, while '+CL' variants apply the contrastive fine-tuning, and (iii) λ is the factor that defines the interpolation with the static CLWE space, obtained with VECMAP   (see Figure 1 and §2.2). Note that λ = 1.0 implies no interpolation with static CLWE space, i.e., WEs come purely from the multilingual LM/SE. Important Disclaimer. We note that the main purpose of the chosen evaluation tasks and experimental protocols is not necessarily achieving stateof-the-art performance, but rather probing different model variants in different cross-lingual lexical tasks, and offering fair and insightful comparisons

Results and Discussion
Bilingual Lexicon Induction (BLI). Table 1 displays our main BLI results, aggregated over all 28 language pairs of GT-BLI, for two multilingual LMs (MBERT and XLM-R) and their SE counterparts (LABSE and XMPNET). Two trends hold across the board. First, multilingual SEs, LABSE and XMPNET, largely outperform their multilingual LM counterparts, MBERT and XLM-R. The gains are visible in all four experimental configurations (with/without contrastive cross-lingual lexical specialisation × with/without interpolation with the VECMAP CLWE space). This confirms our intuition that multilingual SEs, having been (additionally) trained on parallel data (Feng et al., 2022;Reimers and Gurevych, 2020), should better reflect the cross-lingual alignments at the lexical level than off-the-shelf multilingual LMs, which have not been exposed to any cross-lingual signal in pretraining. The poor cross-lingual lexical alignment in the representation spaces of MBERT and XLM-R also reflects in the fact that with those encoders, we only surpass the baseline VECMAP performance by a small margin (+1.1 for XLM-R, +1.6 for MBERT) after subjecting them to contrastive lexical fine-tuning and interpolating their word encodings with VECMAP WEs.
The behavior of SEs, on the other hand, is much more favorable. LABSE, for example, surpasses baseline VECMAP performance with interpolation alone, even without the contrastive lexical finetuning. When contrastively fine-tuned (and then interpolated with VECMAP) both LABSE and XMP-NET surpass the baseline VECMAP performance by a much wider margin (+6.4 and +5.2, respectively). This implies that our contrastive fine-tuning exposes more of the high quality cross-lingual lexical knowledge from multilingual SEs.
Both (i) contrastive cross-lingual lexical learning (+CL) and (ii) interpolation with VECMAP consistently improve the performance for all four encoders: we reach peak scores by combining contrastive fine-tuning and interpolation with static CLWEs (+CL (λ)). Contrastive fine-tuning crucially contributes to the overall performance: compared to interpolation alone (noCL (λ)), +CL (λ) brings an average gain of over 6 BLI points. Table 2 shows the BLI results on 10 low(er)resource language pairs from PanLex-BLI. While overall relative trends are similar to those observed for high(er)-resource languages from GT-BLI (Table 1), the gains stemming from cross-lingual contrastive lexical fine-tuning are substantially larger in this case. The best-performing configuration -contrastive fine-tuning and interpolation (+CL (0.4)) applied on LaBSE -surpasses VECMAP by 11 BLI points on average (compared to 6 points on GT-BLI), with gains for some language pairs (e.g., HE-KA, ET-HE) approaching the impressive margin of 20 BLI points. This finding indicates that cross-lingual lexical knowledge stored in multilingual SEs is even more crucial when dealing with lower-resource languages.
In Table 3 we compare the results of LaBSE (as the best-performing multilingual SE) against VECMAP on PanLex-BLI in a scenario with less external bilingual supervision: |D| = 1k. Interestingly, in this setup LABSE already substantially outperforms VECMAP out of the box (noCL (1.0)); contrastive lexical fine-tuning (+CL (1.0)) and interpolation with VECMAP embeddings (+CL (0.5)) again bring further substantial gains, and we again observe a strong synergistic effect of the two components: +CL (1.0) yields gains over noCL (1.0) for 9/10 language pairs, and +CL (0.5) re-sults in further boosts for all 10 pairs. Further, the contrastively fine-tuned LABSE seems to be much more resilient to training data scarcity than VECMAP: reduction of the training dictionary size from 5k to 1k reduces the performance of LABSE +CL (λ) by 27% (from 34.7 to 25.2 P@1 points) compared to a massive performance drop of almost 70% for VECMAP (from 23.5 to mere 7.4 P@1).
Cross-Lingual Lexical Semantic Similarity (XLSIM). The average XLSIM results are summarized in Table 1. They again corroborate one of the main findings from BLI experiments: multilingual SEs store more cross-lingual lexical knowledge than multilingual LMs. This is validated by substantial gains of SEs over corresponding LMs across all configurations in Table 1. Interestingly, due to their contrastive learning objectives on sentence-level parallel data (Feng et al., 2022), LABSE and XMPNET provide very strong XLSIM results when used off-the-shelf (noCL (1.0)), outperforming the CLWE VECMAP embeddings. Contrastive lexical fine-tuning with 5k word translation pairs (+CL (1.0)) in this case does not bring any gains. However, the opposite is true for multilingual LMs: contrastive cross-lingual lexical fine-tuning on only 5k word translation pairs brings large benefits in the XLSIM task (e.g., compare noCL (1.0) and +CL (1.0) configurations for MBERT and XLM-R), and turns them into more effective lexical encoders. This result corroborates a similar finding from prior work in monolingual setups (Vulić et al., 2021a). Finally, interpolation with static CLWEs benefits the final XLSIM performance of all four underlying multilingual encoders: interpolated vectors (λ = 0.5) yield highest scores across the board, substantially surpassing gains both VECMAP and WEs from fine-tuned encoders (λ = 1.0).
Interpolation with Static CLWEs. A more detailed analysis over different λ values for BLI and XLSIM, summarized in Figure 2, reveals that interpolation can bring large performance gains, especially for λ in the [0.3, 0.5] interval. The optimal λ value is, however, task-and even dataset-dependent. For instance, for low-resource BLI on PanLex-BLI more knowledge comes from the multilingual encoders as VECMAP CLWEs are of lower quality for such languages: in consequence, the optimal λ value 'moves away' from the static CLWEs towards encodings obtained by fine-tuned multilingual SE. We also note that larger benefits from interpolation   are observed when VECMAP CLWEs are combined with contrastively fine-tuned multilingual SEs: cf., the large gains in Figure 2b and in Table 3 for the LABSE +CL model variant.
Cross-Lingual Entity Linking (XEL). Experiments on XL-BEL (Liu et al., 2021d), summarized in Table 4, demonstrate that additional contrastive tuning with word or phrase pairs can greatly boost performance of multilingual LMs: even finetuning with 5k word translation pairs without any domain-specific knowledge yields strong benefits for XLM-R. As expected, using a much larger and domain-specific external database UMLS yields much higher scores and is more crucial for performance. In fact, contrastively fine-tuning on UMLS generally improves XEL performance with all four underlying models. Again, we observe that SE-based (XMPNET) configurations outperform the respective LM-based (XLM-R) configurations across the board. This finding again indicates that multilingual SEs store more cross-lingual lexical knowledge than multilingual LMs: this difference is particularly salient when the models are used off-the-shelf without any additional contrastive fine-tuning, and XMPNET retains the edge over XLM-R even after task-specific fine-tuning with the UMLS-based domain-specific knowledge.
What is more, for FI, RU, and TR, the multilingual XMPNET-based variants match or surpass the performance of respective XEL models trained on top of monolingual LMs (e.g., for FI, a model based on the Finnish BERT) reported by Liu et al. (2021d). This further validates our hypothesis that multilingual SEs store rich multilingual lexical knowledge, which is then also exposed in domainspecific (multilingual) UMLS fine-tuning, yielding performance gains. Contrastive fine-tuning on UMLS synonyms (+CL+UMLS variants) expectedly outpeforms fine-tuning on (5k) generaldomain word translations (+CL), indicating that in specialized domains, if available, in-domain crosslingual lexical signal should be exploited.

Conclusion and Future Work
In this work, we investigated strategies to expose cross-lingual lexical knowledge from pretrained models, including multilingual language models (LMs) and multilingual sentence encoders (SEs). Based on an extensive evaluation on a suite of crosslingual lexical tasks, we verified that multilingual SEs (e.g., LABSE and XMPNET) are superior to multilingual LMs (MBERT and XLM-R) in terms of stored cross-lingual lexical knowledge. Moreover, we proposed new methods to further fine-tune their representations based on contrastive learning to 'rewire' the models' parameters and transform them from LMs and SEs into more effective cross-lingual lexical encoders. We also empirically demonstrated that for some lexical tasks performance can be further boosted through interpolation of type-level lexical encodings with static cross-lingual word embeddings. These yield gains for all underlying models, and are especially significant for resource-poor languages and in low-data learning regimes. While our work has focused on two widely used state-of-the-art multilingual SEs, the contrastive fine-tuning framework is versatile and model-independent and can be directly applied on top of other multilingual SEs in future work. We will also investigate other more sophisticated contrastive learning strategies, look into ensembling of knowledge extracted from different SEs, and expand our evaluation to more tasks and languages.

Limitations
This work focuses on lexical specialization of multilingual encoders, off-the-shelf LMs (experiments with mBERT and XLM-R Base) and, in particular multilingual encoders specialized for sentencelevel semantics (experiments with LABSE and XMPNET). While these are all widely used models, they are arguably among the smaller pretrained multilingual encoders. Due to computational constraints, we have not evaluated the effectiveness of the proposed cross-lingual lexical specialisation for larger multilingual LMs, e.g., XLM-R-Large (Conneau et al., 2020) or mT5 (Large, XL, and XXL) (Xue et al., 2021). It is possible that these larger multilingual LMs would close (some of) the performance gap w.r.t. multilingual SEs. Such large LMs, however, are effectively available to fewer researchers and practitioners. Our work includes less resource-demanding LMs and SEs, making their lexically specialized variants that we offer, more widely accessible.
Lexical input (i.e., words or phrases) are provided to each multilingual encoder fully "in isolation" (see §2), without any surrounding context. However, the alternative of using external corpora and averaging-over-context (Litschko et al., 2022), which we have not evaluated in this work for clarity and space constraints, might yield slightly improved task performance. Nonetheless, the 'in isolation' approach has been verified in previous work (Vulić et al., 2021a;Litschko et al., 2022;Li et al., 2022) as very competitive, and is more lightweight: 1) it disposes of any external text corpora and is not impacted by the external data; 2) it encodes words more efficiently due to the absence of context. Moreover, it allows us to directly study the richness of cross-lingual information stored in the encoders' parameters, and its interaction with addi-tional cross-lingual signal from bilingual lexicons.
The contrastive cross-lingual lexical fine-tuning we proposed in this is work is bilingual. It leverages a small bilingual dictionary D for each language pair and specializes the multilingual encoders (LMs and SEs) independently for each language pair. Assuming interest in cross-lingual lexical tasks between all pairs of N L languages, this entails N L ·(N L −1) 2 fine-tuning procedures and as many resulting bilingual models. Although our contrastive fine-tuning is relatively fast and lightweight, given that it leverages at most 5k translation pairs, for large N L it could easily exceed the computational and time budget for most users.
Intuitively, for each bilingually fine-tuned model, we evaluate the performance for that respective language pair. Currently, we do not investigate the spillover effects that a bilingual lexical fine-tuning of multilingual encoders could have on lexical representations of other languages. Such an analysis, planned for future work, would be particularly interesting in the context of low-resource languages, unseen from the point of view of cross-lingual lexical fine-tuning, and in particular closely related low-resource languages. For instance, if we are doing cross-lingual lexical fine-tuning for language pairs involving Turkish, are there spillover benefits for low(er)-resource Turkic languages such as Uyghur or Kazakh?
Finally, we acknowledge that our choice of lexical tasks as probing tasks is non-exhaustive: we put focus on standard tasks from previous work on (multilingual) lexical semantics that are especially convenient as cross-lingual lexical probes: such tasks directly test and compare the quality of cross-lingual lexical representations obtained via different methods.

A List of Languages
The list of languages used in this work, along with their ISO 639-1 codes, is available in Table 5.

B BLI Results across Individual Language Pairs
Additional experiments and analyses over individual language pairs and other λ values, which further support the main claims of the paper, have been relegated to the appendix for clarity and compactness of the presentation in the main paper: Table 6. It provides results over all 28 language pairs in GT-BLI with 2 multilingual LMs and 2 multilingual SEs in the noCL variant without contrastive fine-tuning. Table 7. It provides results over all 28 language pairs in GT-BLI with 2 multilingual LMs and 2 multilingual SEs in the +CL variant with contrastive fine-tuning.

D Models and Evaluation Data
URLs to the models used in this paper are provided in