{'fraction_non_alphanumeric': 0.03563829787234043, 'fraction_numerical': 0.009884751773049646, 'mean_word_length': 5.08277163655972, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Caveats of Measuring Semantic Change of Cognates and Borrowings using Multilingual Word Embeddings

Cognates and borrowings carry different aspects of etymological evolution. In this work, we study semantic change of such items using multilingual word embeddings, both static and contextualised. We underline caveats identified while building and evaluating these embeddings. We release both said embeddings and a newly-built historical words lexicon, containing typed relations between words of varied Romance languages.


Introduction
Languages are in constant evolution over time; words appear, disappear, and their syntactic form and semantic function evolve (Blank and Koch, 1999). However, languages evolutions can be closely inter-related, following phenomena of interactions and inheritance. Cognates and borrowings, which are the targets of our study, are direct consequences of these phenomena. Cognates are words which descend from the same ancestor word (their proto-form) belonging to a shared common direct parent language. For example, the French word chat 'cat' is cognate with Spanish gatto and Romanian cȃtus ,ȃ , as they all descend from Latin cattus 'cat', a direct ancestor of these three languages. When a word is an evolution of a form which does not come from a direct ancestor, it is called a borrowing. English cat also comes from cattus, 1 but as Latin is not a direct ancestor of English, it is therefore a borrowing of English to Latin. We consider the relation between cat and chat to be of 'borrowing' type by extension. Borrowings mostly occur to designate 'realities that were unknown before the adopting speech community got in contact with the "giving" culture and its language' or to replace already existing meanings by the word of the related dominant culture (Krefeld, 2013). To study semantic variation, we look at our words' glosses, which are expressions of their meaning, here as their English translations or definitions. In our previous example, while the French and Spanish cognates both retained the original sense 'cat', the Romanian cognate went through a semantic change and is translated as 'handcuff'.
Semantic change studies historically relied on specific word relations, cognates and 'borrowings' (Durkin, 2015), found through the comparative method (formalised by Osthoff and Brugmann (1878)). The last few years have seen the emergence of new tools such as contextualised embeddings to study semantic variation (Martinc et al., 2020), enabling the comparison of word senses across domains, periods and languages. We join both approaches and expand on the work of Uban et al. (2021), who use 'static' (non-contextualised) embeddings to study semantics of cognates and borrowings in contemporary Romance languages and English. In this work, we use static as well as contextualised embedding to study the semantic evolution of cognates and borrowings, for both contemporary and older Romance languages, as well as English. To this end, we first create a dataset of cognates and borrowings from the widely studied Romance family (contemporary: Spanish, French, Italian, Portuguese, Romanian, old: Latin, Old Spanish, Middle French), to which we add English. 2 Then, we compare several methods to tackle the issue of obtaining, for low-resource historical languages, embeddings spaces aligned with the ones of contemporary languages. Both dataset and embeddings are released with the paper. 3 Lastly, we use these embeddings to study semantic shift for both diachronic (between parent and child) and synchronic (between children) cognates or borrowing relations, and find that contextualised embeddings allow us to reach more accurate conclusions. At each step, we highlight the possible pitfalls.

Related works
Cognates and borrowings transcribe different aspects of their languages history, and are often studied through the lens of orthographic (Ciobanu andDinu, 2015, 2019) or phonetic combined with semantic variation (Kondrak, 2001). Uban et al. (2021), which we extend, study semantic variation in modern Romance languages between cognates and borrowings by considering their modern-day embeddings as a 'snapshot in time' of their meaning. As their dataset is not available, we can not use as a benchmark; however, like several public etymological databases, among which CogNet (Batsuren et al., 2019), containing cognates and borrowings without differentiating between both relation types, and EtymDB2 (Fourrier and Sagot, 2020), too small for our needs but which differentiate between both types, we build a dataset using the Wiktionary 4 as etymological source.
Semantic change across languages is actively researched in the linguistic and sociology research communities (Boberg, 2012), as it offers valuable information for sociological and historical analysis. In the NLP domain, many authors apply diachronic embeddings models to more than one language (Hamilton et al., 2016;Schlechtweg et al., 2020), but without considering their interactions. Some work studies variations between languages or dialects, diachronically (Martinc et al., 2020;Montariol and Allauzen, 2021) or synchronically (Hovy and Purschke, 2018;Beinborn and Choenni, 2020). However, although several annotated datasets are available to evaluate diachronic semantic change detection methods (Schlechtweg et al., 2020), crosslingual semantic change does not have such resource and cognates and borrowings seem like a promising proxy for evaluating these methods.

Datasets and Corpora Construction
We create a dataset of cognates and borrowings in all languages under study. To complement it, we need corpora in each language to train or extract word embeddings; such corpora are publicly available for highly studied languages. We use a sample of the OSCAR corpus (Ortiz Suarez et al., 2019;Abadji et al., 2021) for contemporary languages and Latin. For Middle French and Old Spanish, we use less well-known resources.

Reference dataset construction
From the latest version of the Wiktionary, our goal is to construct a simple relational set of triplets (lang, lexeme, gloss) to other triplets for cognates and borrowings.
Parsing and general information extraction (for lexeme, language, relations) is described in Appendix A. 5 As extracting glosses proved less straightforward, we detail it here. We encountered three types of problem. 1) In the Wiktionary, some words have English translations as glosses, while others have English definitions: for example, the first definition of 'eau' (water) is 'Water, a liquid that is transparent, colorless, odorless and tasteless in its pure form, the primary constituent of lakes, rivers, seas and oceans', while for 'fort' (strong) it is 'strong; powerful' and 'skilled, proficient, successful, ...', a translation. Splitting glosses on punctuation to store the different semantic aspects as words is therefore indispensable in translation cases, but introduces mistakes when definitions are present. These cases were manually checked, but some mistakes might still remain. 2) All English words are defined (which makes sense, as the Wiktionary technically is an English multilingual dictionary). In order to have an homogeneous base, and as we try to keep translations only, we therefore make the choice to use English lexemes as their own 'translation' to English. 3) Some words (especially in Latin) are only defined as inflections or derivations of other words (e.g. capitum, only defined as 'genitive plural of caput'). In those cases, the gloss is not retained. After cleaning (also detailed in App. A), we construct our database, looking only at inheritance relations (App. A.2). Though cognate-typed relations exist in the Wiktionary, we deliberately choose to ignore them, as they can induce noise for our task: to define cognacy, we stood so far on the side of historical linguistics, but the term can sometimes more broadly refer to words with shared form and meaning, regardless of etymology (Frunza and Inkpen, 2009). This underlines the attention to sources which needs to be paid when constructing one's own database.
Statistics by language are detailed in App. A.4, For Old Spanish (OSP, 10th to 15th century), we extract data from the Digital Library of Old Spanish Texts 6 , then clean it using the transcription norms described on the website.

Cross-lingual embeddings
We compare the semantic function of words in cognates and borrowings pairs. To this end, we explore various ways of obtaining aligned word embeddings in all languages (multilingual embeddings), using static and contextualised embeddings. The former are trained using FastText (Bojanowski et al., 2016) and aligned a posteriori, while the latter are extracted using the multilingual language model mBERT (Devlin et al., 2019) from corpora in all the languages under study. Trained embeddings and language models can be found for all our contemporary languages. However, historical languages such as Middle French and Old Spanish suffer from a scarcity of resources that we have to address.

Static embeddings
Available FastText embeddings. They were trained on Wikipedia data and either already crosslingually aligned for our contemporary languages (Bojanowski et al., 2016), or available unaligned for Latin (Grave et al., 2018).
Training FastText embeddings. OSP and FRM do not have available embeddings: we therefore train some, using default subword tokenisation and an embedding size of 300. 7 However, we expect the quality of these new embeddings to be lower than the pre-trained ones, as 1) the imposed embedding size is likely too big with respect to the training data size, which could affect embedding ability to store relevant information, and 2) we were not able to define an adapted preprocessing. 8 Aligning all embeddings spaces. Alignment is needed to obtain a coherent representation space between languages, and can be done either in a supervised or unsupervised way (Lample et al., 2017;Conneau et al., 2017). Preliminary experiments of unsupervised alignment (Alaux et al., 2018) led to extremely poor results. Consequently, we use bilingual lexicons 9 to supervise the alignment of Latin embeddings with Spanish, with around 2k bilingual word pairs used for supervision. Having no such dictionary for OSP/FRM, we use transparent words with their closest language (respectively SP/FR) to perform a supervised alignment, extracting for each language a bilingual lexicon of around 8k transparent words.
Extracting embeddings To build word embeddings, we had to manage un-homogeneous data with respect to diacritic: many cognates and borrowings seem absent from the embeddings vocabulary, especially for languages with diacritics (FR, RO, ES) or spelling variations (FRM) not homogenised in the embedding training corpora. We define a set of rules to extract embeddings despite word form variations. To embed word glosses (when made up of several words/sentences), we remove stopwords and compute the mean of all sequence word embeddings.

Contextualised embeddings
For contemporary languages and Latin, we use a sample of the OSCAR corpus (Ortiz Suarez et al., 2019;Abadji et al., 2021) to build our contextualised embeddings, as, given its very large size (e.g. for Spanish, more than 25 billion tokens), working on the whole corpus would be timeintensive. For the other languages, we use the corpora descibed in Section 3.2.
We use an mBERT 10 model trained on 104 languages, including Latin and all our contemporary languages, from the transformers library (Wolf et al., 2020). Its training on Wikipedia data, allows for fairer comparison with FastText embeddings.
Massive multilingual pre-trained language models have been shown to perform well on new languages in a zero-shot fashion (Muller et al., 2021), especially those closely related to already seen high resource languages. Thus, we expect mBERT to perform well on OSP and FRM, but we also compare fine-tuning it on our FRM and OSP corpora using the masked language modelling task. We study cognates and borrowings representations in context, by computing the average embedding across all target word occurrences in corpora of their respective languages (Martinc et al., 2020). We compute word embeddings as the sum of the last 4 encoder layers of the model. When a word is divided into sub-words, we take the average of the sub-word embeddings For word gloss embeddings (that we see as a representation of meaning), as we often have several words or a sentence as definition, we can directly generate their embeddings without contextualisation in a corpus. When the gloss is composed of several words, we try both averaging the representations of all tokens in the gloss, and using the embedding of the CLS representation. To compare them, we compute the cosine similarity between the target word embedding and the embedding of its associated gloss. Taking the CLS embedding leads to a similarity of 0.61 on average, while the average of all token embeddings leads to 0.67; we choose the latter to represent word meanings.

Results
Our metric is cosine similarity, commonly used in semantic change detection (Kutuzov et al., 2018). Our results are summarised in Table 1 (full results in App. B). Language pairs are split into parent to child (with LA, FRM, or OSP), and child to child (between contemporary languages) relations. We also differentiate cognates and borrowing pairs whose meaning stayed the same (un-shifted, equal gloss between the two items) or changed between the two languages (shifted, different gloss between the two items). 11 We display similarity (across all our languages) between cognates / borrowings and their counterparts in an un-shifted (line 1) or shifted (l. 2) pair. We also display the average difference between these two scores (l. 3), this time computed per language pair: we expect it to be a measure of the models ability to capture semantic shift. The last two lines show the average embedding similarity between an item and its meaning, 12 which should be constant on average for a given language pair, since it reflects embedding alignment distance between the languages of interest and English. 13 Embedding space quality. For FastText , the average similarity between item and meaning (l. 4 and 5) varies considerably from one language pair to another, which indicates variation in embedding alignment quality between English and other languages. This score also varies inside a given language pair (between borrowing/cognates or shifted/un-shifted words), which could further indicate embedding space quality problems. Indeed, an item embeddings and the embedding of its English gloss should always be relatively similar when using properly aligned embeddings spaces. We also observe that, contrary to expectations, publicly available pre-aligned embeddings (child-tochild) often have even higher variance and lower item-meaning similarity (therefore a worst alignment to English) than our aligned low-resource historical embeddings (parent-to-child). On the other hand, for mBERT embeddings, this similarity score is constant (with a slight variation between cognates and borrowings, likely explained by the fact that language pairs distribution between cognates and borrowings is different), which reflects a high embedding alignment quality. One should therefore be wary of conclusions drawn from the aligned FastText embeddings, even publicly available pre-aligned ones, which might lead to incorrect assumptions by introducing hidden factors into play. We will therefore draw conclusions only us-   ing mBERT.
We also compared vanilla and fine-tuned OSP and FRM mBERT embeddings (Tables 8 and 9 in App. B); fine-tuning shows no significant improvement, though for some edge cases, it seems to increase semantic shift sensitivity slightly while decreasing similarity with other embedding spaces; consequently, we keep the simplest approach, the vanilla mBERT model. When working on historical data, it is interesting to study whether finetuning results justify its cost, or if zero-shot transfer can directly provide good enough results.
Global comparison Using mBERT embeddings, the only difference in similarity scores for items occurs between un-shifted and shifted word embeddings, with un-shifted pairs similarity being on average 4 points higher than shifted pairs (not necessarily statistically significant). Some outliers cases can be found in the per-language tables (see Table 6 in App. B), where shifted cognates have higher intra-pair similarity compared to un-shifted cognates for the same language pair. However, this situation only happens for languages with less than 20 cognates examples of shifted or un-shifted pairs (e.g. OSP-FRM, 12 shifted cognates), and are likely not significant.
There is virtually no difference between cognates or borrowings embeddings similarity. As a side note, FastText embeddings would have shown that cognates are more similar than borrowings, and a word is more similar to its parent than to its siblings: a hasty analysis using bad quality embeddings could have lead us to draw seductive but erroneous conclusions from the FastText embeddings.
Focus In order to investigate differences at the language pair level for the mBERT embeddings, we focus on two language pairs which have at least 20 samples for both shifted and un-shifted pairs of cognates and of borrowings: FR-ES and FR-IT (Table 2). 14 Both present a trend where borrowings are more similar than cognates and un-shifted words more similar than shifted words (as expected).

Conclusion
In this work, we create a cognate and borrowing dataset for English and Romance languages from different periods, as well as two aligned embeddings sets for all languages. When assessing embedding quality and alignment, we show that Fast-Text embeddings, even when already pre-trained and aligned, are poorer than the mBERT ones on all respects. We therefore use the latter to study semantic change between cognates and borrowings: as expected, un-shifted word pairs are on average more similar than shifted ones. Furthermore, we observe a trend between cognates and borrowings, the latter being seemingly more similar than the former. Further analysis would be needed to determine whether this difference can be confirmed, by looking at chosen cognate and borrowings of similar histories in more languages. In summary, properly designed embeddings can be used to support historical lexicographic studies, while well-understood phenomena underlying cognates and borrowings can help design and evaluate cross-lingual word embeddings.

A Extracting cognates and borrowings data A.1 Extraction
Parsing the Wiktionary The Wiktionary dumps mixes several formatting types, mostly HTML for the page tags and a pseudo-markdown for the internal structure of each article, which is not homogeneous between entries. The first step of processing was 1) to cut the Wiktionary by page, by literally cutting it on page HTML tags, and 2) at the same time, to only keep the title (lexeme) using HTML title tags and the text (core of the page) without the rest of the HTML using HTML text tags. Some pages were automatically discarded, if containing "Wiktionary", "App." or "Thesaurus" in their titles, as they are out of scope for the database.
Storing words and relations Once each page was cut, we cleaned the text, by extracting lexeme (first line), langs (second level pseudo markdown separation), and associated information (third levels pseudo markdown separations). The associated information was then cleaned using regexes, to find meanings (lines starting with an enumeration marker), descendants (using 'desc', 'desctree', and 'bor=1' as markers), ascendants (using 'inh' and 'root' as markers), 15 and supposed cognates (using 'cog' as marker). Lexemes were normalized using unicodedata. This allowed us to construct a list of Word objects, storing lexeme, lang, gloss, parent words, children words, and plausible cognates. (Related words were stored as "word_lang" in order to filter them). For each word, we added to its ancestor the set of its ancestors' ancestors, and we converted gloss for English lexemes to the English lexeme itself.

A.2 Constructing our cognates and borrowing sets
Lastly, we converted this list to our cognate and borrowing sets. For each word, we first stored indirect parents as borrowing relations (borrowing set) and direct parents as cognate relations (cognate set), for parent languages in our languages of interest. Then, we looked at each direct ancestor's children (no matter the direct ancestor language): if a given child was direct, both its relation to the parent and to the initial word were stored as cognates (for language pairs of interest). Else, we stored both relations in our 'borrowings' set (id.). In other terms, two words are kept if they share a common proto-form. If their ancestor is direct, we save them as cognates, else borrowings. We use an extended version of the notions of cognacy and borrowing defined in the introduction, and consider that the proto-words are also both cognates with their direct descendants, and in a borrowing relationship with their indirect descendants.

A.3 Cleaning
Extraction problems Splitting the document on HTML page limits was sometimes linked to pages not being cut at the right place, and the title tag not being recognised: some lexemes were stored as '<tag>' (they were removed). Some irregularities in meaning definitions appeared, such as #English not being removed, or some reference urls being accidentally added to the English meanings. All these were manually managed.

B Full results tables
The tables contain the number of cognate pairs kept for each language pairs, as well as an embedding similarity score between 1) both cognates/borrowings of a given pair, 2) both glosses of a given pair, 3) each cognate/borrowing to its gloss. Results are split by language pair and category (meaning shift or no meaning shift).     Table 9: Statistics when using mBERT embeddings, with OSP/FRM finetuning ( f t ) or without, for Old Spanish and Medieval French borrowings with shifted and unshifted pairs. The 'shift measure' is the average difference between semantic item similarity, between non-shifted and shifted pairs. We observe that the difference between shifted and non-shifted items decreases this time, when compared to cognates, for OSP-EN and FRM-ES, and increases for FRM-EN. We consider that variations are not consistent enough to draw conclusions.