{'fraction_non_alphanumeric': 0.03257039241848849, 'fraction_numerical': 0.01081118067672517, 'mean_word_length': 4.967741935483871, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
WiC-TSV-de: German Word-in-Context Target-Sense-Verification Dataset and Cross-Lingual Transfer Analysis

Target Sense Verification (TSV) describes the binary disambiguation task of deciding whether the intended sense of a target word in a context corresponds to a given target sense. In this paper, we introduce WiC-TSV-de, a multi-domain dataset for German Target Sense Verification. While the training and development sets consist of domain-independent instances only, the test set contains domain-bound subsets, originating from four different domains, being Gastronomy, Medicine, Hunting, and Zoology. The domain-bound subsets incorporate adversarial examples such as in-domain ambiguous target senses and context-mixing (i.e., using the target sense in an out-of-domain context) which contribute to the challenging nature of the presented dataset. WiC-TSV-de allows for the development of sense-inventory-independent disambiguation models that can generalise their knowledge for different domain settings. By combining it with the original English WiC-TSV benchmark, we performed monolingual and cross-lingual analysis, where the evaluated baseline models were not able to solve the dataset to a satisfying degree, leaving a big gap to human performance.


Introduction
Being able to distinguish between different meanings of a word is a crucial pre-processing step for a wide variety of down-stream tasks such as document retrieval, sentiment analysis, or relation extraction. Traditionally, the formulation of the Word Sense Disambiguation (WSD) task was used to tackle this problem, where for a given target word, the corresponding best-matching entry from an external word sense inventory was retrieved. Though this task formulation enjoys great popularity, it comes with downsides that especially show in domain-specific, real-world scenarios: since the target senses are linked to external inventories, WSD systems need to be modelled accordingly to these resources, which not only reduces the flexibility of the models, but also enforces the assumption of the availability of complete data within these resources. In certain scenarios however, it might be impractical or even impossible to model all possible senses. The more recent task formulation of Word-in-Context (WiC) breaks with this dependencies on external sense inventories by focusing on the question whether two contexts use the target word in the same sense. This binary formulation increases the flexibility, but focuses on identifying similar usages of a target instead of assigning a specific sense to a target. In domain-specific and enterprise settings, the focus often lies on a defined set of concepts, i.e., a relatively small subset of all possible senses, which needs to be identified and disambiguated within textual data. The task formulation of Target Sense Verification (TSV) combines the independence of sense inventories and the possibility to identify and assign specific senses, by formulating the disambiguation as a binary classification task: Given a context and a target sense, the task is to verify whether the target word in the context is used in the target sense. In order to create well-performing and reliable TSV models, high-qualitative datasets are necessary, both for their training and evaluation process. However, the number of such datasets is still extremely low: to the best of our knowledge, the WiC-TSV dataset (Breit A. et. al., 2021) is the only resource created for this purpose, which is available in English only. As the creation of large-scale datasets in different languages forms a major bottleneck for the advancement of this area (Pasini, 2020), the development of systems that are capable of performing language transfer on the task are preferable, especially for low-resourced languages. In this paper, we are presenting WiC-TSV-de, a German Word-in-Context Target-Sense-Verification dataset, which not only enables the training and analysis of German TSV models, but also -in combination with the English version-pathens the way for evaluating the cross-lingual transfer capabilities of these approaches. Due to domain-bound test subsets, this resource further allows the evaluation of how well a given model can adapt to a certain domain. The rest of the paper is structured as follows. Section 2 gives an overview of existing disambiguation datasets, while Section 3 provides a formal introduction to the TSV task and German sense inventories. In Section 4, we elaborate on the creation process of WiC-TSVde and its characteristics, followed by the analysis of mono-lingual and cross-lingual baseline models in Section 5 and 6. We conclude our findings in Section 7.

Related Work
Disambiguation Datasets There is a rich number of evaluation datasets targeting different aspects of the standard WSD task, e.g., broad, general frameworks (Raganato et al., 2017;Vial et al., 2018) domainspecific (Agirre et al., 2010;Faralli and Navigli, 2012) and language specific ones (Henrich and Hinrichs, 2012;Okumura et al., 2010;Scarlini et al., 2020). In 2019, (Pilehvar and Camacho-Collados, 2019) added a new flavour to the disambiguation model evaluation landscape by introducing the Word-in-Context dataset, where the focus lies on deciding whether or not the same sense is used in two different contexts. WiC was also integrated as a subtask in the general language understanding framework SuperGLUE (Wang et al., 2019). Further extended WiC datasets were introduced by (Raganato et al., 2020), (Liu et al., 2021), and , for more details, see below. Finally, (Breit A. et. al., 2021) introduced the initial English WiC-TSV dataset, which not only allows the evaluation of the generalisation capabilities of TSV-models, but also their ability to adapt to certain domains, as different domain-specific test subsets were included.
Cross-Lingual Transfer Analysis The analysis of cross-lingual transfer capabilities in disambiguation models has a long tradition -with according SemEval tasks already being published 15 years ago showing interest in this area (Agirre et al., 2007)-and has not lost its relevance today (Gella et al., 2019;Procopio et al., 2021;Ataman et al., 2021). This trend is also visible for recent WiC datasets. XL-WiC (Raganato et al., 2020) contains training sets in three different languages as well as evaluation sets for 12 different languages with varying degrees of resource availability. As a further extension, AM2iCo (Liu et al., 2021) provides datasets from 14 different languages, including difficult adversarial examples, as well as corresponding training sets for 10 of these languages, enabling diverse cross-lingual analysis. The same year, MCL-WiC  has been introduced as an entirely manually-annotated dataset for multi-and cross-lingual WiC, whose evaluation sets are available in 5 different languages.

Preliminaries
TSV The task of Target Sense Verification can be formally described as the binary classification of an instance x, where each x consists of a context c containing a target word w, and a target sense s w represented by one or multiple sets of sense descriptors d s . The classification aims at determining whether the intended sense of the word w used in the context c matches the target sense s. Sense descriptors could be of varying types such as descriptions, hypernyms, or synonyms. In a multilingual setting, the target sense could also be indicated by e.g., its translations.
For the described task, an English dataset was published in previous work  containing 3832 instances, where the training and development set consist of domain-independent instances only (retrieved from WordNet and Wiktionary), while the test set also includes three domainspecific subsets, being Cocktails, Medicine, and Computer Science. As sense descriptors, definitions and hypernyms collected from existing Semantic Web resources were provided. The data of WiC-TSV can be found at https://github.com/ semantic-web-company/wic-tsv German Sense Inventories For English disambiguation tasks, the most prominent resource is WordNet (Fellbaum, 1998), a large lexical database that collects words, associates them with meanings by grouping them in so-called synsets, and organises these sets by the means of conceptual-semantic and lexical relations. The idea of WordNets was adopted for many different languages, including German, where two major resources exist: GermaNet (Hamp and Feldweg, 1997) and OdeNet (Siegel and Bond, 2021).
GermaNet was initiated in 1996 by the Univeristy of Tübingen, Germany, and has since then grown to a mature online thesaurus, organising more than 150,000 synsets. However, the resource underlies rather strict licensing requirements, making the easy re-use, and distribution difficult.
OdeNet, on the other hand, is an initiative that aims at providing German thesauric knowledge in an open and easily accessible way. However, as this resource is still in developmental state, certain aspects are not fully covered yet: while OdeNet contains more than 120,000 lexical entries in about 36,000 synsets, the resource only contains approximately 19,000 definitions and not even 1000 examples.
A different approach of building a lexical resource is taken by Wiktionary, an online dictionary available in a wide variety of languages, where the content is collected in a collaborative, crowd-sourced way. In comparison to expert-built lexicons, Wiktionary is therefore more coarse-grained, as the entries focus more on the general understanding of meanings, than on the linguistic correctness. The collaborative approach on the one hand helps to keep the resource up-to-date with new terms and term usages, and on the other hand improves the coverage of domain specific word senses. Wiktionary not only contains words and their different meanings, but also provides templates to add synonyms, examples sentences, and hypernym and hyponym relations for these senses. However, since entries in this resource are represented by encyclopedicstyle pages, the content of these entries is only provided in an semi-structured way, and is prone to errors. Still, with over 1.1M German entries, Wiktionary plays an important role as sense inventory. I would like to add to this that the female marmots and one-yearold marmots scream much more than the older male marmots.

Die Bezeichnung für das männliche Murmeltier
The name for the male marmot

WiC-TSV-de: The Data Set
Following the definition of the TSV task, we created a dataset of 4117 instances in German language. Each instance consists of a context containing the target word and two different kinds of target sense descriptors: hypernyms and definition (see Table 1). Target words are all single-word nouns, contexts and definitions are tokenised. While the training and the development set consist only of domain-independent instances, i.e., they do not focus on a certain domain, the test set also contains domain-bound subsets, where the target senses are domain-specific. The construction of these differ-ent kinds of instances as well as the characteristics of the dataset are described below.

Domain-Independent Instances
Domain-independent instances were constructed from the German Wiktionary. For this purpose, after an initial cleaning step, we scraped all entries of German nouns that have a definition and at least one example associated with it. Entries where only one sense fulfilled the above criteria were either rejected or used as positive example, while entries with multiple valid senses produced both positive and negative examples. Negative examples were created by randomly interchanging the target senses and examples. During the creation process, examples in which the target word did not appear (e.g., because a synonym was used), or for which the sense numbering was incorrect (i.e., the example was annotated to be connected to sense [2], but there was no corresponding sense [2]) were removed. Great care was taken to avoid information leakage (e.g., created by using the same example with two different target senses) and to keep the number of positive and negative examples balanced. This procedure resulted in 3537 instances, that were split with a 72:12:16 1 ratio into training, development and test set.

Domain-Bound Instances
For the domain-bound instances, first a set of domainspecific ambiguous nouns and their domain-specific target senses was manually created. Then, example contexts were collected manually by incorporating search engines. The contexts which were retrieved from a variety of resources including news articles, blog posts, and recipes, were required to be written in proper German, whereby the formality of the language used was incidental. A further restriction regarding the resources constricted the collection from dictionary and encyclopedia content, in order to maintain a realistic test use case. Domain-bound instances were collected for the following domains: Gastronomy (FOOD) For this domain, ambiguous names of (parts of) fruits and vegetables (32%), dishes (44%), meat cuts (20%), and tools (4%) were identified as target words. Due to the diversity of the field, we could not identify a single resource to retrieve information about definitions and hyperyms for all the targets, but used separate online resources for fruits, vegetables and dishes 2 3 , and tools and meat cuts 4 5 . Hunting (HUNT) This subset is based on the vocabulary used by hunters to describe body parts or subcategories of game (76%), as well as their tracks (10%),  Remarkably, there is no overlap between the domainbound (test) target senses and the target senses in the training set. The overlap between the different test subsets is neglectable: seven domain-independent instances use target words, that are also present in one of the domain-bound sets, where only a single one of these instances targets the same domain-specific sense.

Qualitative Characteristics
Apart from the purely quantitative statistics, taking a closer look at the qualitative characteristics of the dataset and its subset can help interpreting the performance of tested models.
Even though examples taken from Wiktionary are not bound to a specific domain, they may include instances that are domain-specific due to the broad coverage of the resource. We did not filter these examples, as the presence of some domain-specific instances does not contradict the assumption that the instances are retrieved from a general domain corpus, since these corpora do not categorically exclude the usage of domainspecific terminology. However, great care was taken that the target senses included in the test set are not present in the training and development set. Taking a look at the target senses included in the domain-bound test-subsets, we can see differences in their commonality: In Medicine for example, for most of the instances, domain-specific terms that are familiar to a broad, non-expert audience are used, as higher specialised terms are only ambiguous to a limited extent because German incorporates many foreign terms in their medical language. On the other hand, for the instances in the Hunting domain, a highly specialised terminology is used, where the target sense never corresponds to the most common sense of the target word. For Gastronomy and Zoology, part of the target senses (e.g., fruits and vegetables in Gastronomy) describe the most commonly used sense of the target word, while others are lesser known senses. For quantifying this commonality we analysed for each domain-bound instance in the dataset (a) whether the target sense corresponds to the first sense of the target word listed in Wiktionary and (b) whether the target sense is listed at all in Wiktionary or in Duden 11 -a well established German dictionary. Here, we use the ordering of the senses in Wiktionary as an estimate for their commonality, being well-aware that due to the collaborative nature of this resource the equation of the first with the most frequent meaning is merely an assumption, and not enforced quality 12 . A summary of the target sense commonality for the different subsets can be found in Table 3. In order to further create highly challenging test instances, different strategies were implemented when manually collecting contexts for the domain-bound target senses. Of special interest are target words that have a second meaning within the same (or a neighbouring) domain. For example, in the hunting domain the term "Bär" is both used to refer to a bear as well as to a male marmot. These in-domain ambiguities are a great starting point to generate highly challenging test instances, where it is not sufficient to predict the domain in order to correctly verify the target sense. If such in-domain ambiguities were not applicable for a given target sense, the discovery of examples that applied mixing of contexts formed another strategy of generating hard examples. In these instances, an outof-domain sense is used in an in-domain context or vice-verse, for example, in the context "If we now look at an object, for example an apple, the light rays reflected from this apple hit our cornea", the target apple -the fruit-is used in the context of the eyeball, which in German translates to "eye apple". Finally, the notion of trigger words was introduced, where not the entire context, but only specific words draw the connection to the target sense from a different domain. An example would be "Rabbits also like the flower with the intense scent.", where the term flower -describing the plant-could also be used to refer to the tail of rabbits.

Human Baseline
For retrieving an upper bound of performance for the presented dataset, we created a Human Baseline from a sub-sample of the test set. Herefore, 250 instances were randomly selected and split into two batches with 150 instances each, resulting in an overlap of 20%. Each of these batches was assigned to an annotator. The annotators -being German native-speakers and non-experts in all of the four focus domains-were instructed to solve the dataset without the usage of external knowledge sources (e.g., when they are not familiar with the meaning of the target sense). A summary of the human performance evaluation can be found in  Table 3: Qualitative characteristics of domain-bound instances in terms of strategies applied to create hard examples and commonality of the target sense; ambig. stands for in-domain ambiguity, mixing for mixing of contexts, trigger for trigger word. For the target sense commonality, presented are the percentage of instances whose target sense is listed as (1st WIK ): the first sense in Wiktionary and (any WIK /any Duden ): any sense in Wiktionary or Duden than domain-focused ones, with an average accuracy of 83.6 (with individual scores of 81.6 and 85.5) compared to 97.3 (98.4 and 95.9). This phenomenon is in line with previous studies and can be explained by the fact that general-domain meanings are often closer together, only having nuanced differences. Therefore, it is harder to evaluate whether the given target sense fits, or if there could be another meaning that is even closer. This is also reflected in the high recall for domainindependent examples. Domain-specific meanings on the other hand are often more clearly distinguishable, resulting in higher annotation scores. Still, the evaluation performance of human annotators on the domain-bound examples is surprisingly high, especially on domains with highly specialised language such as Hunting, where all examples were correctly annotated. The lowest domainfocused annotation score was achieved in the domain of Medicine with an average accuracy of 94.7, resulting from the individual scores of 100 and 89.5.

Experimental Setup
To establish a baseline for the introduced dataset, we evaluate the performance of a basis model in two different settings: monolingual and cross-lingual. For our experiment we considered HyperBERT3, a classifier based on a pre-trained contextualised BERT model, as our baseline model to fine-tune on our dataset. First, we apply a weak supervision mechanism described in (Huang et al., 2019) to our input data, by surrounding the target word in the context with a special symbol (in this work, we used $). As shown in the original paper, this weak supervision mechanism helps the model identifying the target token. Then, the definition and hypernyms are concatenated -separated by the same special symbol-and fed together with the context into the BERT model. The resulting representations (i.e., for the [CLS] token, target tokens, and the concatenated sense descriptors) are fed into a top-level classifier to create the final predictions. The implementation of HyperBERT3 can be found at https://github. com/semantic-web-company/wic-tsv/ blob/master/HyperBert/HyperBert3.py.
For the monolingual setting, we used HyperBERT with bert-base-german-cased as the base model, which we fine-tuned and evaluated on WiC-TSV-de. For the cross-lingual setting, a model based on bert-basemultilingual-cased was fine-tuned on the English WiC-TSV training and development set, and evaluated on WiC-TSV-de. For each of the two settings, we finetuned the HyperBERT3 model for 8 epochs, and chose the best performing model. For the final results, we calculated the average of three runs.

Results
Monolingual Setting An overview of the performance of HyperBERT3 in monolingual setting can be seen in Table 4. The overall performance remains with an F 1 score of 71% relatively low, especially in terms of precision. This underlines the challenging nature of the dataset, as the baseline model leaves a big gap of almost 20 percent points to human annotation performance. For comparison, the gap between the bestperforming model and human performance on the original English WiC-TSV dataset was 8.6 percent points. The domain-independent instances -despite their relative coarse-granularity-pose a major challenge to the TSV model, which was only able to achieve an average of 71.7% F 1 on this subset. However, the performance achieved on these instances is with 13.8 percent points difference closest to the human annotation performance, compared to the other subsets.
While the average performance of domain-independent and domain-bound instances are similar, there are some differences between the different domains. The most difficult domain seems to be Hunting with an average F 1 score of 62%, followed by Zoology. On the most successful subset, Gastronomy, HyperBERT3 reached an F 1 score of 76%. These performance difference could be partly explained by the qualitative characteristics of the domain-bound subsets: the Hunting and Zoology domain contain a greater number of in-domain ambiguous target words and context-mixing instances, leading to examples where the correct label could not be guessed from identifying the domain of the whole context. Interestingly, these types of instance seem to cause less erroneous predictions in the Gastronomy domain, which contains more of these hard examples than the medical domain, but shows a better performance. Another interesting aspect that arises from this analysis is that the performance on all subsets is leaning towards  Table 4: Test set performance of the monolingual (HBERT de ) and cross-lingual (HmBERT en de ) Hyper-BERT3 baseline models, an All-True classifier, and human annotators on WiC-TSV-de, in terms of accuracy, precision, recall, and F 1 for the positive label. For the models, the mean performance and standard deviation out of three runs is presented. The overall performance (top) is further split into the performances on the domain-independent (WIK) and the domain-bound subsets (FOOD, MED, HUNT, and ZOO).
recall, except for the Hunting domain. A possible explanation for this could be the aforementioned highly specialised, yet ambiguous language used in this area, which results in low commonality of the target senses. However, it should also be noted that the standard error of the recall is quite high for most domains.
Cross-Lingual Transfer Setting The outcomes of the cross-lingual transfer setting are also to be found in Table 4. It can be seen that the overall performance is very low, being only 1.9 percent points above the F 1 score of the naive All-True lower-bound baseline. The resulting classifier indeed is close to an all-true classifier, as not even 13% of its predictions on the entire test set were False labels. Generally, the incorporated multilingual BERT model seems to have difficulties to adapt to the presented task, as the model also performed quite poorly in a monolingual setting, i.e., when both fine-tuned and tested on the WiC-TSVde, reaching an accuracy of only 61% (Precision: 62.7, Recall: 54.6). When comparing the different domainbound subsets, a similar image than in the monolingual analysis is drawn: while the best performance was achieved on Medicine, and Gastronomy, with 61.0% and 59,1% accuracy, the Zoology subset is the most difficult one. Interestingly, the performance difference of the multilingual model on Zoology and Hunting is notable bigger than is is in the monolingual equivalent.
Domain-difficulty To estimate the effect of challenging examples on the performance of the TSV models we calculated the Pearson correlation of F 1 scores and the extent to which a certain strategy (or combinations of) contributed to the creation of the domain-bound subsets (cf Table 3). A summary of the is presented in Figure 1.
For the monolingual HyperBERT3 model, we identify that, overall, all Hard Example Strategies show an expected negative correlation (Pearson coefficient between -0.71 and -0.95) and that the correlation increases when combining different strategies. The strongest correlation was observed on the sum of indomain ambiguous and trigger word examples. Curiously, the analysis on the Target Sense Commonality suggests a weaker connection between the performance of the tested models and the usage of uncommon target senses. While the correlation between performance and percentage of instances whose target sense is listed as the first sense in Wiktionary is only 0.33, the Pearson coefficient when comparing whether the target sense is listed at all ranges from 0.55 to 0.74, depending on the resource compared against. Interestingly, the correlation to the expert-curated Duden is much higher than to the crowd-sourced Wiktionary, which could indicate that expert-curated sources are a more reliable source regarding sense commonality. Due to the general low performance of the model, the analysis of multilingual HyperBERT3 in the crosslingual setting, yield to inconclusive outcomes. While the correlation coefficient for all types of Hard Example Strategies and their combinations are negative, the correlations are weaker than for the monolingual setting, with Pearson coefficients between -0.16 (mixing) and -0.77 (ambig). For the Target Sense Commonality analysis, the coefficients range from -0.24 (first sense in Wiktionary) to 0.18 (any sense in Duden). Nevertheless, from this analysis it is observable that the introduced strategies for collecting contexts indeed lead to hard instances. It shall be noted, however, that the number of datapoints of these correlation analysis is quite low and therefore the presented correlation values rather represent an indicated trend than reliable statistics.

Conclusions
In this paper, we introduced WiC-TSV-de, a German dataset for evaluating the disambiguation capabilities of Target Sense Verification models. Due to its binary formulation, the dataset allows for the development of flexible models that are independent of sense inventories. While the training and development set consist of instances that are not bound to any specific domain only, the test set also contains domain-bound instances originating from fours different domains, being Gastronomy, Medicine, Hunting, and Zoology, allowing to evaluate the domain transfer capabilities of models. These domain-bound instances incorporate adversarial examples that exploit in-domain ambiguities, mixing of contexts and trigger words, to make the dataset more challenging. Nevertheless, human annotation performance of over 90% underlines the overall clarity of the test instances. By combining WiC-TSV-de with the original English WiC-TSV, the dataset can further be used to analyse the cross-lingual transfer capabilities of disambiguation models. Initial experiments on WiC-TSV-de showed that the BERT-based baseline model was not able to perform satisfyingly on the dataset, leaving almost 22 percent points gap to human performance. In general, domains with more adversarial examples appear to be harder for the model. When evaluating in cross-lingual setting, the baseline model showed even worse performance of only approximately 57% accuracy, as the model adapted an all-true-classifier-like behaviour.
This dataset therefore opens up avenues for the development of models that can successfully perform domain-and language-transfer, and thus are capable of performing well in a variety of different settings without the need of large amounts of scenario-specific training data.