{'fraction_non_alphanumeric': 0.035960214231063506, 'fraction_numerical': 0.017002465357476834, 'mean_word_length': 5.026383196721311, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 4, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Corpus Co-Occurrence, Dictionary and Wikipedia Entries as Resources for Semantic Relatedness Information

Distributional, corpus-based descriptions have frequently been applied to model aspects of word meaning. However, distributional models that use corpus data as their basis have one well-known disadvantage: even though the distributional features based on corpus co-occurrence were often successful in capturing meaning aspects of the words to be described, they generally fail to capture those meaning aspects that refer to world knowledge, because coherent texts tend not to provide redundant information that is presumably available knowledge. The question we ask in this paper is whether dictionary and encyclopaedic resources might complement the distributional information in corpus data, and provide world knowledge that is missing in corpora. As test case for meaning aspects, we rely on a collection of semantic associates to German verbs and nouns. Our results indicate that a combination of the knowledge resources should be helpful in work on distributional descriptions.


Motivation
Research in data-intensive lexical semantics aims to empirically define and induce features that (a) capture the various meaning aspects of the words to be described, and (b) one can obtain automatically; the goal is to determine the meaning as well as the similarity or dissimilarity of words, sentences, paragraphs, or even documents. Following the distributional hypothesis, namely that 'each language can be described in terms of a distributional structure, i.e., in terms of the occurrence of parts relative to other parts' (Harris, 1968), distributional, corpus-based descriptions have frequently been applied to model aspects of word meaning. For example, variants of the vector space model (Salton et al., 1975) that uses words in documents to describe the contents of the respective documents, have been used in various NLP tasks and applications including word sense discrimination (Schütze, 1998), anaphora resolution (Poesio et al., 2002), general models of semantic similarity (Lin, 1998;Schulte im Walde, 2006;Sahlgren, 2006;Padó and Lapata, 2007), and also in psycholinguistic models of semantic priming (Lund et al., 1995;Lowe and McDonald, 2000;Vigliocco et al., 2004).
Distributional models that use corpus data as their basis have one well-known disadvantage: Even though the distributional features based on corpus co-occurrence were often successful in capturing meaning aspects of the words to be described, they generally fail to capture those meaning aspects that refer to world knowledge, because coherent texts tend not to provide redundant information that is presumably available knowledge. This fact has recently been illustrated by Schulte im Walde et al. (to appear): A collection of semantic associates to German verbs and nouns was used as a test case for covering meaning aspects by corpus co-occurrence (among other analyses). 1 The assumption was that the evoked words, i.e., the associates, reflect highly salient linguistic and conceptual features of the respective stimulus words. Among other analyses, Schulte im Walde et al. demonstrated that corpus co-occurrence did only account for 67% of the associate types to verbs and for 70% of the associate types to nouns. The remaining 30-33% were partly due to lemmatisation mismatches and the domain (newspaper) and size of the corpus. However, further cases of associations that did not appear in co-occurrence with their stimulus words reflected world knowledge, and were therefore unlikely to be found in the immediate context of the stimuli at all (e.g., Wasser 'water' for auftauen 'defrost', Überraschung 'surprise' for Geschenk 'present', and gelb 'yellow' for Ananas 'pineapple').
The question we ask in this paper is whether dictionary and encyclopaedic resources might complement the distributional information in corpus data, and provide world knowledge that is missing in corpora. Dictionary and encyclopaedic knowledge resources have a long history as semantic resources, from the early days in machine translation (Bar-Hillel, 1960), approaches to word sense disambiguation (Lesk, 1986), and more recently -relying on Wikipedia-in NLP tasks such as text categorisation (Gabrilovich and Markovitch, 2006), word sense disambiguation (Mihalcea, 2007), and co-reference resolution (Ponzetto and Strube, 2007). These resources provide detailed information about word senses, and include world knowledge to varying degrees. (Cf., for example, Weber (1996) and Engelberg and Lemnitzer (2004) about dictionary and encyclopaedic information.) We rely on the same association norms that have been used by Schulte im Walde et al., and check on how many and which of the semantic associates are found within the dictionary/encyclopaedic entries of the respective stimulus words. We compare the results against the previous co-occurrence analyses, to decide whether the different types of information complement each other. The positive outcome is interesting for work on distributional descriptions, by indicating resources that potentially add knowledge to word meaning descriptions.

Association norms
This section introduces our methods for collecting human associations to German verbs and nouns, 2 and a distributional representation of the data as stimulus-associate type frequencies. For more details on the data collection, the reader is referred to Schulte im Walde et al. (to appear).

Associates to verb stimuli
The data collection of associates to verb stimuli was performed as a web experiment, which asked native speakers to provide associations to German verbs. 330 verbs were selected for the experiment, drawn from a variety of semantic classes and from different corpus frequency ranges. Each collection trial consisted of a verb presented in a box at the top of the screen. Below the verb was a series of data input lines where participants could type their associations. Participants had 30 seconds per verb to type as many associations as they could. In total, we collected 79,480 associate responses distributed over 39,254 different response types. Table 1 provides an example of the token-per-type association frequencies, listing the 10 most frequent responses for the polysemous verb klagen 'complain, moan, sue'.

Associates to noun stimuli
The data collection of associates to noun stimuli was performed as an offline experiment (Melinger and Weber 2006), 3 which asked native speakers to provide up to three associations to German nouns. 409 German nouns referring to picturable objects were chosen as target stimuli, again drawn from a variety of semantic classes and from different corpus frequency ranges. No time limits were given for responding, though participants were told to work swiftly and without interruption. In total, they collected 116,714 associate responses distributed over 31,035 different response types.

Knowledge resources
This section introduces the resources that contributed to the characterisation of the association norms.

Corpus data
A German newspaper corpus from the 1990s was used for co-occurrence analyses between verb/noun stimuli and associate responses. The corpus contains approximately 200 million words of newspaper text.

Dictionary: WDG
The dictionary-based analysis relies on the freely available "Wörterbuch der Deutschen Gegenwartssprache" (WDG). 4 The online resource WDG consists of 130,000 entries. In each entry, there is a detailed description of senses, idioms and compounds containing the respective word. For our analyses, we downloaded all entries for our verb and noun stimulus words. This resulted in 706 entries (for 33 stimulus words, the dictionary did not provide any entry), containing a total of 51,561/593,318 words (types/token), with an average of 330/840 words (types/token) per entry. Table 3 provides as example a part of the WDG entry for the ambiguous noun Apfel 'apple', referring to two senses of Apfel, 'fruit of apple tree' and 'apple tree (coll.)', each followed by example contexts (in italics). Morphological information is omitted in the example.

Encyclopaedia: Wikipedia
Wikipedia is a free multi-lingual online encyclopaedia. 5 The resource represents the result of a collaborative effort, where anybody can create and edit entries. This paper uses the German version of Wikipedia, which contained over 650,000 entries as of October 2006. For our analyses, we downloaded all Wikipedia articles which were labelled by any of the verb or noun stimulus words. If the downloaded article indicated an ambiguity, we also downloaded the linked pages, thus using all available articles for the various senses of the respective word. This resulted in 2,447 articles for 542 stimulus words (for 197 stimulus words, Wikipedia did not provide any entry), containing a total of 270,014/2,848,150 words (types/token), with an average of 492/1,164 words (types/token) per article. Table 4 provides as example a part of the Wikipedia linked page for the ambiguous noun Apfel 'apple', and the beginnings of the respective Wikipedia articles for the two linked senses Äpfel and Kulturapfel.

Analyses
The analyses check for each stimulus-associate pair (token and type), whether it is covered by the knowledge sources introduced in the previous Section 3.

Corpus-based analysis
This analysis is taken from Schulte im Walde et al. (to appear). We checked whether the stimulus-associate pairs co-occurred in our corpus within a window of 20 words to the left and to the right of each other. 6 Table 5 and Table 6 present the results. The 'all' row shows the percentage of associate responses that were found in co-occurrence with their stimulus words across all parts-of-speech (POS) of the responses; the following rows are broken down for the parts-of-speech of the re- 6 We are aware of the fact that a co-occurrence strength of 1 does not provide any strong evidence for the relatedness of the stimuli and the responses. The original analyses in Schulte im Walde et al. therefore broke the results down to several co-occurrence strengths. In this article, however, we rely on a strength of 1 in accordance with the other analyses.
sponses. Both for the verb and noun stimuli, 1% of the words were missing in the corpus and therefore a priori missing in the analyses.

POS
Types

Comparison and interpretation
This section contains the main body of our work. We compare the coverage of the three resources (Section 4.4.1), the type of knowledge that is covered (Section 4.4.2), and discuss the role of frequency for the coverage (Section 4.4.3).

Comparison of resource coverage
Comparing the overall 'all' proportions of stimulus-association pairs that are covered by the three knowledge resources shows that the corpus data covers more pairs than Wikipedia, which in turn covers more than the dictionary. An important factor regarding the coverage is, of course, the size of the various resources, which we will address in Section 4.4.3. For the moment, we concentrate on comparisons that are within-resource: The resources differ in a. the proportions of coverage and missing entries when comparing associates to nouns vs. verbs, b. the coverage with respect to the parts-of-speech of the responses, c. the token-type ratio of the covered stimulus-association pairs.
Concerning a. -The resources differ in the parts-of-speech of the stimuli about which they provide knowledge. While the corpus is missing 1% of both verb and noun stimuli (about which it consequently cannot provide knowledge), the dictionary covers all verb-association pairs but fails to cover 7% of the noun-association pairs because the respective noun stimulus entries were missing; Wikipedia is missing only 2% of the noun-association pairs but 56% of the verb-association pairs, because the respective stimulus entries were missing. Accordingly, the overall coverage of the resources with respect to noun vs. verb stimuli is similar for the corpus co-occurrence (70/67%) and the dictionary (12/13%), but differs strongly for Wikipedia (26/6%). Whether these differences represent a general tendency of the resource types, and whether they apply across to other, similar resources (e.g., to other dictionaries, and other encyclopaedias) remains an open question within this paper but is certainly an interesting issue to address, as it might provide insight into preferences for resource types according to the parts-of-speech of words that are described by the respective resource knowledge.
Concerning b. -The resources differ in the proportions of the various parts-of-speech of the associations that they cover. With respect to association types to noun stimuli, Wikipedia covers more noun responses than adjective responses, and more adjective responses than verb responses (N > ADJ > V); for corpus co-occurrence and the dictionary, in contrast, the proportions are related as V > ADJ > N. With respect to association types to verb stimuli, all resources provide a strong coverage of adverb responses; the other parts-of-speech of the responses are covered similarly strong within each resource, with small differences in the proportions. Comparing the coverage of the various parts-of-speech, of course, needs to be related to the overall numbers of the parts-of-speech within the resources. For example, Wikipedia contains a total of 192,655/883,059 nouns (types/tokens), 19,527/260,156 verbs, and 41,164/238,475 adjectives; WDG contains a total of 22,979/135,113 nouns (types/tokens), 10,375/69,269 verbs, and 11,602/48,471 adjectives. This comparison shows, however, that the proportions of the associate parts-of-speech that are found within the resources are not correlated with the overall part-of-speech proportions in the resources; thus, we can infer that there must be other, resource-based reasons to the different patterns of part-of-speech coverage. The high coverage of adverbs in all three resources can be explained by the token-type ratio of adverbs: even though adverbs are an open class in German, the absolute number of adverb types in natural language text and speech is much lower than those of verbs, nouns, and adjectives, but at the same time the token-per-type ratio is higher (for example, Wikipedia contains 861/92,321 adverbs (types/tokens), and the dictionary contains 452/13,185 adverbs); furthermore, the grammatical restrictions within German clause structure are lower. So there is a high prior probability to find adverbs in the vicinity of a verb.

Concerning c. -The resources differ in the token-type ratio of their overall coverage.
Comparing the 'all' proportions for token vs. type coverage gives a factor of 1.2/1.2 for the noun-association and the verb-association pairs covered by corpus co-occurrence; 1.8/1.7 in Wikipedia, and 2.3/2.0 in the dictionary. The higher the ratio is, the more of the stronger associate responses were among the ones that were found in the respective resources. We conclude, that -even though the dictionary has a lower overall coverage of stimulus-associate pairs than the other two resources-it covers rather strongly associated responses, and its knowledge therefore might be considered as a better fit to association data than the other resources. Wikipedia, then, has in turn a better fit than corpus co-occurrence. Table 11 and Table 12 address our key question, whether dictionary and encyclopaedic knowledge sources complement corpus co-occurrence and thus standard distributional analyses. As a first step towards this question, the tables provide a cross-comparison of the resource coverage with respect to the three resources. The tables show how many of the stimulus-associate pairs were covered by the resource in the first column vs. the other resources listed in columns 2-4. For example, concerning the noun-association types, corpus co-occurrence covered 55% more of the stimulus-associate pairs than the dictionary, and 46% more than Wikipedia. The tables complement the tables in Section 4.1-4.3, also showing that the corpus coverage is larger than the dictionary and Wikipedia coverage; in addition, the tables show that each resource covers some stimulus-associate pairs that the other resources do not cover, to varying degrees. The proportions in the tables refer to the stimulus-associate types, as we are interested in the kind of information that is provided by the knowledge resources.   To illustrate the differences in knowledge that is covered by the three resources, we rely on examples, in relation to the cross-comparison tables. The appendix in Section 7 lists the 10 strongest stimulus-associate pairs for each cross-comparison. Following a descriptive approach, we find stimulus-associate pairs that might indeed be considered as world knowledge and are covered by Wikipedia or the dictionary (but not by corpus co-occurrence), such as Karotte 'carrot' -orange 'orange', Zebra 'zebra' -Streifen 'stripes', Reibe 'grater' -Käse 'cheese', auftauen 'defrost' -Wärme 'heat', einfrieren 'freeze' -Kühlschrank 'fridge'. Similarly, however, we find pairs that might be considered as world knowledge and are covered by corpus co-occurrence but not by the dictionary or Wikipedia, such as Iglu 'igloo' -Eskimo 'Eskimo', Stecker 'connector' -Strom 'electricity', Zitrone 'lemon' -sauer 'sour', donnern 'thunder' -Gewitter 'thunderstorm', lehren 'teach' -Schule 'school'. These examples are based on intuitions only but nevertheless illustrate that in sum, we cannot conclude from the examples that dictionary and encyclopaedic resources provide more world knowledge than corpus co-occurrence. In order to distinguish the various knowledge types on a more general basis, one would have to work with word pairs (or other data) that are annotated for their semantic relation. Nevertheless, Tables 11-12 and the examples allow us to conclude that the knowledge in the various resources complements each other, and that combining the knowledge that is provided by corpus co-occurrence, dictionaries and encyclopaedias might therefore enhance distributional descriptions of words.

Taking size and frequencies into account
A central concern in the interpretation of the results, which we have not addressed so far, is of course the various sizes of the resources and the individual entries. In this section, we discuss the potential objections the reader might have concerning the resource sizes. (1) It is possible that we do not reach a co-occurrence coverage of stimulus-associate pairs of 100% just because our corpus is not large enough. We cannot prove that this is not the case because there is always a size limit of a corpus. The best way to address this concern would be to use the currently largest available corpus, the World Wide Web, to check the co-occurrences of stimulus-associate pairs. However, we believe that even with the largest corpus one would probably not reach 100% coverage. This assumption was recently strengthened by Schulte im Walde and Melinger (to appear) who demonstrated that the co-occurrence of verb-association pairs (from the same source as in this paper) increases with an increasing corpus, but the increase becomes smaller with a larger corpus, thus it seems to reach a ceiling. (2) The corpus is much larger than the dictionary and Wikipedia articles, where the length of the articles varied between 4 and 3,177 words in WDG, and between 33 and 21,071 words in Wikipedia. In total, including all articles we downloaded for WDG and Wikipedia, the articles contained 593,318 words in WDG and 2,848,150 words in Wikipedia. Related to the differences in the sizes of the articles and the resource parts we worked with, of course the absolute frequencies of the associates we searched for (and therefore also the a priori probabilities to find an association by itself and within a specific stimulus entry) also varied strongly. It is difficult to overcome the frequency difficulties, though. One could adjust the corpus size to the sizes of the articles (i.e., only use a random part of the corpus whose size roughly corresponds to the sizes WDG and Wikipedia provide), but corpus co-occurrence is not expected to be as focused information as a dictionary or encyclopaedia article, so this would not do justice to the corpus analysis. Alternatively, one could provide statistical association scores (such as log-likelihood) for the stimulus-associate pairs within the resources. This kind of information would add a measure of how strong the associations between stimuli and responses are, according to their expected vs. observed frequencies within the resources. However, this information does not refer to our overall question within this paper, whether we find the stimulus-response pairs within a certain resource or not. In conclusion, we cannot easily overcome the size and frequency issues related to this article. As a temporary solution, we added the resource frequencies to all example stimuli and associates in the appendix. The frequencies illustrate that a stimulus-response pair that is missing within a certain resource is not necessarily missing because the resource frequency of the respective response is too low. For example, the association Krieg 'war' appears 136 times in the dictionary "corpus" (i.e., the sum of all articles we downloaded), but it is not found within the entry of the stimulus Soldat 'soldier'; the association Licht 'light' appears 206 times in the Wikipedia "corpus", but not within the article of the stimulus Lampe 'lamp'; the stimulus abspecken 'lose weight' appears 200 times in the corpus, and the association Gewicht 'weight' appears 4,591 times in the corpus, but they are not found in co-occurrence with each other. Thus, there must be additional, resource-dependent reasons why a stimulus-associate pair is not covered by one of the resources.

Conclusion
This work presented an analysis of association norms that checked on how many and which of the semantic associates were found within corpus co-occurrence vs. dictionary and encyclopaedic articles of the respective stimulus words. We demonstrated that the resources differ in a) the parts-of-speech (nouns vs. verbs) they provide information for; b) the parts-of-speech (nouns vs. verbs vs. adjectives vs. adverbs) they predominantly use within their word descriptions, and c) the strength of the semantic relatedness between described and describing words (taking association strength as an indicator of semantic relatedness).
Even though we did not find evidence for our hypothesis that dictionary and encyclopaedic information provides more world knowledge than corpus co-occurrence, the information we found in the three resource types complements each other, to various degrees. Even taking the various sizes of the resources into account, our results indicate that a combination of the used types of knowledge resources should be helpful for distributional descriptions in data-intensive semantics, to empirically model word meaning and word similarity.

Appendix
This appendix lists the 10 strongest stimulus-associate pairs for each cross-comparison of resources. The pairs are accompanied by the association strength (i.e., how often an association was provided in response to a certain stimulus; column 1), the part-of-speech of the association (column 4), as well as the frequencies of the stimuli and the associations within the respective resources (in brackets following the stimulus/associate words).