{'fraction_non_alphanumeric': 0.030370161517702502, 'fraction_numerical': 0.010834978048092803, 'mean_word_length': 5.340143003064352, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Human Associations Help to Detect Conventionalized Multiword Expressions

In this paper we show that if we want to obtain human evidence about conventionalization of some phrases, we should ask native speakers about associations they have to a given phrase and its component words. We have shown that if component words of a phrase have each other as frequent associations, then this phrase can be considered as conventionalized. Another type of conventionalized phrases can be revealed using two factors: low entropy of phrase associations and low intersection of component word and phrase associations. The association experiments were performed for the Russian language.


Introduction
A lot of approaches have been proposed for automatic extraction of idioms, collocations, or multiword terms from texts as potential candidates for inclusion in lexical or terminological resources (Bonial et al., 2014;Gelbukh and Kolesnikova, 2014;Pecina, 2010;. However, developers of computational resources need clear guidelines for the introduction of phrases into their resources. Special instructions on introducing multiword terms exist for constructing information-retrieval thesauri (ANSI/NISO, 2005). Developers of WordNet-like thesauri, a very popular type of resources, discuss the problem of introducing multiword expressions in their resources in several works Vincze and Almasi, 2014). For example, it is supposed that wordnets have to include only lexicalized concepts as synsets (Miller, 1998). However, Agirre et al., (2006) stress that boundaries of lexicalization are very difficult to draw. Bentivogli and Pianta (2004) argue that there is a necessity to include non-lexicalized phrases into wordnets.
Multiword expressions comprise a broad scope of phrases including idiomatic expressions, noun compounds, technical terms, proper names, verbparticle and light verb constructions, conventionalized phrases, and others (Calzolari et al., 2002;Sag et al., 2002;Baldwin and Kim, 2010). For some of these constructions, such as idioms, it is evident that they should be included in computational lexicons. But for many of other expressions, for example, conventionalized phrases, it is not easy to make a decision about the necessity of their inclusion. To distinguish a multiword expression, it is important to analyze if it has any "idiosincrasies", which can be lexical, syntactical, semantical or statistical.
Conventionalized phrases have statistical idiosyncrasy and usually only one approach is proposed in literature to distinguish such phrases from other compositional phrases. This is so-called substitutionability test, which shows if the phrase components can be easily substituted with their synonyms (Sag et al., 2002;Farahmand et al., 2015;Farahmand and Henderson, 2016;Pearce, 2001;Senaldi et al., 2016).
In this paper, we show that there are at least two more types of statistical idiosyncrasy (and related tests) to distinguish conventionalized expressions: • association idiosyncrasy when components of a phrase are highly associated with each other, and • relational idiosyncrasy when a phrase has lexical associations that significantly differ from the associations of its component words; usually it means that the phrase denotes a specific entity or process with a set of its own properties and relations.
We provde evidence for these types of phrase idiosyncrasy in association experiments in Russian, in which we asked Russian native speakers what associations they had for phrases and their component words. We have found that the human association experiment is a very efficient tool to detect conventionalized phrases with high accuracy. To the best of our knowledge, this is the first attempt to use human associations for distinguishing conventionalized phrases.
The structure of the paper is as follows. In Section 2 we consider types of phrase idiosyncrasy. Section 3 describes the specificity of RuThes thesaurus, from which we take phrases for the experiments. Section 4 presents the association experiment and its results. In Section 5 we test embedding models on their capability to distinguish conventionalized phrases. Section 6 reviews related work concerning approaches of annotating compositionality/noncompositionality/conventionalization of noun phrases.

Types of Idiosyncrasy of Multiword Expressions
Multiword expressions are phrases that have some specificity (idiosyncrasy). Because of this, it is useful to collect them and store in lexicons and thesauri (Calzolari et al., 2002;Sag et al., 2002;Baldwin and Kim, 2010). The idiosyncrasy can be lexical when a component of a phrase appears only within this phrase (Baldwin and Kim, 2010). It can be syntactical when the syntactic behavior of a phrase differs from usual (for example, fixed word order). Semantical idiosynrasy can be revealed when the meaning of a phrase cannot be inferred from the meanings of its components. If a phrase has one of the above-mentioned types of idiosynrasy it can be called a lexicalized expression (Sag et al., 2002;Baldwin and Kim, 2010).
Statistical idiosyncrasy presupposes that the components of a phrase co-occur more often than expected by chance. Besides, the frequency of phrases with statistical idiosynrasy is much higher than the frequency of the phrase with one component changed to its near-synonym (weather forecast vs. weather prediction), as the result of the substitutionability test (Sag et al., 2002;Farahmand and Henderson, 2016). Phrases with statistical idiosyncrasy (often called conventional-ized phrases) can be syntactically and semantically compositional.
In many cases conventionalized phrases are difficult to distinguish. For example, one of the often mentioned conventionalized phrase traffic lights looks fully compositional. However, if we examine the meaning of this phrase, we can see that the denoted entity can be categorized as a road facility; it has signals; it is usually constructed on road intersections; it is needed for regulating road traffic, etc. This means that the phrase traffic lights has thesaurus relations with the correponding words (facilities, road, signals, regulation) that cannot be inferred from the meanings of its component words traffic and lights.
A lot of similar examples can be found. Compositional seat belt has relation to the safety concept. Food courts are usually located in shopping centers, and therefore compositional phrase food court has relation with the shopping center concept, etc. These relations can be very useful in such NLP applications as textual entailment.
Thus, we can suppose that conventionalized phrases have not only statistical idiosyncrasy, but also relational idiosyncrasy, which can be revealed easier than using the substitutionability test. The same idiosynrasy can be found in unclear cases of possible lexicalized expessions.
In (Mel'čuk, 2012) so-called quasi-idioms are discussed. According to Mel'čuk, a phrase AB is a quasi-idiom or weak idiom iff its meaning: 1) includes the meaning of both of its lexical components, neither as the semantic pivot, and 2) includes an additional meaning C as its semantic pivot. Mel'čuk (2012) gives an example of barbed wire, which is an obstacle, but neither barbed nor wire are obstacles. Thus, it seems than semantic pivot in this case is the hypernym relation, that cannot be inferred from the phrase component words. It means that the quasi-idiom is a subtype of relational idiosyncrasy.
In this paper we show that this relational idiosynrasy can be found in association experiments with native speakers. Besides, we can also reveal the association idiosyncrasy of conventionalized phrases in these experiments.
thesaurus RuThes 1 (Loukachevitch and Dobrov, 2014). The RuThes thesaurus is a linguistic ontology for natural language processing, i.e. an ontology, where the majority of concepts are introduced on the basis of actual language expressions.
RuThes has considerable similarities with WordNet: the inclusion of concepts based on senses of real text units, representation of lexical senses, detailed coverage of word senses. At the same time, the differences include attachment of different parts of speech to the same concepts, formulating names of concepts, attention to multiword expressions, the set of conceptual relations, etc.
In particular, the developers of the RuThes thesaurus have special rules for including phrases that appear compositional into the thesaurus. Such phrases are introduced if they have specificity in relations with other single words and/or expressions (Loukachevitch and Lashevich, 2016). The following subtypes of these expressions can be considered: • A phrase is a synonym to a single word; for example, çåìåëüíûé ó÷àñòîê (landing lot) is a synonym to word çåìëÿ (land), or a phrase has a frequent abbreviation: çàðàáîòíàÿ ïëàòà çàðïëàòà (employee wages); • A phrase has a synonymous phrase and this fact cannot be simply inferred from the components of the phrase: ìîáèëüíûé òåëåôîí (mobile phone)ñîòîâûé òåëåôîí (cell phone); • A phrase generalizes several single words. Such phrases as òðàíñïîðòíîå ïðîèñøåñòâèå (transport accident) or ó÷åáíîå çàâåäåíèå (educational institution) often look compositional but they have a very important function of knowledge representation: they gather together similar concepts; • A phrase has relations that do not follow from its component words. For example, the compositional phrase äîðîaeíîå äâèaeåíèå (road traffic) has numerous relations with other phrases that cannot be inferred from its components, for example, hyponyms (lefthand traffic, one-way traffic), related concepts (car accident, traffic jam), etc.

Association Experiment
For the experiment, we took two-word noun phrases (Adjective + Noun and Noun + Nounin-Genitive) that have high frequency in Russian newswire text collections. The multiword expressions were of two main groups. The first group (Thesaurus group) included multiword expressions from the RuThes thesaurus. We chose phrases that either look fully compositional (increase of prices) or that have one of components is used in a known (=described in dictionaries) metaphoric sense. This group contained 15 phrases. Another group of phrases comprised fully compositional noun phrases not included in the thesaurus, for example, end of January, mighty earthquake, result of work, etc. The non-thesaurus group contained 36 phrases.
We asked respondents (mainly university students) to think of single-word associations to noun phrases. In a separate experiment, we collected associations to the component words of the same phrases. We wanted to understand if the collected associations can serve as a base for distinguishing thesaurus phrases from non-thesaurus phrases (and as a consequence, conventionalized phrases from non-conventionalized). Twenty six native speakers gave their associations for the thesaurus phrases and twenty nine respondents participated in the experiment with non-thesaurus phrases. Forty seven people gave associations for single words.
The study was conducted via Google Forms. The respondents were asked to provide singleword associations. However, some participants could think only of multiword expressions. Such associations were also taken into account. Table  1 contains examples of obtained associations and their frequencies for some thesaurus phrases.
From the associations obtained, we calculated the following characteristics (Tables 2, 3): • entropy of answers for single words and phrases (currently, only entropy of phrase associations was found useful and included in the tables); • intersection between associations of component words and phrase associations (columns Ph1 and Ph2 in Tables 2, 3); and • number of times when one component word served as an association of another component word (columns A12 and A21 in Tables  2, 3). Table 2 contains the results for the thesaurus phrases, and Table 3 shows partial results for the non-thesaurus phrases.
We can see that for thesaurus phrases, the components are associated with each other more often than for non-thesaurus phrases. The average value of such associations for thesaurus phrases is 10 times greater than for non-thesaurus phrases. For some thesaurus phrases, both components are highly connected with another component. Withing non-thesaurus phrases, such frequent mutual associations were not found.   phrases included: ïðîãðàììíîå îáåñïå÷åíèå (software program), çåìåëüíûé ó÷àñòîê (landing lot), êâàäðàòíûé ìåòð (square meter), ýëåêòðîííàÿ ïî÷òà (electronic mail), çàðàáîòíàÿ ïëàòà (employee wages), ìîáèëüíûé òåëåôîí (mobile phone), ëåíòà íîâîñòåé (news feed), and òîðãîâûé öåíòð (shopping center). Besides, we found that the average level of entropy (4.07) of phrase associations is much higher for non-thesaurus phrases than for thesaurus phrases (3.24). This means that associations of thesaurus phrases are more concentrated, more motivated by the phrase. But at the same time some clearly compositional non-thesaurus phrases also have fairly low entropy of associations, for example, ïðåññ-ñëóaeáà àäìèíèñòðàöèè (press-service of the administration).
We can also see that the phrases differ in the number of intersections between the associations obtained for a phrase and for its components. It seems natural that the already found conventionalized phrases have numerous intersections of this kind (Table 2) because the phrase and its components are closely related to each other.
On the contrary, other thesaurus phrases have a relatively small number of such intersections.  It means that the thesaurus phrases evoke their own associations more often. For example, the phrase ïîâûøåíèå öåí (increase of prices) has frequent associations with the words èíôëÿöèÿ (inflation) (16 of 25) and êðèçèñ (crisis), which were not mentioned as associations for its component words. On average, intersection between associations of the phrase and its component associations for non-thesaurus phrases is four times less than for thesaurus phrases.
It can also be seen that non-thesaurus phrases with low entropy of associations can have large numbers of intersections between the component associations and the phrase associations. In such cases, low entropy of the phrase associations is mainly detemined by its components, for example, their probable syntactic dependencies. Only one of the non-thesaurus phrases has both low entropy of phrase associations and a few number of intersections of the phrase and component associations at the same time: íà÷àëî ãîäà (begin-ning of the year). It is highly associated with calendar months: January and September. For thesaurus phrases, a relatively high number of intersections between the phrase and component associations was revealed for most arguable thesaurus phrases: òðàíñïîðòíîå ïðîèñøåñòâèå (transport accident) and òåìïåðàòóðà âîçäóõà (air temperature).
Thus, we can suppose that if a phrase has a low level of entropy of associations together with a small number of the same associations for the phrase and its components then it is also conventionalized.
We can introduce the threshold as 0.8*Max-Entropy of answers. MaxEntropy is the maximal entropy we can obtain if respondents give equiprobable answers. In the current experiment, the threshold is equal to 3.76 for thesaurus phrases and 3.89 for non-thesaurus phrases. In our experiment, such conventionalized phrases include ó÷åáíîå çàâåäåíèå (educational institute), ïîâûøåíèå öåí (increase in prices), äîðîaeíîå äâèaeåíèå (road traffic), ãëàâíûé ãåðîé (main hero), ìåäèöèíñêàÿ ïîìîùü (medical aid).
A a result, we can say that we have found two signs of phrase conventionalization in the association experiment described: • component words are frequently associated with each other, and • associations of a phrase have both low entropy (less than 0.8*MaxEntropy) and a low level of intersection between component and phrase associations (less than 20%).
Using all three factors (association of component words to each other, entropy of phrase associations, and intersection of component word associations and phrase associations), it is possible to differentiate thesaurus phrases and non-thesaurus phrases with greater than 94% accuracy.
It is interesting to compare current results with the smaller amounts of associations. With this aim, we took the first 15 associations obtained for single words and phrases. The same abovementioned thesaurus phrases have frequent mutual associations between components (that is, have association idiosyncrasy).
Phrases ìåäèöèíñêàÿ ïîìîùü (medical aid) and òåìïåðàòóðà âîçäóõà (air temperature) had entropy of associations more than 0.8*MaxEntropy. Only two non-thesaurus phrases had both low entropy (less than 0.8*MaxEntropy) and the low level of intersection between assotiations of the phrase and its components: ôèíàë ëèãè (league final) and íà÷àëî ãîäà (beginning of the year). As a result, in this smaller experiment, the obtained associations can distinguish thesaurus phrases with accuracy more than 92%.

Detecting the Conventionalized Expressions with Distributional Models
We compared the results of the association experiment with the results of distributional models. In previous works, it was supposed that non-compositional phrases can be distinguished with comparison of the phrase distributional vector and distributional vectors of their components: it was supposed that the similarity is less for non-compositional phrases (Cordeiro et al., 2016a;Gharbieh et al., 2016). We used a Russian news collection (0.45 B tokens) and generated phrase and word embeddings with word2vec tool. For the phrases under consideration, we calculated cosine similarity between the phrase vector v(w 1 w 2 ) and the sum of normalized vectors of phrase components v(w 1 + w 2 ) according to formula from (Cordeiro et al., 2016a).
To evaluate different parameter sets, we located all phrases in the ascending order of similarity scores. We wanted to check if the thesaurus phrases with idiosynrasy obtain lesser values of word2vec similarity than non-thesaurus phrases without any specificity. We utilized MAP (mean average precision measure) to evaluate the quality of ordering.
We experimented with different parameters of word2vec and evaluated them with MAP on our data. We found that the best word2vec model (200 dimensions, 3 word window size) achieved quite low value of MAP ( 0.391), which means that it is very difficult for current embedding models to differentiate thesaurus and non-thesaurus phrases in our experiment.
We can also calculate MAP for the same phrase list ordered accoring to the increased entropy of phrase associations. And here we obtain MAP equal to 0.642. Thus, entropy of human associations without accounting additional factors pre-dicts thesaurus phrases significantly better than the embedding models.

Related Work
The annotation of multiword expressions on compositionality/non-compositionality of noun compounds has been studied in several works (Cordeiro et al., 2016b;Reddy et al., 2011;Ramisch et al., 2016). Reddy et al. (2011) created the set of 90 noun compounds. The phrases were taken from Word-Net. For each compound, the following types of tasks have been given: a judgement on how literal the phrase is and a judgement on how literal each noun is within the compound. They used 30 turkers to obtain judgements on the compound compositionality in each task. Ramisch et al. (2016) asked respondents about the degree to which the meaning of an expression follows from its components: separately from each component and from both components in total. The authors of the paper stress that such indirect annotation provides reliable and stable data. However, this approach was confronted with difficulties concerning the inconsistency of the answers in some cases. For example, English speakers agreed on the level of head and head + modifier compositionality for phrase dirty word, but disagreed when judging the modifier: it was fully idiomatic for some, but others thought that the phrase just contained an uncommon sense of dirty.  try to formulate the procedural definition of multiword lexical units that should be included in the Polish wordnet so that lexicographers could apply these principles consistently. Then they asked linguists to classify phrases using this definition into three categories: multiword lexical unit, not multiword lexical unit, and don't know. They concluded that a group of 5-7 linguists is able to decide whether multiword lexical units should be introduced in a wordnet with the appropriate agreement. However, this approach was considered too expensive.
In another experiment,  directed linguists to answer questions based on non-compositionality criteria of phrases including metaphoric character, hyponymy toward the syntactic head, ability to be paraphrased, nonseparability, fixed word order, terminological register, etc. Then the answers were used to train the decision tree algorithm to predict inclusion or non-inclusion of an expression into the Polish wordnet. However, the obtained decision trees were different for the various phrase sets under analysis. Farahmand et al. (2015) describe the annotation of non-compositionality and conventionalization of noun compounds. They asked the annotators to make binary decisions about compositionality of phrases. Compositional compounds were further annotated as conventionalized or nonconventionalized. A compound was considered as conventionalized in neither of its constituents can be substituted for their near-synonyms. Sometimes the decision was diffucult because such phrases could really exist (floor space vs. floor area).
To annotate the compounds, five experts were hired. In such a way, the authors (Farahmand et al., 2015) tried to avoid problems with crowdsourcing, which can lead to flaws in the results (Reddy et al., 2011). The authors stress that identifying conventionalization is not a trivial task and that human agreement on this property can be quite low. The examples of found compositional, but conventionalized phrases included: cable car, food court, speed limit, etc. The task of this study to distinguish conventionalized or non-conventionalized phrases among compositional compounds is the closest to our work.
For Russian there are two large resources of human associations. The well-known Russian Association dictionary (Karaulov et al., 1994) is currently obsolete. Another assoicaion-oriented project Sociation.org 2 collected a lot of current Russian associations but it does not have associations for the phrases under analysis.
Practical conclusions from the above-described experiments and related work are as follows: • In annotating compositionality/noncompositionality of multiword expressions by crowdsourcing as in (Cordeiro et al., 2016b;Reddy et al., 2011;Ramisch et al., 2016), it is also useful to ask respondents about their associations for the phrase and its components to detect relational idiosyncrasy, • In expert analysis of multiword expressions for inclusion into computational resources as in Farahmand et al., 2015), it is useful to ask experts about additional lexical or conceptual relations that the 2 http://sociation.org/ phrase have and that do not follow from the phrase components, • In computational approaches of extracting non-compositional multiword expressions, it is useful to compare contexts of phrase occurrence and contexts of its component word occurrences trying to detect weirdness in the phrase context.

Conclusion
In this paper, we have shown that if we want to obtain human evidence about conventionalization of some phrases, we can ask native speakers about associations they have for a phrase and its component words. We have found that there are two forms of manifesting conventionalized phrases. First, we can consider that a phrase is conventionalized if its component words have frequent associations to each other. The second type of conventionalized phrases can be revealed on the basis of two factors: low entropy of phrase associations and a low number of intersections between component word and phrase associations. These three factors allows predicting conventionalized phrases with high accuracy. We have also shown that the existing embedding models distinguish conventionalized phrases from non-conventionalized significantly worse.
In our opinion, developers of thesauri should consider the relational specificity (idiosynrasy) of multiword expressions, which can help them to decide on inclusion of specific phrases into their resources. Weird word co-occurrences with the phrase in comparison with its component contexts can be considered as an additional factor to detect conventionalized expressions in computational approaches.