{'fraction_non_alphanumeric': 0.033149594242841834, 'fraction_numerical': 0.007885469300260297, 'mean_word_length': 4.684508268059182, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Automatic Extraction of Turkish Hypernym-Hyponym Pairs From Large Corpus

In this paper, we propose a fully automatic system for acquisition of hypernym/hyponymy relations from large corpus in Turkish Language. The method relies on both lexico-syntactic pattern and semantic similarity. Once the model has extracted the seeds by using patterns, it applies similarity based expansion in order to increase recall. For the expansion, several scoring functions within a bootstrapping algorithm are applied and compared. We show that a model based on a particular lexico-syntactic pattern for Turkish Language can successfully retrieve many hypernym/hyponym relations with high precision. We further demonstrate that the model can statistically expand the hyponym list to go beyond the limitations of lexico-syntactic patterns and get better recall. During the expansion phase, the hypernym/hyponym pairs are automatically and incrementally extracted depending on their statistics by employing various association measures and graph-based scoring. In brief, the fully automatic model mines only a large corpus and produces is-a relations with promising precision and recall. To achieve this goal, several methods and approaches were designed, implemented, compared and evaluated.


Introduction
In this study, we describe how to acquire hypernym/hyponymy relations from a Turkish corpora (Sak et al., 2008) in a fully automatic way. The system extracts possible hypernym/hyponym pairs by using lexico-syntactic patterns, then it expands the hyponym list depending on semantic similarity.
Few studies have been published for Turkish Language, BalkaNet (Bilgin et al., 2004) is the first WordNet project for Balkan languages such as Turkish, although the project has not yet been completed. Some attempts used a Turkish Dictionary, TDK 1 . (Yazıcı and Amasyalı, 2011;Güngör and Güngör, 2007;Orhan et al., 2011;Şerbetçi et al., 2011) All studies of semantic relation are mostly based on a Turkish dictionary. Our study is the major corpus-driven attempt at integrating lexico-syntactic patterns and a bootstrapping approach.

The Methodology
Once the system has simply extracted possible hypernyms by using lexico-syntactic patterns from a Turkish corpus of 490M tokens, it incrementally expands the list by using a bootstrapping algorithm. It uses the most reliable pattern to determine the hypernym/hyponym pairs. For each hypernym, the most reliable candidate hyponyms (seeds) are passed to the bootstrapping algorithm. The algorithm incrementally expands the seeds by adding new seeds depending on a scoring function. The approach employs two different patterns; one is a lexico-syntactic pattern to obtain is-a pairs and the second is a syntactic pattern to compute co-occurrence of the words in a fine-grained way.

Candidate Hypernym/Hyponym
The most important lexico-syntactic patterns for Turkish are: 1. "NPs gibi CLASS" ( CLASS such as NPs), 2. "NPs ve diger CLASS" (NPs and other CLASS) 3. "CLASS lArdAn NPs" (NPs from CLASS) 4. "NPs ve benzeri CLASS" (NPs and similar CLASS) The most reliable pattern is the first pattern that matched over 200,000 cases in the corpus from which 500 reliable hypernyms could be compiled.

Elimination Rules
Some incorrect hyponyms are extracted due to some factors. The objective of this step is to exclude these kinds of non-hyponyms and to acquire more reliable candidates. A partial exclusion can be performed as follows; • In the first pattern,we observed that real hyponyms tended to appear in the nominative case. The rule implies if a noun was not in nominative case, it would be eliminated. • If an item occurs only in a single match with the pattern, it will be eliminated. The assumption is that some matches to the pattern are accidental. • The more general a word is, the more frequent it is. This rule is that, if a candidate hyponym has a higher frequency (df) than its hypernym, it will be ignored.

Statistical Expansion
Filtered hyponym list can remain some erroneous candidates. To improve precision, we can sort the candidates by their pattern frequency. The first K of these words can then be used as original seeds for expansion phase, where K can be experimentally chosen (e.g. 5). The system expands the seeds recursively by adding new seeds one by one. The algorithm will stop producing when it reaches sufficient number of items.
Bootstrapping Algorithm: The algorithm is designed as shown in FIGURE1-A. It first extracts hypernym/hyponym pairs and then applies bootstrapping with a scoring function, where ascoring-f denotes an abstract scoring function for selecting new hyponym candidate. Our scoring methodologies can be categorized in two groups. The first is based on a graph model, the other simply uses semantic similarity between candidates and seeds. We call the former graph-based scoring and the latter simple scoring. All scoring functions take a list of seeds and propose a new seed.
Graph-Based Scoring: Graph-based algorithms define the relations between the words as a graph. Each word is represented as a vertex and the relation between the words is represented as weighted edge. Some researchers proposed a similar approach (Widdows and Dorow, 2002). Graph-based scoring was implemented as in FIGURE1-B in which each neighbor is compared not only with seed words but also with other neighbors to avoid infections. Simple Scoring: This method employs only the edge information between each candidate and the seeds. Therefore, the candidate which is the closest to the centroid of all seeds will be the winner. As shown in FIGURE1-C, the algorithm computes the similarity between a candidate and the seeds.
Edge Weighting: Both graph-based and simple scoring functions employ a similarity measurement to make a decision. Edge weighting schema that we used in the study are as follows: 1. IDF/co-occur: co-occurrence * inverse document frequency (IDF) of candidate. 2. Binary: If a seed and a candidate co-occur at least once in the corpus: 1, else 0. 3. Dice: occur(i, j)/( f r eq i + f r eq j ) where occur(i, j) is the number of times the wor d i and wor d j co-occur together, and f r eq is the number of times a wor d occurs in corpus. 4. Cosine similarity: To compute the similarity between the words, a word space model in which words are represented as vectors is used. The words can be represented in a matrix. cel l i j represents the number of times wor d i and wor d j co-occur together. The matrix is a simple representation of a graph. Co-occurrence can be measured with respect to sentences, documents, or a given window of any size. The conventional way to compute co-occurrence is to use all neighbors within a window by eliminating stop words. This approach has proved to be good at capturing sense and topical similarity (Manning and Schütze, 1999). For example, train and ticket can be found to be highly similar by this method. However, we need to apply more fine-grained methodologies to capture words sharing the same type such as train and auto or ticket and voucher.
To obtain such type similarity, the solution is to use syntactic patterns for computation of co-occurrence. For instance, nouns are considered similar when they are in particular patterns such as " N and N" or "N,N,...,N and N". A similar approach was also used by (Cederberg and Widdows, 2003). Words (nouns) considered similar would either all be subject, or all object or all indirect object. This approach makes the model more fine-grained than other conventional ways of computing bi-grams.

Experimental Setup and Implementation
We implemented a utility program which can be used to verify and reproduce the results presented in the paper. We used a web corpus of 490M tokens and a morphological parser as language resources (Sak et al., 2008). The model parses the corpus and converts each tokens into the form of surface/lemma/pos. For the experiment and evaluation, the most frequently occurring hypernyms is selected. All settings are described as follows: 1. Lexico-syntactic pattern (pattern): After extracting instances, some candidates are eliminated by elimination rules as described before. 2. Graph Scoring/binary (gr-bin): All distance/edges of the graph are weighted in a binary way. 3. Graph Scoring/co-occurence (gr-co): The edges of the graph are weighted by cooccurrence of words. 4. Simple Scoring/binary (sim-bin): Distance between words is 1, if they co-occur; else 0.

Simple Scoring/dice (sim-dice):
Edges are weighted by dice coefficient. 6. Simple Scoring/co-occurrence(sim-co): Edges are weighted by the co-occurrence frequency between words. 7. Simple Scoring/cosine (sim-cos): The words are represented as vectors in a matrix. Edge is the cosine similarity between word vectors.

Results and Evaluation
For the evaluation phase, we checked the model against 17 selected hypernyms; country, city, mineral, sport, illness, animal, fruit, bank, event, vegetable, newspaper, tool, profession, device, drink, sector and organization. In order to measure the success rate, we manually extracted all possible hyponyms of all the classes.  We tested the system within the seven different settings described above. The pattern extracted a number of hypernym/hyponym pairs. The expansion algorithms take the first five candidates as initial seeds (IS) suggested by the pattern module, then expands them to the size of the pattern capacity. Looking at TABLE 1, it seems that pattern module outperforms other expansion algorithms in terms of precision. In order to improve recall, we conducted a second experiment; the expansion algorithms expand IS to the size of actual hyponym list rather than the pattern capacity. And we eventually get a better recall value as shown in TABLE 2.
As third experiment, we incrementally altered the number of IS to investigate changes in recall. We used 10, 15, 20, 25, 30 and the pattern capacity as IS size. The pattern capacity is the number of the entire output proposed by the pattern. The average results are shown in TABLE 3. The results indicate that increasing IS gets better accuracy. This is because pattern module indeed gives promising results but it is limited. TABLE 1 shows that the average score of the pattern is % 90. Since this accuracy is reliable, the expansion algorithms can simply and reliably exploit the outputs of the pattern algorithm.  There is no significant differences between the accuracy of the different expansion algorithms. gr-bin, sim-bin and sim-co seem to be the best scoring functions. The graph-based algorithms and cosine similarity weighting are costly and time-consuming. We computed the bi-gram information and weighted our graph by using specific syntactic pattern in a more fine-grained manner. It means only the words co-occurring in "N,N and N" pattern are accepted as bigram. Therefore sim-co or sim-bin which simply computes the relation are very successful. When looking a troublesome hypernyms having low accuracy in all tables, we face a classical word sense problem. Depending on the sense distribution, the expansion algorithm changes the direction of sense into frequently used senses.

Conclusion
In this paper, we proposed a fully automatic model based on syntactic patterns and semantic similarity. We utilized two patterns: First, the most productive and reliable lexico-syntactic pattern was used to discover is-a hierarchy. We observed that hypernym/hyponm pairs are easily extracted by means of the pattern for Turkish Language. In order to get more precision, we designed some elimination criteria. It gave higher precision but a limited number of pairs with low recall. Second, syntactic pattern was used to compute co-occurrence and expand the list to get higher recall. To discover more hyponyms, we designed a bootstrapping algorithm which incrementally enlarged the pair list.  In this modular system, we conducted several experiments to analyze is-a semantic relation and to find the best setup for the model. When we look at the third experiment as shown in TABLE 3, pattern algorithm gave promising results. This module successfully built initial seeds. In order to solve the recall problem, we improved the model capacity to discover new candidates. Both graph-based and simple scoring methodologies were applied and we observed that both approaches had a good capacity to get higher recall, such as 71.6 and 72.5.
A real application could be designed as follows: the all reliable candidates proposed by the pattern method might be used as initial seeds to make the model more robust. Moreover, the pattern module can be refined to obtain more secure candidates. For the sake of simplicity, a simple scoring method with binary weighting (sim-bin) would be the best setup with respect to the results.
The results showed that the fully automated model presented in the paper successfully disclose is-a relations by mining a large corpus. In future work, we will design a preprocessing phase in order to avoid the problems coming from polysemy and other factors.