{'fraction_non_alphanumeric': 0.02990817998247419, 'fraction_numerical': 0.01847830228216558, 'mean_word_length': 4.587058322690506, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Relating Keywords to the Top Ten News of the Year in Korean Newspapers

This paper takes an in-depth look at the relationship between mechanically extracted keywords and ‘Top Ten News of the Year’ compiled by the news editors. A previous study that briefly touched on the topic concludes there does not seem to exist any meaningful connection between the two. In this paper, we set up a more elaborate way of comparing and connecting the two, and argue that there is a certain reasonably good converging point. The corpus we make use of for our experiment is a subset of the Trend 21 corpus which is a collection of Korean major newspapers (2000-2013). For keyword extraction, loglikelihood ratio was made use of. Extraction of collocation for each keyword was needed, for which a version of Mutual Information was utilized. Finally a detailed comparison of the top ten news with the top 100 keywords was conducted from several points of view.


Introduction
There is a growing use of the keyword methodology as an analytic tool to efficiently analyze texts or corpora, and we can say it now has established itself as a viable, and, importantly, objective alternative to the traditional and rather introspective method of discourse or cultural interpretation (Scott & Tribble, 2006, Bondi & Scott, 2010Archer, 2009;Baker et al., 2013). One issue that needs to be addressed is how the new methodology relates to the introspective way of selecting keywords from a given text or corpus. Are the two supposed to be different from each other? Then why so? Or do they have to be comparable or even identical to each other? And if they do, how can we test the comparability or convergence?
In this paper we raise these questions on the basis of Korean newspaper corpora and some lists of 'Top Ten News (T10N for short)' compiled, presumably introspectively, by the newspaper editors at the end of every year. Specifically we compare the top 100 keywords with T10N, and see how well they converge with each other. Some previous studies seem to suggest a tentatively negative conclusion on the issue of any systematic relationship between the introspective and quantitative keywords in general (Bondi & Scott, 2010), or between quantitative keywords and T10N .
It should be noted at the outset that each item in T10N is not a keyword per se. Though it can trivially turned into a small set of keywords on the basis of the surface description of the news item, we take the item as an abstract concept to which a set of keywords can be mapped, thus making it possible to compare the quantitatively derived keywords and the more abstract key concepts that are selected introspectively. We argue that T10N provides an interesting testing ground for the significance of the keywords extracted or the role of the 'human factor' in the selection process of T10N.
This paper briefly reviews some of the previous studies on the issue at hand in Section 2, introduces the corpora used in Section 3, provides in Section 4 two lists of T10N to be analyzed, discusses methodological issues in Section 5, and reports the results in Section 6 with related discussion, which is then followed by conclusion.
According to Stubbs (2010), there are three kinds of 'keywords', two of which are of our immediate concern in this paper: One is the 'cultural keywords', for example like the one compiled by Williams (1976/85), that are intended to capture the essence of a culture through selection of the terms or keywords that would represent some major aspects of the culture. It certainly would involve processes of understanding, interpretation, and abstraction on the part of the person that does the compiling job. In other words, they represent human interpretation and understanding of the phenomena, i.e., the culture. Let us call them 'introspective keywords', focusing on the methodology, so that they may cover not only the cultures but also other broad aspects of society. The other concept of keywords refers to the analytic tool mentioned in the previous section. They are called 'statistical keywords', which are extracted on the basis of the (relative) frequency distribution of words in the given corpora.
How are the two related to each other? Previous studies touched on the issue of the relationship between the two kinds of keywords, and suggested that they should be treated as separate kinds, little significant relationship being observed.
For example, Stubbs (2010: 32) contends that they are "only loosely conceptually related, and perhaps only marginally compatible." Scott (2010: 45) also states "[i]t is perfectly true that automatic analysis works differently from human identification in the case of keyness …." It may be true that human selection of the key words or phrases to capture the bigger picture of the society or culture is different from purely mechanical extraction of keywords from discourse. However it would also be true that the bigger picture is formed through discourse. It is mediated, communicated, and created through discourse. In other words, introspection based human keywords are not created from nothing; ideally they should reflect well the culture or society in question, and the discourse that constitutes and represents the society. Again ideally if we can get hold of the whole set of discourses which presumably reflect the culture and society as a whole, we can expect there would be a certain converging point between introspective keywords and quantitative keywords.
We will assume one of such case can be provided by news texts of a certain period of time. In particular, the top ten news summarize a year's major events which had been reported in the news articles of the year. Suppose we have fairly large corpora that reflect the social events from which we can extract keywords on the one hand, and also human interpretation of the major events in the form of T10N on the other. How would they match up with each other then?
There is one previous study that dealt with such question. ) compared T10N of 2009 and the keywords of the same year based on the Korean major newspapers, and found out only eleven out of 100 keywords match T10N. They concluded that "[the two] are more different from each other than would be expected (2011: 178).
[Translation by the author]" While the presumably introspective selection of the major news of the year need not perfectly match the keywords, it is surprising that the major news does not seem to reflect the texts of the newspapers. This paper will take a careful look at this question again, using similar but more fine-tuned sets of data and different sets of tools for keyword extraction and interpretation. We find there is an interesting and meaningful converging point between the two, and even the items that do not match well seem to shed some light on how the human interpretation process works.

Target and Reference Corpora
T10N are selected annually, so it is reasonable to assume that the most relevant discourse would be the whole news of the year.
For this experimental study, we are going to use the same set of T10N as that in , but we use different subsets as the target and reference corpora. In order to explain the difference, we first need to introduce the Trend 21 Corpus from which we take a subcorpus.
The Trend 21 Corpus is a collection of newspaper articles from 2000 to 2013 , Choe & Lee, 2014. All the newspaper articles in four major daily newspapers published in Korea (Chosun, Dong-a, Joongang, and Hankyoreh) constitute the corpus, and new data are processed and added after the end of each year when the data are provided by respective news media. The corpus currently contains over 600 million 'ejels', or chunks between spaces in Korean which typically consist of a content word and some agglutinating particles.
For the experiment in this paper we mainly use a subset of the corpus, primarily the data that cover the four year span (2006-2009) of a newspaper, Chosun, referring to a larger set when necessary. Specifically, we take the news texts of 2009 of Chosun as the target corpus, and those of 2006-8 of the same newspaper as the reference corpus. We assumed the corpus of the previous three years as the reference would be "moderate sized (Scott, 2010:52; See also Jeon & Choe (2009))". This is different from  where they took all the news texts from the four major newspapers as the target and reference corpus, and then compared the results with T10N of Chosun. As is well known, newspapers may differ among themselves in terms of their respective stance on the social, cultural, and especially political issues (Baker et al., 2013). Thus assuming possibly different stances may influence the composition of each news texts and also the selection of T10N, we decided to compare the keywords from a particular newspaper with their own selection of T10N, thus limiting the effects of other factors to the minimum.

Lists of Top Ten News
There are typically two kinds of T10N compiled by each newspaper in Korea. One covers the national events, and the other international ones. Chosun had the following news items as their selection of T10N for the national and international major events of the year 2009, respectively.

Id
News item Note that the English translations of the news items provided in the above tables are not exactly the same as they appear in the newspaper but somewhat in an abbreviated and compact form, again to save the space. We also took the liberty of ignoring some metaphoric descriptions, and added

PACLIC 28
! 31 some extra information so that those that are not familiar with the events may get a better grasp of them. For example, 'cn1' would be literally translated as "Kim Swu Hwan, Kim Dae Jung, Roh Moo Hyun … Major Figures in modern history are now in history," which simply refer to the deaths of the three major players in Korean politics and society for over 40 years. We also added the "Classification" column, again as a way to help the readers, especially those that are not familiar with the events described, to comprehend the overall picture as well as the characteristics of each event.

Keyword extraction
The statistical procedures typically used for keyword extraction are Dunning's Log Likelihood (LL) and chi-square (Scott & Tribble, 2006). Some authors used T-score for the calculation . The standard text tools such as WordSmith (Scott, 2012) and AntConc (Anthony, 2011) provide keyword extraction procedures like log-likelihood and chi-squared, and it is generally known that there is not much difference between the two (Rayson, 2003;Bondie & Scott, 2010). In this paper, we used a version of LL described in Rayson (2003: 50). Let us suppose we have the following contingency table. Corpus Size c d Table 3: Contingency table Then the log-likelihood ratio is calculated as follows, where N refers to the total value of the four cells.
The formula was implemented in a Perl script, rather than using any of the well-known tools, because the size of the data for the current study was rather huge and it was not easy, if not impossible, to handle them in the readily available tools. In order to confirm that the custom-made script works as expected, some test results were compared with those from WordSmith and AntConc on the basis of the same set of data, a Shakespearean play Romeo and Juliet, and there were minimal differences among the three results.
Since our concern in this paper is the topic rather than the style of the data, we limited our search to the words/morphemes that are nouns (/NNG) and proper names (/NNP), ignoring all the other categories.

Collocation extraction
For many of the extracted keywords, it was obvious which T10N item each of them belong to. But for many others, the connection was not that clear. There were several reasons for this. For one thing, a keyword may be ambiguous. For example, 지원[jiwon] in Korean may either mean 'support' or 'application', and we need to figure out in which sense the word was selected as a keyword. Another reason is that it was difficult to decide in which context a certain keyword was used. 중소기업[jungsogieop] means 'small and medium sized enterprises', and it is difficult to know whether it has anything to do with T10N or not. A third reason is that some keywords may be linked to more than one item in the T10N list. 오바마(Obama), as President of the most influential country in the world, can obviously be related to many news items. Finally there were a few keywords with baffling identities. 김정운 [Kim Jung-un], apparently a personal name, was listed as a keyword, and it was not clear at first why the name cropped up as a keyword.
These problems can be solved if we take the context into consideration, of course. A widely used method is to browse the keywords in the KWIC style. But when there are so many data to be checked, a more efficient method is called for which will succinctly summarize the contextual information. One such method would be collocation, which looked good enough for our purpose so we made use of it in this study. Thus for each keyword, a set of collocation words, or more exactly a set of morphemes were gathered that co-occur in the same news item.
There are well-known collocation extraction methods like the t-test and Mutual Information. While the t-test seems to have some issues with low frequency words, Mutual Information has been considered too skewed to them, assigning a very high value, for example, to a bigram whose members occur only once in the given corpus. In this paper, we used a version of Mutual Information, called Log-Frequency based Mutual Dependency (LFMD, Thanopoulos et al., 2002), which is designed to add some frequency effect to Mutual Information. The metric is given below, which was again implemented in Perl: where D is:

Results and Discussion
Once the keywords were extracted and sorted in a descending order of their LL value, we checked each of the top 100 keywords for possible matchup with the twenty items of the national and international T10N provided in Tables 1 and 2. The collocation word list for each keyword was constantly consulted in the process. A sample of the table used for the process is provided in Appendix at the end of this paper.

Keywords that relate to the national T10N
30 out of the top 100 keywords were found to be linked to the national top 10 news. Their mutual relationship is provided in the following table, where the T10N news items and their related keywords are shown side by side. Each keyword is followed by its English gloss, and then its rank in the 100 list shown in parenthesis.  It seems like each item in T10N is reasonably well represented in the top 100 keyword list. Each item, except for 'cn2', has at least one keyword that supports its selection. Half of the national T10N ('cn1', 'cn3', 'cn4', 'cn5', 'cn6'), or five out of top 6 news, are linked to at least three top 100 keywords. Top three major news given in Table 4, namely 'cn1', 'cn3', and 'cn4', have at least five matching keywords. Overall, we might be able to say that the top five items in Table 4 support rather strongly the convergence between the introspection based major news and the statistically derived keywords.
The bottom four items in Table 4 is not that well supported by the list of top 100 keywords, but still each finds a keyword in the list that can be linked. We will come back to the missing one 'cn2' in Section 6.3.
The first three names that appear as keywords for 'cn4: The new North Korean leader Kim Jong Un, the second NK nuclear test' show why collocation information is needed for proper classification. 오바마(Obama) would rather be expected to be linked to some international news, and no doubt Obama, as President of the most influential country of the world, would be featured in many international news. However, when the collocation words of the name were checked, a crucial one seems to be '핵[haek]/nuclear'. Obviously, as much as 'Obama' appeared in many other news, the name was significantly associated with the word 'nuclear' and the most noteworthy mention of the word 'nuclear' in Korea in 2009 was in the context of the North Korean nuclear test. The same applies to another name "(Steven) Bosworth" in 'cn4'. Finally, '김정운 [Kim Jung-un]', apparently a personal name, was listed as a keyword, and even to a person that is well versed in Korean national affairs the name looked puzzling at first. Its collocation revealed the name refers to the newly emerging North Korean leader. His name was initially wrongly identified as 김정운 in the media, rather than the correct 김정은 as was later to be known through the North Korean media, befitting to the secrecy and mystery that surrounds the country.
Even with some collocation information, there were truly ambiguous cases, and thus we had to add an extra classification category 'cn5/cn4' in Table 4. The two keywords associated with the category, namely '로켓[rokes]/Rocket (15)' and '발사[balsa]/Launch(32)', when their collocated words were considered, were clearly linked either to the failed launching of the spacecraft Naroho in the South, and to the launching of the missile Daephodong in North Korea.
We therefore tentatively classified it as belonging to 'cn5/cn4'.

Keywords that relate to the international T10N
The same number of keywords, namely, 30 out of the top 100 keywords was found out to be linked to the international T10N, as shown below.
Cat: Cla. keywords(rank) ci1: Economic Crisis 위기/Crisis (9)  Expanding global economic crisis, weak dollars' and 'ci2: Swine flu caused over 10,000 deaths' were the two prominent international news of the year that were amply reflected in the keywords. They were national news as well as international ones as Korea was also affected by both the economic crisis and the epidemic, and thus people in Korea were keenly following the news.
Economic crisis and the swine flu epidemic are the two pieces of news that affect the lives of the general public very much, and obviously the news media were clearly aware of them, thus dealing with them very widely and repeatedly as the related keywords show. Likewise, global warming and the subsequent climate change is one of the grave issues that largely bother the minds of the general public. Thus the 2009 United Nations Climate Change Conference, or the Copenhagen Summit was apparently covered well in the newspaper as the keywords in 'ci7: Copenhagen summit failed to meet the expectation' show in Table 5.
The other news item in the table is about Japan. Hatoyama became the first Prime Minister from the modern Democratic Party of Japan in 2009, defeating the long-governing Liberal Democratic Party. The power change in Japan, a closely related neighboring country to Korea, was an obviously newsworthy item to Koreans, and so was covered accordingly in the news media.

T10N that do not have any matching top 100 keywords
Among the national T10N news items 'cn2' was the only exception that did not have any supportive words in the top 100 keywords. The description of 'cn2' is 'Korea will host the G20 Summit in 2010' as shown in Table 1. The key phrase in the description is 'G20', but it turns out that the tagger wrongly analyzed it as 'G' and '20'. G20 이 G /SL+20/SN +이/JKS Since the source of the problem was located, it was possible to get the log-likelihood value for the expression "G20" separately. Had it been treated as a single unit, its LL value would be 2287.92, which means "G20" would rank as the ninth item in the top 100 keyword list (See Appendix). Every national T10N in Table 1 is supported by at least one associated keyword.

Item
O1 1% O2 2% LL Word 1277 0.01 267 0.00 + 2287.92 Table 6: Log-likelihood value for 'G20' On the other hand, as for the international T10N news items, only four of them could find their linked keywords in the top 100 keyword list, as Table 5 shows. Note that three of them, the most heavily covered ones ('ci1', 'ci2', 'ci7') in the media, concern global issues which would also affect the lives of the local general public. It is highly likely they would have made national T10N even if they were not covered by the other list.
Among the rest of the news items in Table 2, five of them deal with regional issues like China, US-led war in Afganistan, Europe, Japan, and a US-North Korean issue. The other two concern well-known popular figures like Michael Jackson the pop star and Tiger Woods the sports star. Only one of these seven items has a single related word in the top 100 keyword list. It is obvious that not particularly many news articles were written on the global scale topics in the newspaper and yet the editors felt they should be included in T10N. We will come back to this point later.

Top 100 keywords that do not belong to any of the T10N items
There were 40 keywords in the top 100 list that were left out of the national and international T10N. Four of them were included due to some other factors than the news stories themselves. For example, the use of the word '편집자[pyeonjipja]/ Editor' seems to have spiked up in 2009, but it was exclusively used as part of the editorial comments to some of the articles, rather than as part of the news stories. Many of the other keywords were used individually, having little to do with other words in the top 100 keyword list. However, there were several clusters of keywords each of which seemed to point to a particular event or topic. The keywords in the 'o_edu' category concerns college entrance system, particularly the newly introduced one by universities in Korea that seemed to be gaining huge momentum, urged by the Ministry of Education. Education, especially the college entrance system is everybody's concern in Korea, and even a slight change in the system has huge repercussions on the society in general. Obviously, the new "Admissions Officer" system was one of the top national issues in 2009 and thus was much talked about in the media. The following chart shows that the use of the keywords '사정관[sajeonggwan]/ Admissions_Officer' and '입학[iphak]/ Admissions' greatly increased simultaneously in 2009 in the four major newspapers in Korea. Another hot topic which is everybody's concern is unemployment or difficulty of getting a job. Growing number of the unemployed has become a social issue. It was a much talked about issue of the year again, as the keywords in the 'o_job' in Table 7 show. Incidentally, the first keyword of the category, '잡월드[jabwoldeu]/ Job_World (5)' turned out to be the name of the website created by the particular newspaper, Chosun, together with other institutions, as part of a social campaign to help the unemployed to find a job. Out of 1,161 occurrences of the word in the four major newspapers in 2009, the vast majority (1,151) have appeared in Chosun. The other two categories in Table 7, along with their keywords, again have a lot to do with everyday life of the ordinary people: a new housing project in the metropolitan area, and newly introduced popular IT items like smartphones and the twitter.
The rest of the keywords, 14 of them, seemed to deal with individual issues separately, and did not aggregate well among themselves. One thing to note before we close this section is the personal or pen names that showed unusual degree of keyness though not related to any of the T10N.
cat keywords(rank) p_invst 박연차/Bakyeoncha (13), 건호/Geonho(98) p_ent 장자연/Chang_Jayon(48) p_soc 미네르바/Minerva(100) Table 8: Keywords of proper names The category 'p_invst' is related to the political scandal that implicated a former president, which many believe eventually led to his suicide. The two keywords in the category refer to the principal figures in the scandal, close associates of the late president. The other two categories in Table 8 are again related some social and political scandal.
Many of the keyword clusters or keyword discussed in this section could have made the national T10N list but the editors chose otherwise.

Conclusion
So how well do the statistically derived keywords and the introspection based T10N converge? Based on the results of our analyses, over 60 percent of the top 100 keywords make positive contribution to the convergence. Seen from the opposite point of view, national T10N is well supported by the keywords while international T10N is markedly less so. So we can conclude that though the two do not match with each other perfectly, they converge reasonably well.
Then what is the source of the difference between them? Here we provide some speculative remarks.
One thing that influences the introspection based selection or decision of T10N is a higher abstraction process involved. For example, 'cn3: National construction projects' is an abstraction over more than one separate event or project, and so are 'cn1: Politics/Death' and 'cn4: North Korea'. Secondly, the introspection based selection is likely to be influenced by the historical context. The choice of 'death' as the top national news of the year in 'cn1', rather than the second or the third or even below, would make more sense if we take it into consideration that the three people involved have left a huge impact in recent history of Korea. So it is not just their death, but in a sense the end of era in Korean political and social history that mattered in the selection.
The third factor that apparently plays a role in the selection of the T10N is the geographical balance, especially in the case of the international T10N, namely from 'ci3' to 'ci6' in Table 2, for which there were not to be found any related keywords in the top 100 list. The fourth factor is the sectional or topical balance of the newspaper (Tables 1, 2). Otherwise it is rather difficult to explain the inclusion of the last four items ('cn7' to 'cn10') in Table 4 at the expense of other events that are more prominently reflected in the linked keywords. The final factor that seems to matter is what we might call topic subsumption. Although the name '박연차/Bakyeoncha' and '건호/Geonho' appeared particularly frequently in 2009 (See Section 6.4), the event was eclipsed by a much bigger related news which was the death of the former president allegedly involved.