{'fraction_non_alphanumeric': 0.015822784810126583, 'fraction_numerical': 0.002109704641350211, 'mean_word_length': 5.683098591549296, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Segmenting Korean Compound Nouns usingStatistical Information and a Preference

This paper presents a method of segmenting Korean compound nouns by using statistical information and a preference rule. Statistical information is represented by CFP(Compound noun Formation Probability) that consists of both frequencies of aaxes and frequencies of two-syllabled and three-syllabled nouns. A preference rule is MNPR(Minimal Noun Preference Rule) that prefers a structure pattern of a compound noun with minimal number of unit nouns. Moreover, we apply three kinds of heuristics in order to segment compound nouns including unknown unit nouns. Experimental results show that the precision of the proposed method is approximately 96% on average. Furthermore, the experiments prove the proposed method can segment compound nouns including unknown nouns and maintain the constant precision rate in segmenting compound nouns extracted from various domains.