{'fraction_non_alphanumeric': 0.020134228187919462, 'fraction_numerical': 0.0019175455417066154, 'mean_word_length': 4.553191489361702, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Some Statistical Properties and Zipf’s Law in Korean Text Corpus

Some statistical characteristics of Korean texts are analyzed by experiments on large corpora. We obtain the number of occurrences of syllables and of words in Korean texts. The entropy of syllables is estimated using finite context model. Digram and trigram entropy of syllables are also estimated. The entropy of words is estimated using the same model. We try to examine how Korean text obeys the well-known Zipf’s law. Two mathematical models are constructed by modifying Mandelbrot distribution and are simulated for Korean texts. The coefficient B in Mandelbrot distribution is determined for our models by experiment. We compare Zipf’s law in Korean text with that in English and in French. According to Mandelbrot, the coefficient B is B > 1 in all the usual cases, however, we obtain B < 1 in some range of the rank-frequency distribution of Korean text. We also checked that the coefficient B does not depend on the kind and on the size of corpus but on the language.