{'fraction_non_alphanumeric': 0.011387163561076604, 'fraction_numerical': 0.0, 'mean_word_length': 5.081761006289308, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Construction of Equivalence Classes of Foreign Words through Automatic Identiication and Extraction

A foreign word in Korean has its origin in the source language like English, and there may be more than one foreign word transliterated from the same word in the source language. This paper addresses this problem of one-to-many relationship between a source language words and the corresponding foreign words, as well as the problem of identifying and extracting a foreign word from a phrase. We describe our algorithm that rst iden-tiies a phrase containing a foreign word and then extracts the foreign word part from the phrase based on statistical information. The algorithm then constructs an equivalence classe for the automatically extracted foreign words that should be mapped to a single source language word. We report our experimental results for each of the three stages, which are promising enough to be used for applications like information retrieval.