{'fraction_non_alphanumeric': 0.025870646766169153, 'fraction_numerical': 0.0, 'mean_word_length': 5.448717948717949, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Pre-processing Tasks for Rule-based English-Korean Machine Translation System

This paper presents necessary pre-processing tasks for practical English-Korean machine translation. The pre-processing task consists of a problem that requires pre-processing and a solution for the problem. There are many differences between English and Korean, so it is difficult to resolve the differences using parsing and transfer rules. Also, source sentences often include non-word elements, such as parentheses, quotation marks, and list markers. In order to resolve the differences efficiently and make source sentences appropriate to translation system by arranging them, we propose pre-processing for source sentences. This paper studies various pre-processing tasks and classifies into several groups according to the time when the tasks are performed in English-Korean machine translation system. In experiment, we show the usefulness of the defined pre-processing tasks for generating better translation results.