{'fraction_non_alphanumeric': 0.024308466051969825, 'fraction_numerical': 0.008382229673093043, 'mean_word_length': 4.712918660287081, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Knowledge-based sense pruning using the HowNet : an alternative to word sense disambiguation

Word sense disambiguation (WSD) is one of the basic problems in natural language processing. Traditional WSD methods provide only one meaning for each word in a passage. However, we believe that textual information alone may not be sufficient to determine the exact meaning of each word which may better be resolved when higher-level knowledge becomes available. In this paper, we propose an alternative to WSD that we call “sense pruning”. The objective now is to reduce the number of plausible meanings of a word as much as possible so as to reduce the amount of work in later processing. Sense pruning is guided by information derived from HowNet a recently developed knowledge base. Two criteria were used for the evaluation: recall rate and complexity reduction (which is the reduction in the number of possible meanings of a sentence). Effect of the length of the analytical window was studied. For a corpus of 103 Chinese passages from Sinica, Taiwan, with an analytical window of nine words, we obtained a recall rate of 94.14% and reduced the number of possible sentence meanings by 65.3%.