{'fraction_non_alphanumeric': 0.017582417582417582, 'fraction_numerical': 0.006593406593406593, 'mean_word_length': 5.155405405405405, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Binary Feature Classification for Word Disambiguation in Statistical Machine Translation

In statistical machine translation, there are many source words that can present different translations. The usual approach to disambiguating these words is to use a target language model. These models are based on local phenomena and in many cases are not capable of properly translating some of these words. To deal with this problem, a new approach is proposed that is based on the so-called naive Bayesian classifier. After a process of automatic selection of ambiguous words, an adequate binary feature set that maximizes the information gain is chosen. A Bayesian classifier can be used to choose the most adequate translation of a word, using this feature set. Experimental results on a parallel corpus (approximately 650,000 Spanish-Catalan sentence pairs) show the benefits that the proposed method can achieve.