{'fraction_non_alphanumeric': 0.024967148488830485, 'fraction_numerical': 0.0, 'mean_word_length': 4.953125, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A hybrid approach to Chinese-English machine translation

A hybrid method for Chinese-English machine translation is presented. A rule-based analysis is combined with statistical data. The rule-based lexical analyzer and syntactic analyzer leave some amount of ambiguity that is resolved using a statistical approach. A hidden Markov model (HMM) is used to return a score for each part of speech. An improved probabilistic context free grammar (PCFG) is used for syntactic scoring, and a probabilistic score function is used for the semantic aspect. In addition, parameters are usually estimated poorly when the training data is sparse. Smoothing the parameters is thus important in the estimation process. A back-off procedure is used to smooth the parameters.