{'fraction_non_alphanumeric': 0.01773049645390071, 'fraction_numerical': 0.0, 'mean_word_length': 5.277777777777778, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Statistical Machine Translation : Robust parameter estimation from noisy corpus

In this report, we describe our study of effect of noise on parameter estimation for statistical machine translation. So far, no study has been done on this topic, even though the algorithm used for parameter estimation for statistical machine translation (the EM algorithm) is known to be highly sensitive to noise. We present in detail the experiments performed to observe the influence of noise on parameter estimation, and the various methods investigated to counter this effect.