{'fraction_non_alphanumeric': 0.023109243697478993, 'fraction_numerical': 0.012605042016806723, 'mean_word_length': 5.4391891891891895, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Fundamental and New Approaches to Statistical Machine Translation

Statistical Machine Translation (SMT) is an approach to automatic text translation based on the use of statistical models and examples of translations. Although Machine Translation (MT) systems developed according to other paradigms are still in use, mainly rule-based or example-based MT, SMT dominates academic MT research and has gained significant commercial interest over the last two decades. Machine Translation was conceived as one of the first applications of the newly invented electronic computers back in 1940’s. Communications between Warren Weaver (director of the Natural Sciences Division of the Rockefeller Foundation) and his fellow researchers are often mentioned as the first attempt to use computers for translation, and can be also thought of as a pioneering idea for using statistical models for the task. According to [17, p.22], Warren Weaver proposed in 1947: