{'fraction_non_alphanumeric': 0.02882882882882883, 'fraction_numerical': 0.0018018018018018018, 'mean_word_length': 5.541176470588235, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Tuning statistical machine translation parameters using perplexity

Statistical machine translation (SMT) involves many tasks including modeling, training, decoding, and evaluation. In this work, we present a methodology for optimizing the training process to get better translation quality using the well known GIZA ++ SMT toolkit. The methodology is based on adjusting the parameters of GIZA ++ that affect the generation of the translation model. When applying the methodology, an average improvement of 7% has been achieved in the translation quality.