{'fraction_non_alphanumeric': 0.017928286852589643, 'fraction_numerical': 0.0, 'mean_word_length': 5.79054054054054, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Developing and improving a statistical machine translation system for English to Setswana : a linguistically-motivated approach

This paper describes the methods that were followed in the development and improvement of a statistical machine translation system for translation from English to Setswana. Setswana is regarded as a resource scarce language and therefore an adequate amount of parallel data is not freely available. The methods created attempt to improve the quality of a machine translation by manipulating the data during processing. The methods include the creation of sentence reordering, term deletion and term replacement rules. The rules were applied to training and testing data in the preand postprocessing stages of development. The systems were compared to one another to detect whether the quality of the machine translation improved. Keywords—statistical machine translation, pre-processing, postprocessing, sentence reordering, English, Setswana, term replacement, term deletion