{'fraction_non_alphanumeric': 0.011873350923482849, 'fraction_numerical': 0.0, 'mean_word_length': 6.228571428571429, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Linguistic Categorisation in Machine Translation using Stochastic Finite State Transducers

In the last years, statistical machine translation has already demonstrated its usefulness within a wide variety of translation applications. In particular, finite state models are always an interesting framework because there are well-known efficient algorithms for their representation and manipulation. Nevertheless, statistical approaches have rarely been performed taking into account the linguistic nature of the translation problem. This document describes some methodological aspects of building category-based finite state transducers that are able to consider a set of linguistic features in order to produce the most linguistically appropriate hypotheses.