{'fraction_non_alphanumeric': 0.024051803885291396, 'fraction_numerical': 0.0, 'mean_word_length': 5.518072289156627, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Chinese-English transliteration using weighted finite-state transducers

This paper proposes a novel method for Chinese-English transliteration based on multiple models by using weighted finite-state transducers (WFST). WFST provide a unified framework for integrating the various components of a speech-to-speech translation system, such as speech recognition and machine translation, motivated by their flexibility in integrating multiple sources of information and other interesting properties. We built a grapheme-based model, a phoneme-based model, an extended phoneme-based model and so on. Combining those models with unified framework of WFST, we can build a combining transliteration model for Chinese-English. The advantage of this method lies in that we can better account for such behavior by combining those information sources from different model to maximize the use of the data available. Our experiments show that the resulting system outperforms single-model systems with the model directly trained with Chinese-English name pairs for Chinese-English translation.