{'fraction_non_alphanumeric': 0.016853932584269662, 'fraction_numerical': 0.0, 'mean_word_length': 4.746543778801843, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
AUTOMATIC ROMANIZATION FOR THAI

There is a common need in romanizing words in the languages other than English for the global communication. Especially the romanization of proper names are inevitable. Since there is no a mutual standard, writing a Thai word in English letters is not trivial, and it is quite a labor intensive task if it cannot be computerized. In this paper, we propose a new romanization system aiming at initiating the standardization process and implementing in a computer assisting module. The romanization is not a simple one-to-one matching. We need some linguistic rules to restrain the possible combination in terms of pronunciation availability and syllable construction. Before romanizing a Thai syllable, we need to break a word into a sequence of syllables. From the sequence of syllables, we then generate a roman script for it. In this paper, we prepare a syllable construction rule to drive the NFA to produce all possible sequences. The probabilistic n-gram is introduced to find the most probable one. Since there are a lot of ambiguities in breaking a word into a sequence of syllables, we rank the candidates according to their probabilities and apply a general beam search method to reduce the search space.