{'fraction_non_alphanumeric': 0.02432389182269163, 'fraction_numerical': 0.011841894703152504, 'mean_word_length': 4.840654205607477, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Rule-based Syllable Segmentation of Myanmar Text

Myanmar script uses no space between words and syllable segmentation represents a significant process in many NLP tasks such as word segmentation, sorting, line breaking and so on. In this study, a rulebased approach of syllable segmentation algorithm for Myanmar text is proposed. Segmentation rules were created based on the syllable structure of Myanmar script and a syllable segmentation algorithm was designed based on the created rules. A segmentation program was developed to evaluate the algorithm. A training corpus containing 32,283 Myanmar syllables was tested in the program and the experimental results show an accuracy rate of 99.96% for segmentation.


Introduction
Myanmar language, also known as Burmese, is the official language of the Union of Myanmar. It is spoken by 32 million as a first language, and as a second language by ethnic minorities in Myanmar (Ethnologue, 2005). Burmese is a member of the Tibeto-Burman languages, which is a subfamily of the Sino-Tibetan family of languages. Burmese is a tonal and analytic language using the Burmese script. This is a phonologically based script, adapted from Mon, and ultimately based on an Indian (Brahmi) prototype (Daniels and Bright, 1996). Burmese characters are rounded in shape and the script is written from left to right. No space is used between words but spaces are usually used to separate phrases.
The Myanmar language still remains as one of the less privileged Asian languages in cyberspace. Many people have put considerable effort into the computerization of the Myanmar script. However, Myanmar still lacks support on computers and not many NLP tools and applications are available for this language. A standard encoding is needed for the language processing of Myanmar script; however, there is not yet any official national standard encoding for Myanmar script.
This study focuses on the syllable segmentation of Myanmar text based on the UTN11-2 1 encoding model for Myanmar script. Myanmar script has been granted space in Unicode (U+1000-U+109F) since version 3.0. In Unicode version 4.0, the Unicode consortium defined standards for encoding Myanmar script and canonical order. The current version of Unicode is 5.0. However, there are only a few Unicode-compliant Myanmar fonts that fully follow the Unicode encoding standard. Local font developers and implementers have produced fonts that follow only part of the Unicode standards and many of these partially-compliant fonts are widely used in cyberspace. In 2006, Myanmar proposed additional characters 2 to be added to the Unicode version 5.0. The proposed characters for the Burmese script are as follows: The Unicode technical committee has accepted these proposed characters for inclusion in future versions of the Unicode standard. 3 If the proposal is adopted, this will become the standard encoding for Myanmar script. Therefore, this paper employs the proposed encoding model for the syllable segmentation of Myanmar text.

Related Work
The lack of official standard encoding hinders localization of Myanmar language and no previous work on the syllable segmentation of Myanmar script was found. Although character codes for Myanmar languages have been allocated in UCS/Unicode (U+1000-U+109F), lack of implementation makes them unavailable to local end users (Ko Ko and Mikami, 2005). We can learn, however, from related works done for other languages which have similarities to Myanmar. Many attempts have been made in Thai language processing for syllable and word segmentation. Poowarawan (1986) proposed a dictionary-based approach to Thai syllable separation. Thai syllable segmentation was considered as the first step towards word segmentation and many of word segmentation ambiguities were resolved at the level of syllable segmentation (Aroonmanakun, 2002). Thai syllable segmentation can be viewed as the problem of inserting spaces between pairs of characters in the text and the character-level ambiguity of word segmentation can be reduced by extracting syllables whose structures are more well-defined (Sornil and Chaiwanarom, 2004). Most approaches to Thai word segmentation use a dictionary as their basis. However, the segmentation accuracy depends on the quality of the dictionary used for analysis and unknown words can reduce the performance. Theeramunkong and Usanavasin (2001) proposed a non dictionary-based approach to Thai word segmentation. A method based on decision tree models was proposed and their approach claimed to outperform some well-known dictionary-dependent techniques of word segmentation such as the maximum and the longest matching methods.

Myanmar Alphabets
In order to clarify the syllable structure, characters of the Myanmar script are classified into twelve categories. Each category is given a name and the glyphs and Unicode code points of characters belonging to each category are shown in Table 1. The Myanmar script consists of a total of 75 characters. There are 34 consonant letters in Consonants group, four medials in the Medials group and eight vowels in the Dependent Vowels group. Myanmar Sign Virama is used for stacking consonant letters and it does not have a glyph, while Myanmar Sign Asat is used in devowelising process (e.g. ဆင် ). There are three dependent various signs in Group F. The Group I consists of three independent vowels (ဤ, ဧ, ဪ) and three independent various signs (၌, ၍, ၏). The characters in Group I can act as stand-alone syllables. Group E consists of four independent vowels (ဣ, ဥ, ဦ, ဩ) and Myanmar Symbol Aforementioned (၎). Each of the independent vowels in group E has its own syllable but they can also combine with other signs to form a syllable (e.g. ဥက¾ ာ). Myanmar Symbol Aforementioned in Group E can never stand alone and it is always written as ၄င် း as a short form of လည် းေကာင် း. Myanmar Letter Great Sa is always preceded by a consonant and is never written alone (e.g. မနု ဿ). There are ten Myanmar digits in the Digits group. The group P consists of two Myanmar punctuation marks. Myanmar script uses white space between phrases, which is taken into account in this study. Non-Myanmar characters are not included in this study.

Category
Name

Syllable Segmentation Rules
Typically, a syllable boundary can be determined by comparing pairs of characters to find whether a break is possible or not between them. However, in some cases it is not sufficient to determine a syllable boundary by just comparing two characters.
The following sections explain these cases and give examples.

Devowelising
In one syllable, a consonant may appear twice but the second consonant is used for the devowelising process in conjunction with an Asat (U+103A MYANMAR SIGN ASAT). Therefore the character after the second consonant should be further checked for an Asat. If the character after the second consonant is an Asat, there should be no syllable break before the second consonant.

Syllable Chaining
Subjoined characters are shown by using an invisible Virama sign (U+1039 MYANMAR SIGN VI-RAMA) to indicate that the following character is subjoined and should take a subjoined form. In this case, if the character after the second consonant is an invisible Virama sign, there should be no syllable break before the second and third consonant. Although there are two syllables in a subjoined form, it is not possible to separate them in written form and they are therefore treated as one syllable.

Kinzi
Kinzi is a special form of devowelised Nga (U+1004 MYANMAR LETTER NGA) with the following letter underneath, i.e., subjoined. In this case, if the character after the second consonant is an Asat and the next character after Asat is an invisible Virama sign (U+1039 MYANMAR SIGN VIRAMA) then there should be no syllable break before the second and third consonant. Kinzi also consists of two syllables but it is treated as one syllable in written form.

Loan Words
Usage of loan words can be found in Myanmar text. Although loan words do not follow the Myanmar syllable structure, their usage is common and the segmentation rules for these words are considered in this study.

Great Sa
There should be no syllable break before great Sa (U+103F MYANMAR LETTER GREAT SA) as great Sa acts like a stacked သÛ and devowelises the preceding consonant.

Contractions
There are usages of double-acting consonants in Myanmar text. The double-acting consonant acts as both the final consonant of one syllable and the initial consonant of the following syllable. There are two syllables in a contracted form but they cannot be segmented in written form and there should be no syllable break between them.

Implementation
Syllable segmentation rules are presented in the form of letter sequence tables (Tables 4-6). The tables were created by comparing each pair of character categories. However, it is not sufficient to determine all syllable breaks by comparing only two characters. In some cases, a maximum of four consecutive characters need to be considered to determine a possible syllable boundary. Two additional letter sequence tables were created for this purpose (Tables 5 and 6). Table 4 defines the break status for each pair of two consecutive characters. Table 5 and 6 define the break status for each pair of three and four consecutive characters, respectively. The symbol U in the Table 4 and 5 stands for undefined cases. Cases undefined in Table 4 are defined in the Table 5, and those undefined in Table 5 are then defined in Table 6.
The syllable segmentation program obtains the break status for each pair of characters by comparing the input character sequence with the letter sequence tables. The syllable break status and definitions are shown in Table 3. The break status -1 indicates a breach of canonical spelling order and a question mark is appended after the ambiguous character pair. The status 0 means there should be no syllable break after the first character. For break cases, a syllable breaking symbol (i.e. B in the flowchart) is inserted at each syllable boundary of the input string. The syllable segmentation process is shown in the flowchart in Break after 1 st character 2 Break after 2 nd character 3 Break after 3 rd character 4 Break after 4 th character

Method and Results
A syllable segmentation program was developed to evaluate the algorithm and segmentation rules. The program accepts the Myanmar text string and shows the output string in a segmented form. The program converts the input text string into equivalent sequence of category form (e.g. CMCACV for ြမန် မာ) and compares the converted character sequence with the letter sequence tables to determine syllable boundaries. A syllable segmented Myanmar text string is shown as the output of the program. The symbol "|" is used to represent the syllable breaking point. In order to evaluate the accuracy of the algorithm, a training corpus was developed by extracting 11,732 headwords from Myanmar Orthography (Myanmar Language Commission, 2003). The corpus contains a total of 32,238 Myanmar syllables. These syllables were tested in the program and the segmented results were manually checked. The results showed 12 errors of incorrectly segmented syllables, thus achieving accuracy of 99.96% for segmentation. The few errors occur with the Myanmar Letter Great Sa 'ဿ' and the Independent Vowel 'ဥ'. The errors can be fixed by updating the segmentation rules of these two characters in letter sequence tables. Some examples of input text strings and their segmented results are shown in Table 7.

Conclusion
Syllables are building blocks of words and syllable segmentation is essential for the language processing of Myanmar script. In this study, a rule-based approach of syllable segmentation algorithm for Myanmar script is presented. The segmentation rules were created based on the characteristics of Myanmar syllable structure. A segmentation program was developed to evaluate the algorithm. A test corpus containing 32,238 Myanmar syllables was tested in the program and 99.96% accuracy was achieved. From this study, we can conclude that syllable segmentation of Myanmar text can be implemented by a rule-based approach. While characters of non-Myanmar script are not considered in this study, the segmentation rules can be further extended to cover these characters. A complete syllable segmentation algorithm for Myanmar script can be further implemented by applying this algorithm.