{'fraction_non_alphanumeric': 0.044108715457030066, 'fraction_numerical': 0.00774369875493471, 'mean_word_length': 4.598385040373991, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Study on Implementation of Southern-Min Taiwanese Tone Sandhi System

In the past two hundred years or so, a sizable corpus of Taiwanese text in Latin script has been accumulated. However, due to the political and historical situation of Taiwan, few people can read these materials at present. It is regrettable that the utilization of these plentiful materials is very low. This paper addresses problems raised by the Taiwanese tone sandhi system by describing a set of computational rules to approximate this system, as well as the results obtained from our implementation. Using the Taiwanese Latinization text as source, we take the sentence as the unit, translate every word into Chinese via a Taiwanese-Chinese dictionary, and obtain the POS information from the CKIP dictionary made by the CKIP group of the Academia Sinica. Using the POS data and tone sandhi rules we formulated based on linguistics, we then tag each syllable with its post-sandhi tone marker. Finally we implemented a Taiwanese tone sandhi processing system which takes a Latinized sentence as input and outputs the tone markers. We were able to obtain an accuracy rate of 97.56% and 88.90% with training and testing data, respectively. We analyze the sources of error for the purpose of future improvement.


1.1.Background
Taiwanese is often used in daily life in Taiwan, but written Taiwanese is less common by far; even so, the history of written Taiwanese stands at well over a century. (Tiu n 2001) At present there are several dozen if not more than a hundred proposed phonetic and writing systems for Taiwanese. (Iu n &Tiu n 1999) The orthography adopted by this article is Peh-oe-ji (POJ, also known as Latinized Taiwanese or Church Latinized Taiwanese).
Under the auspices of the National Museum of Taiwanese Literature, the Department of Taiwanese Literature of Cheng Kung University carried out a project titled 'The Collection and Cataloging of Peh-oe-ji Literature Data'. Although a lot of texts have already been lost due to the changing political situation, this project nevertheless revealed nearly 2,000 Peh-oe-ji books and periodicals, with publication sites spread over Taiwan, Xiamen, Shanghai, Guangzhou, Hong Kong, Singapore, Philippine, London, Japan, etc. The amount of publishing peaked in the 1950's and 1960's. (Iu n &Tan-Te n 2005) The scope covers not only formally published books and periodicals but also non-published items such as personal letters and medical charts. Later on the government, citing supposed detrimental effects of Peh-oe-ji on Mandarin promotion as a reason, banned its use and thus contributed to the rapid decline of this practice. We hope that applied information science might enable more people to access the extant materials collected by the above-mentioned project, as well as contribute to basic and applied research in Taiwanese. As most people nowadays are not familiar with Latinized written Taiwanese, use of state-of-the-art text-to-speech technology would enhance the value of these materials to the general public.
Tone sandhi represents a challenging problem to be solved before we can successfully transform the written Taiwanese text to its natural speech-like tonal contour. This is because the written form of Latinized Taiwanese represents the tones as "base tones", or the tones of syllables when they are read in isolation. At the level of the word, all syllables but the last one are usually read differently (that is, they manifest tone sandhi). At the level of a whole sentence, under most situations only the last syllables next to the boundary of the phrases or structural markers are read as base tones, the others being read as sandhi tones. In fact, besides the "regular tone sandhi" mentioned above, there exist several other kinds of situations which will be discussed one by one later. This paper will first formulate the sandhi rules, which are the key to a correct pronunciation. The inputs of our experiment are mainly the data collected by the above-mentioned project; these data are processed by our sandhi system and the final outputs are these data with sandhi markers. Due to the lack of a set of well-marked data, we do not adopt the statistical model in this experiment but the rule-based model. Two of the authors who are long-time experienced Taiwanese users evaluated the outputs for their accuracy.

Tone Sandhi Problem
Tones in Taiwanese are traditionally analyzed as consisting of piâⁿ, siáng, khì, ji̍ p, each having im and iang but for siáng, so there are seven tones in all. Following the sequence of im-pian, siáng, imkhì, im-ji̍ p, iang-pian, iang-khì, iang-ji̍ p, they are numbered 1 (high flat), 2 (high to low), 3 (low), 4 (middle short), 5 (low rising), 7 (middle flat), and 8 (high short). The descriptions of tone pitch are within the parentheses. For tone diacritics please refer to the following examples. Tone sandhi is a very important characteristic of Taiwanese. At the level of the word, the last syllable is usually read as base tone and the others, as sandhi tones. For example in (1) the underlined syllables are read as base tones, while the others are read as sandhi tones, and dialect difference is represented in parenthesis : (1) tâi ("palteform") / Tâi-gí(gú) ("Taiwanese language") Tâi-gí(gú)-bûn ("written Taiwanese") Tâi-gí(gú) bûn-ha̍ k ("Taiwanese literature") Tâi-gí(gú) bûn-ha̍ k-sú ("history of Taiwanese literature") In fact, at the level of the syllable or the word, tone sandhi may manifest in at least the following several ways: (a) Normal sandhi: using reduplicated syllables as examples (the numbers within parentheses are reading tones).

Historial Review
Compared with the above two systems, our approach has the following major differences: a Taiwanese corpus balanced for both literary and non-literary sources (about 50% each) was prepared; Chinese-to-Taiwanese translation is not an issue; and sentences of any length can be processed. In addition, because the text is of Latinized script, we do not encounter the problems of word segmentation and phonetic marking. However, compared with text written in Han script, there is a more rigorous challenge to deal with homonymy, especially with monosyllabic words.

Data
The input data of our system are Peh-oe-ji. Following its orthography, syllables of a word are joined by hyphens, and words are separated with spaces.
In addition to data drawn from the same project, the testing data also include some other sources we collected. Four sources are selected, as well, including: The testing data also cover two eras but with a longer time span.

Part of Speech Tagging
Because there is no standard on part of speech (POS) for Taiwanese at this moment, we use that of Chinese instead. We obtain the corresponding Chinese translation for each Taiwanese word by looking up the Taiwanese-Chinese On-line Dictionary. (Iu n 2003) We then look up the POS of the Chinese in the 80,000-word CKIP database. Ambiguity problems encountered include: (a) homonymy, especially monosyllabic homonyms; (b) one-to-many mapping when mapping Taiwanese to Chinese; (c) multiple possible POS for each Chinese word.
On the problem of homonymy, we choose the word with the highest query frequency. After checking with the text we find that this strategy works under most situations. Due to one Taiwanese word mapping to multiple Chinese words, and one Chinese word possibly having multiple POSs, there may be multiple POSs for one Taiwanese word. We initially retain all candidate POSs in tagging and only attempt to narrow down the list upon applying the sandhi algorithm. Of the 46 POSs in the CKIP we adopt only the top level. The POS classes that we use are the following 12: A (adjective), C (conjunction), D (adverb), G (postposition), I (interjection), M (special mark), N (noun), P (preposition), R (pronoun), V (verb), S (time), and T (auxiliary). Moreover we adjust certain POSs known to affect tone sandhi. For example, Vh (state intransitive verb, etc.) is marked A, Nh (pronoun) marked R, Ng (postposition) marked G, and Nd (time) marked S.
As for indeterminate words, if they are of the form 'XX' or 'XXX' (duplicate or triplicate syllables), we mark them as A (adjective). Other words are marked as N (noun).

2.3.Tone Sandhi Marks
The marks representing tone sandhis are listed in Table 1. Words with normal sandhi are not marked usually.

Tone Sandhi Rules
Tone sandhi rules are the most important part of this research. The algorithm of sandhi marking is shown in Table 2.   (Syllable level) pre-á: Mark any syllables just before á as pre-á sandhi & 12 Double sandhi 12.1 (Syllable level) beh: Mark any beh as double sandhi $ unless it appears at the end, including those within a word, such as kiong-beh, tih-beh. [ ]) 20 (Word level) R at the end góa / lí / i / gún / góan / lán / lín / in: Mark the pronoun as following sandhi @ if it appears at the end and there is a verb before it We set up the algorithm using the following resources: (a) Tone sandhi rules proposed by linguists; (b) Rules induced from the training data; (c) Our understanding as native-speaking observers of sandhi phenomena; (d) The word segmentation results of the CKIP (noting its POS tagging output); (e) Taiwanese concordancer system (to check the sandhi phenomena of certain words).
It should be noted that some of the sandhi rules proposed by linguists deal with specific contexts and thus cannot be broadly applied; some others carry exceptions. There is therefore some difficulty in converting these rules into an algorithm. So, besides (a), we also formulated some rules from (b) and (c) by analyzing errors in the training data output. In principle sandhi rules are formulated to be applicable to "most situations" --i.e. an accuracy rate over 80% on corpus data. Once applied, the new rules may affect the original rules, so (d) and (e) are our important references in deciding whether or not to apply the new rules.
These sandhi rules work on 4 different levels: the syllable, the word, the part of speech, and the sentence pattern. Some examples: A. The syllable: E.g. koh ("and") and beh ("want") are read as double sandhi regardless of whether they are part of a word or not, such as koh-chài ("again") and kiông-beh ("almost").
B. The word: E.g. che ("this") and he ("that") are read as base tones wherever they appear.
Also, the presence of some words (such as ê ["of"]) would change the sandhi mark of a preceding word. C. The POS: E.g. the last syllable of a noun is read as base tone if the POS of its following word is N (noun), A (adjective), D (adverb), P (preposition), R (pronoun) or V (verb). Some types of POS, such as G (postposition), would affect the sandhi mark of the preceding word.
D. The sentence pattern: Certain words exhibit clause-marking function --for example, iah-sī ("or"); the portion before these words are regarded as clauses. Another example is that bēin the pattern of ē...bē ("whether...will...or not") would be read as neutral tone.
After POS tagging and applying the sandhi rules: The letters within the parentheses are the POSs. Incorrectly processed words are boxed.

Evaluation
Two authors of this paper, who are skilled native speakers familiar with written Taiwanese, evaluated the correctness of the output. Note that in certain contexts more than one sandhi results are acceptable, and depending on discourse considerations some speakers may opt for one sandhi result over others.

Preliminary Results
Preliminary results are listed in Table 3. There are 614 syllables of training data, 15 errors, giving an accuracy rate of 97.56%. There are 955 syllables of testing data with 106 mistakes, or an accuracy rate of 88.90%. Some of the errors in traing data output are attributable to the incompleteness of the sandhi rule set. We expect improved results of at least 2.5% once additional rules are appended.

Analysis of Mistakes and Relevant Issues
Some of the problems we encountered may be taken into account in the future.

POS
In our investigation we use the POS set for Chinese. Whether this approach is suitable for Taiwanese is a linguistic question requiring further investigation. Although a few studies of the POS of Taiwanese are available from as early as the 1930's, currently these data have yet to be made available in digital form, and will need to be reviewed by linguists to ensure that they are suitable for dealing with the sandhi problem. (Tseng 1997) proposes a standard for Taiwanese word segmentation. Unfortunately discussion is lagging. Should a working word segmentation standard emerge, we would also need a dictionary comforming to that standard.

Standardization of Written Taiwanese
Historically the use of Han script to represent Taiwanese has suffered from a high degree of idiosyncrasy in character choice. For documents written in Latin script, most of the differences attributed to dialects can be reconciled by referencing existing dictionaries. Orthographic inconsistency in the use of hyphen is more problematic, as it could affect the result of sandhi processing. Manual standardization of hyphen placement is hardly a solution.