{'fraction_non_alphanumeric': 0.03615184528970483, 'fraction_numerical': 0.011422324424171598, 'mean_word_length': 4.748212351029252, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Simple Approach to Unknown Word Processing in Japanese Morphological Analysis

This paper presents a simple but effective approach to unknown word processing in Japanese morphological analysis, which handles 1) unknown words that are derived from words in a pre-defined lexicon and 2) unknown onomatopoeias. Our approach leverages derivation rules and onomatopoeia patterns, and correctly recognizes certain types of unknown words. Experiments revealed that our approach recognized about 4,500 unknown words in 100,000 Web sentences with only 80 harmful side effects and a 6% loss in speed.


Introduction
Morphological analysis is the first step in many natural language applications. Since words are not segmented by explicit delimiters in Japanese, Japanese morphological analysis consists of two subtasks: word segmentation and part-of-speech (POS) tagging. Japanese morphological analysis has successfully adopted lexicon-based approaches for newspaper articles (Kurohashi et al., 1994;Asahara and Matsumoto, 2000;Kudo et al., 2004), in which an input sentence is transformed into a lattice of candidate words using a pre-defined lexicon, and an optimal path in the lattice is then selected. Figure 1 shows an example of a word lattice for morphological analysis and an optimal path. Since the transformation from a sentence into a word lattice basically depends on the pre-defined lexicon, the existence of unknown words, i.e., words that are not included in the predefined lexicon, is a major problem in Japanese morphological analysis.
Most previous work on this approach has aimed at developing a single general-purpose unknown word model. However, there are several types of unknown words, some of which can be easily dealt with by introducing simple derivation rules and unknown word patterns. In addition, as we will discuss in Section 2.3, the importance of unknown word processing varies across unknown word types. In this paper, we aim to deal with unknown words that are considered important and can be dealt with using simple rules and patterns. Table 1 lists several types of Japanese unknown words, some of which often appear in Web text. First, we broadly divide the unknown words into two classes: words derived from the words in the lexicon and the others. There are a lot of informal spelling variations in Web text that are derived from the words in the lexicon, such as " " (y0u) instead of " " (you) and " " (coooool) instead of " " (cool). The types of derivation are limited, and thus most of them can be resolved by introducing derivation rules. Unknown words other than those derived from known words are generally difficult to resolve using only simple rules, and the lexicon augmentation approach would be better for them. However, this is not true for onomatopoeias. Although Japanese is rich in onomatopoeias and some of them do not  appear in the lexicon, most of them follow several patterns such as 'ABAB,' 'A B ,' and 'AB ,' 1 and they thus can be resolved by considering typical patterns.
Therefore, in this paper, we introduce derivation rules and onomatopoeia patterns to the unknown word processing in Japanese morphological analysis, and aim to resolve 1) unknown words derived from words in a pre-defined lexicon and 2) unknown onomatopoeias.

Japanese morphological analysis
As mentioned earlier, lexicon-based approaches have been widely adopted for Japanese morphological analysis. In these approaches, we assume that a lexicon, which lists a pair consisting of a word and its corresponding part-of-speech, is available. The process of traditional Japanese morphological analysis is as follows: 1. Build a lattice of words that represents all the candidate sequences of words from an input sentence. 2. Find an optimal path through the lattice.
Figure 1 in Section 1 shows an example of a word lattice for the input sentence " " (My father is Japanese), where a total of six candidate paths are encoded and the optimal path is marked with bold lines. The lattice is mainly built with the words in the lexicon. Some heuristics are also used for dealing with unknown words, but in most cases, only a few simple heuristics are used. In fact, the three major Japanese morphological analyzers, JUMAN (Kurohashi and Kawahara, 2005), ChaSen (Matsumoto et al., 2007), 1 'A' and 'B' denote Japanese characters, respectively. and MeCab (Kudo, 2006), use only a few simple heuristics based on the character types, such as hiragana, katakana, and alphabets 2 , that regard a character sequence consisting of the same character type as a word candidate.
The optimal path is searched for based on the sum of the costs for the path. There are two types of costs: the cost for a candidate word and the cost for a pair of adjacent parts-of-speech. The cost for a word reflects the probability of the occurrence of the word, and the connectivity cost of a pair of parts-of-speech reflects the probability of an adjacent occurrence of the pair. A greater cost means less probability. The costs are manually assigned in JUMAN, and assigned by adopting supervised machine learning techniques in ChaSen and MeCab, while the algorithm to find the optimal path is the same, which is based on the Viterbi algorithm.

Types of unknown words
In this section, we detail the target unknown word types of this research.
Rendaku (sequential voicing) is a phenomenon in Japanese morpho-phonology that voices the initial consonant of the non-initial portion of a compound word. In the following example, the initial consonant of the Japanese noun " " (sake, alcoholic drink) is voiced into " " (zake): (1) (eggnog) ta ma go -za ke. Since the expression " " (zake) is not included in a standard lexicon, it is regarded as an unknown word even if the original word " " (sake) is included in the lexicon. There are a lot of studies on rendaku in the field of phonetics and linguistics, and several conditions that prevent rendaku are known, such as Lyman's Law (Lyman, 1894), which stated that rendaku does not occur when the second element of the compound contains a voiced obstruent. However, few studies dealt with rendaku in morphological analysis. Since we have to check the adjacent word to recognize rendaku, it is difficult to deal with rendaku using only the lexicon augmentation approach. Some characters are substituted by peculiar characters or symbols such as long sound symbols, lowercase kana characters 3 , in informal text. First, if there is little difference in pronunciation, Japanese vowel characters ' '(a), ' '(i), ' '(u), ' '(e), and ' '(o) are sometimes substituted by long sound symbols ' ' or ' .' For example, a vowel character ' ' in the Japanese adjective " " (hontou, true) is sometimes substituted by ' ' and this adjective is written as " " (hontô, troo). We call this phenomenon substitution with long sound symbols. As well as long sound symbol substitution, some hiragana characters such as ' '(a), ' '(i), ' '(u), ' '(e), ' '(o), ' '(wa), and ' '(ka) are substituted by their lowercases: ' ,' ' ,' ' ,' ' ,' ' ,' ' ,' and ' .' We call this phenomenon substitution with lowercases.
There are also other types of derivation, that is, some characters are inserted into a word that is included in the lexicon. In the following examples, long sound symbols and lowercase are inserted into the Japanese adjective " " (cool).
(2) (Insertion of (coooool) long sound symbols) (3) (Insertion of lowercases) (coooool) In addition to the unknown words derived from words in the lexicon, there are several types of unknown words that contain rare words such as " " (decontamination), new words such as " " (Twitter), and onomatopoeias such as " " (caw-caw). We can easily generate Japanese onomatopoeias that are not included in the lexicon. Most of them follow several patterns, such as 'ABAB,' 'A B ,' and 'AB ,' and we classified them into two types, onomatopoeias with repetition such as 'ABAB,' and onomatopoeias without repetition such as 'A B .'

Importance of unknown word processing of each type
The importance of unknown word processing varies across unknown word types. We give three example sentences (4), (5), and (6), which include the unknown words " " (fluffy), " " (decontamination), and " " (Twitter), respectively. In these examples, (a) denotes the desirable morphological analysis and (b) is the output of our baseline morphological analyzer, JUMAN version 5.1 (Kurohashi and Kawahara, 2005).  In the case of (4), the unknown word " " (fluffy) is divided into three parts by JU-MAN, and influences the analyses of the adjacent function words, that is, " " (and) is changed to " " (but) and " " (of) is changed to " " (this), which will strongly affect the other NLP applications. The wide scope of influence is due to the fact that " " consists of hiragana characters like most Japanese function words. On the other hand, in the case of (5), although the unknown word " " (decontamination) is divided into two parts by JUMAN, there is no influence on the adjacent analyses. Moreover, in case of (6), although there is no lexical entry of " " (Twitter), the segmentation is correct thanks to simple character-based heuristics for out-of-vocabulary (OOV) words.
These two unknown words do not contain hiragana characters, and thus, we think it is important to resolve unknown words that contain hiragana. Since unknown words derived from words in the lexicon and onomatopoeias often contain hi-ragana characters, we came to the conclusion that it is more important to resolve them than to resolve rare words and new words that often consist of katakana and Chinese characters.

Related work
Much work has been done on Japanese unknown word processing. Several approaches aimed to acquire unknown words from a corpus in advance (Mori and Nagao, 1996;Murawaki and Kurohashi, 2008) and others aimed to introduce better unknown word model to morphological analyzer (Nagata, 1999;Uchimoto et al., 2001;Asahara and Matsumoto, 2004;Nakagawa and Uchimoto, 2007). However, there are few works that focus on certain types of unknown words. Kazama et al. (1999)'s work is one of them. Kazama et al. improved the morphological analyzer JUMAN to deal with the informal expressions in online chat conversations. They focused on substitution and insertion, which are also the target of this paper. However, while our approach aims to develop heuristics to flexibly search the lexicon, they expanded the lexicon, and thus their approach cannot deal with an infinite number of derivations, such as " ," and " " for the original word " ." In addition, Ikeda et al. (2009) conducted experiments using Kazama et al.'s approach on 2,000,000 blogs, and reported that their approach made 37.2% of the sentences affected by their method worse. Therefore, we conjecture that their approach only benefits a text that is very similar to the text in online chat conversations. Kacmarcik et al. (2000) exploited the normalization rules in advance of morphological analysis, and Ikeda et al. (2009) replaced peculiar expressions with formal expressions after morphological analysis. In this research, we exploit the derivation rules and onomatopoeia patterns in morphological analysis. Owing to such a design, our system can successfully deal with rendaku, which has not been dealt with in the previous works.
UniDic dictionary (Den et al., 2008) handles orthographic and phonological variations including rendaku and informal ones. However, the number of possible variations is not restricted to a fixed number because we can insert any number of long sound symbols or lowercases into a word, and thus, all the variations cannot be covered by a dictionary. In addition, as mentioned above, since we  Figure 2: Example of a word lattice with new nodes " ," " ," and " ." The broken lines indicate the added nodes and paths, and the bold lines indicate the optimal path. have to take into account the adjacent word to accurately recognize rendaku, the lexical knowledge alone is not sufficient for rendaku recognition.
For languages other than Japanese, there is much work on text normalization that aims to handle informal expressions in social media (Beaufort et al., 2010;Liu et al., 2012;Han et al., 2012). However, their target languages are segmented languages such as English and French, and thus they can focus only on normalization. On the other hand, since Japanese is an unsegmented language, we have to also consider the word segmentation task.

Overview
We use the rule-based Japanese morphological analyzer JUMAN version 5.1 as our baseline system. Basically we only improve the method for building a word lattice and do not change the process for finding an optimal path from the lattice. That is, our proposed system only adds new nodes to the word lattice built by the baseline system by exploiting the derivation rules and onomatopoeia patterns. If the new nodes and their costs are plausible, the conventional process for finding the optimal path will select the path with added nodes.
For example, if a sentence " ." is input into the baseline system, it builds the word lattice that is described with solid lines in Figure 2. However, this lattice does not include such expressions as " " and " " since they are not included in the lexicon. Our proposed system transforms the informal expressions into their standard expressions such as " " (delicious) and " " (was) by exploiting the derivation rules, adds their nodes into the word lattice, and selects the path with these added nodes.

Resolution of unknown words derived from words in the lexicon
We deal with five types of unknown words that are derived from words in the lexicon: rendaku, substitution with long sound symbols, substitution with lowercases, insertion of long sound symbols, and insertion of lowercases. Here, we describe how to add new nodes into the word lattice.
Rendaku The procedure to add unvoiced nodes to deal with rendaku differs from the others. Since only the initial consonant of a word is voiced by rendaku, there is at most one possible voiced entry for each word in the lexicon. Hence, we add the voiced entries into the trie-based lexicon in advance if the original word does not satisfy any conditions that prevent rendaku such as Lyman's Law. For example, our system creates the entry " " (zake) from the original word " " (sake), and adds it into the lexicon. When the system retrieves words that start from the fourth character in the example (1) in Section 2.2, " ," the added entry " " (zake) is retrieved. Since rendaku occurs for the initial consonant of the noninitial portion of a compound word, our system adds the retrieved word only when it is the noninitial portion of a compound word. Substitution with long sound symbols and lowercases In order to cope with substitution with long sound symbols and lowercases, our system transforms the input text into normalized strings by using simple rules. These rules substitute a long sound symbol with one of the vowel characters: ' ,' ' ,' ' ,' ' ,' and ' ,' that minimizes the difference in pronunciation. These rules also substitute lowercase characters with the corresponding uppercase characters. For example, if the sentence " ." (It is trooly DElicious.) is input, the nodes generated from the normalized string " ." are added to the word lattice along with the nodes generated from the original string.

Insertion of long sound symbols and lowercases
In order to cope with the insertion of long sound symbols and lowercases, our system transforms the input text into a normalized string using simple rules. These rules delete long sound symbols and lowercase characters that are considered to be inserted to prolong the original word pronunciation. For example, if the sentence " ." (It iiisss coooool.) is input, the nodes generated from the normalized string "   ." are added into the word lattice. We do not consider partly deleted strings such as " ." and the combination of substitution and insertion to avoid combinatorial explosion. Therefore, our system cannot deal with unknown words generated by both insertion and substitution, but such words are rare in practice.
Costs for additional nodes Our system imposes small additional costs to the node generated from the normalized string to give priority to the nodes generated from the original string. We set these costs by using a small development data set.

Resolution of unknown onomatopoeias
There are many onomatopoeias in Japanese. In particular, there are a lot of unfamiliar onomatopoeias in Web text. Most onomatopoeias follow limited patterns, and we thus can easily produce new onomatopoeias that follow these patterns. Hence, it seems more reasonable to recognize unknown onomatopoeias by exploiting the onomatopoeia patterns than by manually adding lexical entries for them. Therefore, our system lists onomatopoeia candidates by using onomatopoeia patterns, as shown in Tables 2 and 3, and adds them into the word lattice. Figure 3 shows examples. The number of potential entries of onomatopoeias with repetition is large, but the candidates of onomatopoeias with repetition can be quickly searched for by using a simple string matching strategy. On the other hand, to search the candidates of onomatopoeias without repetition is a bit time consuming com- pared with trie search. However, the number of potential entries of onomatopoeias without repetition is not so large, and thus our system adds all possible entries of onomatopoeias without repetition into the trie-based lexicon in advance.

Setting
We used 100,000 Japanese sentences to evaluate our approach. These sentences were obtained from an open search engine infrastructure TSUB-AKI (Shinzato et al., 2008), which included at least one hiragana character and consisted of more than twenty characters We first estimated the recall. Since it is too costly to create a set of data with all unknown words annotated, we made a set of data with only our target unknown words annotated. We could apply a set of regular expressions to reduce the unknown word candidates by limiting the type of unknown words. We manually annotated 100 expressions for each type, and estimated the recall.
A high recall, however, does not always imply that the proposed system performs well. It might be possible that our proposed method gives bad effects on non-target words. Therefore, we also compared the whole analysis with and without the rules/patterns from the following seven aspects: 4 4 There are two major reasons why we did not use the precision, recall and F-measure metrics to evaluate the overall performance. The first reason is that to create a large set of annotated data is too costly. The second reason, which is more essential, is that there is no clear definition of Japanese 1. The number of positive changes for 100 different outputs: P 100D . 2. The number of negative changes for 100 different outputs: N 100D . 3. The number of different outputs for 100,000 sentences: D 100kS . 4. The estimated number of positive changes for 100,000 sentences: P * 100kS . 5. The estimated number of negative changes for 100,000 sentences: N * 100kS . 6. The relative increase of the nodes: Node inc. . 7. The relative loss in speed: SP loss .
Different outputs indicate cases in which the systems with and without rules/patterns output a different result. First, for each type of rule/pattern, we extracted 100 different outputs and manually classified them into three categories: the system with the rules/patterns was better (positive), the system without the rules/patterns was better (negative), and both outputs were undesirable (others). When these outputs differed in word segmentation, we only compared the segmentation but did not take into account the POS tags. On the other side, when these outputs did not differ in word segmentation, we compared the POS tags. Tables 6-10 list several examples. For example, " " (can feel amused) in Table 6 should be analyzed as one word, but both systems with and without rules for rendaku divided it into several parts, and such a case is labeled as others.
We counted the number of different outputs for 100,000 sentences. We then calculated the estimated numbers of positive/negative changes for the sentences by using the equations: . We also counted the number of created nodes in lattice and calculated the relative increase, which would affect the time for finding the optimal path from the word lattice, and measured the analysis time and calculated the relative loss in speed. Table 4 lists the recall of our system for each unknown word type with the number of words that are covered by the UniDic dictionary. Note that while our system's recall denotes the ratio of actually recognized words, the coverage of UniDic word segmentation, especially for unknown words. That is, we can accept various word boundaries. We thought it is more straight-forward and efficient to compare the differences between a baseline system and the proposed system.  only denotes the number of words included in the dictionary, which can be interpreted as the upper bound of the system based on UniDic. We can confirm our system achieved high recall for each type of unknown word. Since UniDic covered 95% of unknown words of rendaku type, we would be able to improve the rendaku recognition by incorporating UniDic and our approach that takes into account the adjacent word. Except for rendaku, our system's recall was higher than the coverage of UniDic, which confirms the effectiveness of our method. Table 5 summarizes the comparison between the analyses with and without the rules/patterns. In short, our method successfully recognized all types of unknown words with few bad effects. By introducing all the derivation rules and onomatopoeia patterns, there are 4,560 improvements for 100,000 sentences with only 80 deteriorations and a 6.2% loss in speed. In particular, the derivation rules of insertion and substitution of long sound symbols and lowercases produced 3,327 improvements for 100,000 sentences at high recall values (see Table 4) with only 27 deteriorations and a 3.8% loss in speed. We confirmed from these results that our approaches are very effective for unknown words in informal text. Since the number of newly added nodes was small, the speed loss is considered to be derived not from the optimal path searching phase but from the lattice building phase. Table 6 lists some examples of the changed outputs by introducing the derivation rules for rendaku. As listed in Table 4 and 5, the rendaku processing produced more negative changes and the lower recall value compared with the other types. This indicates that rendaku processing is more difficult than resolving informal expressions with long sound symbols or lowercases. Since long sound symbols and lowercases rarely appear in the lexicon, there are few likely candidates other than the correct analysis. On the other hand, voiced characters often appear in the lexicon and formal Table 7: Examples of different outputs by introducing derivation rules for long sound symbol substitution and insertion.

Results and Discussion
text, and thus, there are many likely candidates. Table 7 lists some examples of the changed output by introducing the derivation rules for informal spelling with long sound symbols. We labeled the change of the analysis "OK " (It's OK) as negative because the baseline system correctly tagged the POS of " " unlike our proposed system, but the baseline system could not also correctly resolve the entire phrase. There was no different output that our proposed system could not resolve but the baseline system could fully resolve. Table 8 lists some examples of the changed outputs by introducing the derivation rules for informal spelling with lowercase. We labeled the change of the analysis " " (Yumi's bedclothes) as negative because the baseline system correctly segmented the postpositional particle " " unlike our proposed system. Again for this example, the baseline system could not correctly resolve the entire phrase. Along with the informal spelling with long sound symbols, there was no different output that our proposed system could not resolve but the baseline system could fully resolve. Rules/patterns Table 10: Examples of different outputs by introducing onomatopoeia patterns without repetition. Table 9 lists some examples of the changed outputs by introducing onomatopoeia patterns with repetition. Our system recognized unknown onomatopoeias with repetition at a recall of 89%, which is not very high. However, since there were several repetition expressions other than onomatopoeias, such as " / " (wow wow) as shown in Table 9, we cannot lessen the cost for onomatopoeias with repetition. Table 10 lists some examples of the changed outputs by introducing onomatopoeia patterns without repetition. Our system recognized the unknown onomatopoeias without repetition at a recall of 94% and did not output anything worse than  Table 11: Classification results of unknown words that occur more than two times in KNB corpus.
the baseline output with no loss in speed.
In order to approximate the practical coverage of our method, we classified unknown words that occur more than two times in the Kyoto University and NTT Blog (KNB) corpus 5 into four types: words that are covered by the lexicon created by Murawaki and Kurohashi (2008) (Murawaki's Lexicon), words that are not covered by Murawaki's Lexicon but have entries in Wikipedia, words that are covered only by our method, and the others. Table 11 shows the results. There are total 645 tokens of unknown words that occur more that two times in KNB corpus, 105 of which are newly covered by our method. Since the number of tokens that are covered by neither Murawaki's Lexicon nor Wikipedia is only 187, we can say that the coverage of our method is not trivial.

Conclusion
We presented a simple approach to unknown word processing in Japanese morphological analysis. Our approach introduced derivation rules and onomatopoeia patterns, and correctly recognized certain types of unknown words. Our experimental results on Web text revealed that our approach could recognize about 4,500 unknown words for 100,000 Web sentences with only 80 harmful side effects and a 6% loss in speed. We plan to apply our approach to machine learning-based morphological analyzers, such as MeCab, with Uni-Dic dictionary, which handles orthographic and phonological variations, in future work.