{'fraction_non_alphanumeric': 0.024722662440570524, 'fraction_numerical': 0.008452192287374538, 'mean_word_length': 4.952987421383648, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Combining Hand-crafted Rules and Unsupervised Learning in Constraint-based Morphological Disambiguation

This paper presents a constraint-based morphological disambiguation approach that is applicable languages with complex morphology--specifically agglutinative languages with productive inflectional and derivational morphological phenomena. In certain respects, our approach has been motivated by Brill's recent work, but with the observation that his transformational approach is not directly applicable to languages like Turkish. Our system combines corpus independent hand-crafted constraint rules, constraint rules that are learned via unsupervised learning from a training corpus, and additional statistical information from the corpus to be morphologically disambiguated. The hand-crafted rules are linguistically motivated and tuned to improve precision without sacrificing recall. The unsupervised learning process produces two sets of rules: (i) choose rules which choose morphological parses of a lexical item satisfying constraint effectively discarding other parses, and (ii) delete rules, which delete parses satisfying a constraint. Our approach also uses a novel approach to unknown word processing by employing a secondary morphological processor which recovers any relevant inflectional and derivational information from a lexical item whose root is unknown. With this approach, well below 1 percent of the tokens remains as unknown in the texts we have experimented with. Our results indicate that by combining these hand-crafted,statistical and learned information sources, we can attain a recall of 96 to 97 percent with a corresponding precision of 93 to 94 percent, and ambiguity of 1.02 to 1.03 parses per token.


Introduction
This paper presents a constraint-based morphological disambiguation approach that is applicable languages with complex morphology-specifically agglutinative languages with productive inflectional and derivational morphological phenomena. In certain respects, our approach has been motivated by Brill's recent work (Brill, 1995b), but with the observation that his transformational approach is not directly applicable to languages like Turkish. Our system combines corpus independent handcrafted constraint rules, constraint rules that are learned via unsupervised learning from a training corpus, and additional statistical information from the corpus to be morphologically disambiguated. The hand-crafted rules are linguistically motivated and tuned to improve precision without sacrificing recall. The unsupervised learning process produces two sets of rules: (i) choose rules which choose morphological parses of a lexical item satisfying constraint effectively discarding other parses, and (ii) delete rules, which delete parses satisfying a constraint. Our approach also uses a novel approach to unknown word processing by employing a secondary morphological processor which recovers any relevant inflectional and derivational information from a lexieal item whose root is unknown. With this approach, well below 1% of the tokens remains as unknown in the texts we have experimented with. Our results indicate that by combining these hand-crafted, statistical and learned information sources, we can attain a recall of 96 to 97% with a corresponding precision of 93 to 94%, and ambiguity of 1.02 to 1.03 parses per token.
Automatic morphological disambiguation is a very crucial component in higher level analysis of natural language text corpora. Morphological disambiguation facilitates parsing, essentially by performing a certain amount of ambiguity resolution using relatively cheaper methods (e.g., Gfing6rdii and Oflazer (1995)). There has been a large number of studies in tagging and morphological disambiguation using various techniques. Part-of-speech tagging systems have used either a statistical approach where a large corpora has been used to train a probabilistic model which then has been used to tag new text, assigning the most likely tag for a given word in a given context (e.g., Church (1988), Cutting et al. (1992), DeRose (1988)). Another approach is the rule-based or constraint-based approach, recently most prominently exemplified by the Constraint Grammar work (Karlsson et al., 1995;Voutilainen, 1995b;Voutilainen et al., 1992;Voutilainen and Tapanainen, 1993), where a large number of hand-crafted linguistic constraints are used to eliminate impossible tags or morphological parses for a given word in a given context. Brill (1992; has presented a transformation-based learning approach, which induces rules from tagged corpora. Recently he has extended this work so that learning can proceed in an unsupervised manner using an untagged corpus (Brill, 1995b). Levinger et al. (1995) have recently reported on an approach that learns morpholexical probabilities from untagged corpus and have the used the resulting information in morphological disambiguation in Hebrew.
In contrast to languages like English, for which there is a very small number of possible word forms with a given root word, and a small number of tags associated with a given lexical form, languages like Turkish or Finnish with very productive agglutinative morphology where it is possible to produce thousands of forms (or even millions (Hankamer, 1989)) for a given root word, pose a challenging problem for morphological disambiguation. In English, for example, a word such as make or set can be verb or a noun. In Turkish, even though there are ambiguities of such sort, the agglutinative nature of the language usually helps resolution of such ambiguities due to restrictions on lnorphotactics. On the other hand, this very nature introduces another kind of ambiguity, where a lexical form can be morphologically interpreted in many ways, some with totally unrelated roots and morphological features, as will be exemplified in the next section.
Our previous approach to tagging and morphological disambiguation for Turkish text had employed a constraint-based approach (Oflazer and Kuru6z, 1994) along the general lines of similar previous work for English (Karlsson et al., 1995;Voutilainen et al., 1992;Voutilainen and Tapanainen, 1993). Although the results obtained there were reasonable, the fact that all constraint rules were hand crafted, posed a rather serious impediment to the generality and improvement of the system.
In this paper we present a constraint-based morphological disambiguation approach that uses unsupervised learning component to discover some of the constraints it uses in conjunction with hand-crafted rules. It is specifically applicable to languages with productive inflectional and derivational morphological processes, such as Turkish, where morphological ambiguity has a rather different nature than that found in languages like English. Our approach starts with a set of corpus-independent hand-crafted rules that reduce morphological ambiguity (hence improve precision) without sacrificing recall. It then uses an untagged training corpus in which all lexical items have been annotated with all possible morphological analyses, incrementally proposing and evaluating additional (possibly corpus dependent) constraints for disambiguation of morphological parses using the constraints imposed by unambiguous contexts. These rules choose or delete parses with specified features. In certain respects, our approach has been motivated by Brill's recent work (Brill, 1995b), but with the observation that his transformational approach is not directly applicable to languages like Turkish, where tags associated with forms are not predictable in advance.
In the following sections, we present an overview of the morphological disambiguation problem, highlighted with examples from Turkish.
We then present the details of our approach and results. We finally conclude after a discussion and evaluation of our results.

Tagging and Morphological Disambiguation
In almost all languages, words are usually ambiguous in their parts-of-speech or other lexical features, and may represent lexical items of different syntactic categories, or morphological structures depending on the syntactic and semantic context. Part-of-speech (POS) tagging involves assigning every word its proper part-of-speech based upon the context the word appears in. In English, for example a word such as set can be a verb in certain contexts (e.g., He set the table for dinner) and a noun in some others (e.g., We are now facing a whole set of problems).
In Turkish, there are ambiguities of the sort above. However, the agglutinative nature of the language usually helps resolution of such ambiguities due to the restrictions on morphotactics. On the other hand, this very nature introduces another kind of ambiguity, where a whole lexical form can be morphologically interpreted in many ways not predictable in advance. For instance, our full-scale morphological analyzer for Turkish returns the following set of parses for the word oysa: iOutput of the morphological anaJyzer is edited for clarity, and English glosses have been given.
2Glosses are given as linear feature value sequences corresponding to the morphemes (which are not shown). The feature names are as follows: CAT-major category, TYPE-minor category, R00T-main root form, AGR -number and person agreement, POSS-possessive agreement, CASE surface case, CONV-conversion to the category following with a certain suffix indicated by the argument after that, TAMl-tense, aspect, mood marker 1, SENSE-verbal polarity, DES-desire mood, IMP-imperative mood, 0PToptative mood, COND-Conditional Here, the original root is verbal but the final partof-speech is adjectival. In general, the ambiguities of the forms that come before such a form in text can be resolved with respect to its original (or intermediate) parts-of-speech (and inflectional features), while the ambiguities of the forms that follow can be resolved based on its final part-of-speech.
The main intent of our system is to achieve a morphological ambiguity reduction in the text by choosing for a given ambiguous token, a subset of its ZWith a slightly different but nevertheless common glossing convention.
4Upper cases in morphological output indicates one of the non-ASCII special Turkish characters: e.g., G denotes ~, U denotes /i, etc. parses which are not disallowed by the syntactic context it appears in. It is certainly possible that a given token may have multiple correct parses, usually with the same inflectional features or with inflectional features not ruled out by the syntactic context. These can only be disambiguated usually on semantic or discourse constraint grounds. 5 We consider a token fully disambiguated if it has only one morphological parse remaining after automatic disambiguation. We consider as token as correctly disambiguated, if one of the parses remaining for that token is the correct intended parse. 6 We evaluate the resulting disambiguated text by a number of metrics defined as follows (Voutilainen, 1995a): #Parses Ambiguity -#Tokens

Recall = #Tokens Correctly Disambiguatcd #Tokens ~Tokcns Correctly Disambiguated Precision = ~Parses
In the ideal case where each token is uniquely and correctly disambiguated with the correct parse, both recall and precision will be 1.0. On the other hand, a text where each token is annotated with all possible parses, 7 the recall will be 1.0 but the precision will be low. The goal is to have both recall and precision as high as possible. 3

Constraint-based Morphological Disambiguation
This section outlines our approach to constraintbased morphological disambiguation incorporating unsupervised learning component. Our system with the structure presented in Figure 1 has three main components: 1. the preprocessor, 2. the learning module, and 3. the morphological disambiguation module.
Preprocessing is common to both the learning and the morphological disambiguation modules. The module takes as input to the system raw Turkish text and preprocesses it in a manner to be described shortly.
If the text is to be used for training, the learning module then 1. applies an initial set of linguistically motivated hand-crafted constraint rules to choose and/or delete certain parses, and 5For instance the third and fourth parses for oysa above.
6It is certainly possible that, a parse that is deleted may also be a valid parse in that context. rAssuming no unknown words.
2. uses an unsupervised learning procedure to induce some additional (an possibly corpus dependent) rules to choose and delete some parses. Morphological disambiguation of previously unseen text proceeds as follows: 1. The hand-crafted rules are applied first.
2. Certain parses are deleted using context statistics on the corpus to be tagged.
3. Rules learned to choose and delete parses are then applied.

The Preprocessor
The preprocessing module takes as input a Turkish text, segments it into sentences using various heuristics about punctuation, tokenizes and runs it through a wide-coverage high-performance morphological analyzer developed using two-level morphology tools by Xerox (Karttunen, 1993). This module also performs a number of additional functions: • it groups lexicalized collocations such as idiomatic forms, semantically coalesced forms such as proper noun groups, certain numeric forms, etc.
• it groups any compound verb formations which are formed by a lexically adjacent, direct or oblique object, and a verb, which for the purposes of syntactic analysis, may be considered as single lexical item: e.g., saygz durmak (to pay respect), kafay~ yemek (literally t0 eat the head -to get mentally deranged), etc.
• it groups non-lexicalized collocations: Turkish abounds with various non-lexicalized collocations where the sentential role of the collocation has (almost) nothing to do with the partsof-speech of the individual forms involved. Almost all of these collocations involve duplications, and have forms like w + x w + y where w is the duplicated string comprising the root and certain sequence of suffixes and x and y are possibly different (or empty) sequences of other suffixes.
The following is a list of multi-word constructs for Turkish that we handle in our preprocessor. This list is not meant to be comprehensive, and new construct specifications can easily be added. It is conceivable that such a functionality can be used in almost any language. (See Oflazer and Kuru6z (1994) and KuruSz (1994) for details of all other forms for This module recognizes all such forms and coalesces them into new feature structures reflecting the final structure along with any inflectional information.
• The preprocessor then converts each parse into a hierarchical feature structure so that the inflectional feature of the form with the last category conversion (if any) are at the top level.
Thus in the example above for geldi~imdeki, the following feature structure is generated:  The set of features selected for each part-ofspeech category is determined by a template and hence is controllable, permitting experimentation with differing levels of information. The information selected for stems are determined by the category of the stem itself recursively.
Under certain circumstances where a token has two or more parses that agree in the selected features, those parses will be represented by a single projected parse, hence the number of parses in the (projected) training corpus may be smaller than the number of parses in the original corpus. For example, the feature structure above is projected into a feature structure such as: Unknown Words Although the coverage of our morphological analyzer for Turkish (Oflazer, 1993), with about 30,000 root words and about 35,000 proper names, is very satisfactory, it is inevitable that there will be forms in the corpora being processed that are not recognized by the morphological analyzer. These are almost always foreign proper names, words adapted into the language and not in the lexicon, or very obscure technical words. These are nevertheless inflected (using Turkish word formation paradigms) with inflectional features demanded by the syntactic context and sometimes even go through derivational processes. For improved disambiguation, one has to at least recover any morphological features even if the root word is unknown. To deal with this, we have made the assumption that all unknown words have nominal roots, and built a second morphological analyzer whose (nominal) root lexicon recognizes S + where S is the Turkish surface alphabet (in the two-level morphology sense), but then tries to interpret an arbitrary postfix of the unknown word as a sequence of Turkish suffixes subject to all morphographemic constraints. For instance when a form such as talkshowumun is entered, this second analyzer hypothesizes the following analyses: which are then processed just like any other during disambiguation.S This however is not a sufficient solution for some very obscure situations where for the foreign word is written using its, say, English orthography, while suffixation goes on according to its English pronunciation, which may make some constraints like vowel 8Incidentally, the correct analysis is the 6 th, meaning o.[ my talk show. The 5 th one has the same morphological features except for the root. harmony inapplicable on the graphemic representation, though harmony is in effect in the pronunciation. For instance one sees the form Carter'a where the last vowel in Carter is pronounced so that it harmonizes with a in Turkish, while the e in the surface form does not harmonize with a. We are nevertheless rather satisfied with our solution as in our experiments we have noted that well below 1% of the forms remain as unknown and these are usually item markers in formatted or itemized lists, or obscure foreign acronyms.

Constraint Rules
The which selects and adjective parse following a determiner, adjective sequence, and before a noun without a possessive marker. Another sample rule is: which chooses a nominal form with a possessive marker 2SG following a pronoun with 2SG agreement and genitive case, enforcing the simplest form of noun-noun form noun phrase constraints.
Our system uses two hand-crafted sets of rules, in combination with the rules that are learned by unsupervised learning: 1. We use an initial set of hand-crafted choose rules to speed-up the learning process by creating disambiguated contexts over which statistics can be collected. These rules (examples of which are given above) are independent of the corpus that is to be tagged, and are linguistically motivated. They enforce some very common feature patterns especially where word order is rather strict as in NP's or PP's. 9 The motivation behind these rules is that they should improve precision without sacrificing recall. These are rules which impose very tight constraints so as not to make any recall errors.
Our experience is that after processing with these rules, the recall is above 99% while precision improves by about 20 percentage points. Another important feature of these rules is that they are applied even if the contexts are also ambiguous, as the constraints are tight. That is, if each token in a sequence of, say, three ambiguous tokens have a parse matching one of the context constraints (in the proper order), then all of them are simultaneously disambiguated. In hand crafting these rules, we have used our experience from an earlier tagger (Oflazer and Kuruhz, 1994). Currently we use 288 handcrafted choose rules.
2. We also use a set of hand-crafted heuristic delete rules to get rid of any very low probability parses. For instance, in Turkish, postpositions have rather strict contextual constraints and if there are tokens remaining with multiple parses one of which is a postposition reading, we delete that reading. Our experience is that these rules improve precision by about 10 to 12 additional percentage points with negligible impact on recall. Currently we use 43 hand-crafted delete rules.

Learning Choose Rules
Given a training corpus, with tokens annotated with possible parses (projected over selected features), we first apply the hand-crafted rules. Learning then goes on as a number of iterations over the training corpus. We proceed with the following schema which is an adaptation of Brill's formulation (Brill, 1995b): 9Turkish is a free constituent order language whose unmarked order is SOV.
1. We generate a table, called incontext, of all possible unambiguous contexts which contain a token with an unambiguous (projected) parse, along with a count of how many times this parse occurs unambiguously in exactly the same context in the corpus. We refer to an entry in table with a context C and parse P as incontext(C, P). 2. We also generate a table, called count, of all unambiguous parses in the corpus along with a count of how many times this parse occurs in the corpus. We refer to an entry in this table with a given parse P, as count(P).
3. We then start going over the corpus token by token generating contexts as we go.
(b) The score of the candidate rule is then computed as:

Scorei = incontext(C, Pi) -count(P~) count(Pmax)"
incontext( C, Pmaz) 6. We order all candidate rules generated during one pass over the corpus, along two dimensions: (a) we group candidate rules by context specificity (given by the order in Section 3.3), (b) in each group, we order rules by descending score.
We maintain score thresholds associated with each context specificity group: the threshold of a less specific group being higher than that of a more specific group. We then choose the top scoring rule from any group whose score equals or exceeds the threshold associated with that group. The reasoning is that we prefer more specific and/or high scoring rules: high scoring rules are applicable, in general, in more places; while more specific rules have stricter constraints and more accurate morphological parse selections, We have noted that choosing the highest scoring rule at every step may sometimes make premature commitments which can not be undone later.
1°Either of LC or RC may be empty.
7. The selected rules are then applied in the matching contexts and ambiguity in those contexts is reduced. During this application the following are also performed: (a) if the application results in an unambiguous parse in the context of the applied rule, we increment the count associated with this parse in table count. We also update the incontext table for the same context, and other contexts which contains the disambiguated parse. (b) we also generate any new unambiguous contexts that this newly disambiguated token may give rise to, and add it to the incontext table along with count 1.
Note that for efficiency reasons, rule candidates are not generated repeatedly during each pass over the corpus, but rather once at the beginning, and then when selected rules are applied to very specific portions of the corpus.
8. If there are no rules in any group that exceed its threshold, group thresholds are reduced by multiplying by a damping constant d (0 < d < 1) and iterations are continued.
9. If the threshold for the most specific context falls below a given lower limit, the learning process is terminated.
Some of the rules that have been generated by this learning process are given below: 3.4.1 Contexts induced by morphological derivation The procedure outlined in the previous section has to be modified slightly in the case when the unambiguous token in the rc position is a morphologically derived form. For such cases one has to take into consideration additional pieces of information.
We will motivate this using a simple example from Turkish. Consider the example fragment: ... bir masa+dlr. where the determiner is attached to the noun and the whole phrase is then taken as a VP although the verbal marker is on the second lexical item. If, in this case, the token bit is considered to neighbor a token whose top level inflectional features indicate it is a verb, it is likely that bit will be chosen as an adverb as it precedes a verb, whereas the correct parse is the determiner reading.
In such a case where the right context of an ambiguous token is a derived form, one has to consider as the right context, both the top level features of final form, and the stem from which it was derived. During the set-up of the incontext table, such a context is entered twice: once with the top level feature constraints of the immediate unambiguous right-context, and once with the feature constraints of the stem. The unambiguous token in the right context is also entered to the count table once with its top level feature structure and once with the feature structure of the stem.
When generating candidate choose or delete rules, for contexts where rc is a derived form and rrc is empty, we actually generate two candidates rules for each ambiguous token in that context: 1. if llc, ic and rc then choose/delete Pi.
2. if llc, Ic and stem(re) then choose/delete P~.
These candidate rules are then evaluated as described above. In general all derivations in a lexical form have to be considered though we have noted that considering one level gives satisfactory results.

Ignoring Features
Some morphological features are only meaningful or relevant for disambiguation only when they appear to the left or to the right of the token to be disambiguated. For instance, in the case of Turkish, the CASE feature of a nominal form is only useful in the immediate left context, while the POSS (the possessive agreement marker) is useful only in the right context. If these features along with their possible values are included in context positions where they are not relevant, they "split" scores and hence cause the selection of some other irrelevant rule. Using the maxim that union gives strength, we create contexts so that features not relevant to a context position are not included, thereby treating context that differ in these features as same. 11

Learning Delete Rules
For choosing delete rules we have experimented with two approaches. One obvious approach is to use the formulation described above for learning choose rules, but instead of generating choose rules, pick the parses that score (significantly) worse than and generate delete rules for such parses. We have implemented this approach and found that it is not very desirable due to two reasons: 1. it generates far too many delete rules, and 2. it impacts recall seriously without a corresponding increase in precision.
The second approach that we have used is considerably simpler. We first reprocess the training corpus but this time use a second set of projection templates, and apply initial rules, learned choose rules and heuristic delete rules. Then for every unambiguous context C = (LC, RC), with either an immediate left, or an immediate right components or both (so n Obviously these features are specific to a language. the contexts used here are the last 3 in Section 3.3), a score incontext ( C, Pi ) count ( Pi ) for each parse Pi of the (still) ambiguous token, is computed. Then, delete rules of the sort if LC and RC then delete Pi are generated for all parses with a score below a certain fraction (0.2 in our experiments) of the highest scoring parse. In this process, our main goal is to remove any seriously improbable parses which may somehow survive all the previous choose and delete constraints applied so far. Using a second set of templates which are more specific than the templates used during the learning of the choose rules, we introduce features we were originally projected out. Our experience has been that less strict contexts (e.g., just alc or rc) generate very useful delete rules, which basically weed out what can (almost) never happen as it is certainly not very feasible to formulate hand-crafted rules that specify what sequences of features are not possible. Some of the interesting delete rules learned here are: 1. Delete the first of two consecutive verb parses: 2. Delete accusative case marked noun parse before a postposition that subcategorizes for a nominative noun:

Using context statistics to delete parses
After applying hand-crafted rules to a text to be disambiguated we arrive at a state where ambiguity is about 1.10 to 1.15 parses per token (down from 1.70 to 1.80 parses per token) without any serious loss on recall. This state allows statistics to be collected over unambiguous contexts. To remove additional parses which never appear in any unambiguous context we use the scoring described above for choosing delete rules, to discard parses on the current text based on context statistics} 2 We make three passes 12Please note that delete rules learned may be applied to future texts to be disambiguated, while this step is over the current text, scoring parses in unambiguous contexts of the form used in generating delete rules, and discarding parses whose score is below a certain fraction of the maximum scoring parse, on the fly. The only difference with the scoring used for delete rules, is that the score of a parse Pi here is a weighted sum of the quantity incontext (C, Pi) count ( Pi ) evaluated for three contexts in the case both the lc and rc are unambiguous/

Steps in Disamblguating a Text
Given a new text annotated with all morphological parses (this time the parses are not projected), we proceed with the following steps for disambiguation: 1. The initial hand-crafted choose rules are applied first. These rules always constrain top level inflectional features, and hence, any stems fromn derivational processes are not considered unless explicitly indicated in the constraint itself.
2. The hand-crafted delete clean-up rules are applied.
3. Context statistics described in the preceding section are used to discard further parses.
4. The choose rules that have been learned earlier, are then repeatedly applied to unambiguous contexts, until no more ambiguity reduction is possible. During the application of these rules, if the immediate right context of a token is a derived form, then the stem of the right context is also checked against the constraint imposed by the rule. So if the rule right context constraint subsumes the top level feature structure or the stem feature structure, then the rule succeeds and is applied if all other constraints are also satisfied.
5. Finally, the delete rules that have been learned are applied repeatedly to unambiguous contexts, until no more ambiguity reduction is possible.

Experimental Results
We have applied our learning system to two Turkish texts. Some statistics on these texts are given in Table 1. The first text labeled ARK is a short text on near eastern archaeology. The second text from which fragments whose labels start with C are derived, is a book on early 20 ~h history of Turkish Republic.
In Table 1, the tokens considered are that are generated after morphological analysis, unknown word processing and any lexical coalescing is done. The applied to the current text on which disambiguation is performed.  words that are unknown are those that could not even be processed by the unknown noun processor. Whenever an unknown word had more than one parse it was counted under the appropriate group. We learned rules from ARK itself, and on the first 500, 1000, and 2000 sentence portions of C2400. C270 which was from the remaining 400 sentences of C2400 was set aside for testing. Gold standard disambiguated versions for ARK, C270 were prepared manually to evaluate the automatically tagged versions.
Our results are summarized in the following set of tables. Tables 2 and 3 give the ambiguity, recall and precision initially, after hand-crafted rules are applied, and after the contextual statistics are used to remove parses -all applications being cumulative. The rows labeled BASE give the initial state of the text to be tagged. The rows labeled INITIAL CHOOSE give the state after hand-crafted choose rules are applied, while the rows labeled INI-TIAL DELETE give the state after the hand-crafted choose and delete rules are applied. The rows labeled CONTEXT STATISTICS give the state after the rules are applied and context statistics are used (as described earlier) to remove additional parses.

Disambiguation Ambiguity Recall
Pre.   Tables 5 and 6 present the results of further disambiguation of ARK, and C270 using rules learned from training texts C500, C1000, C2000 and ARK. These rules are applied after the last stage in the ta-bles above. 13 The number of rules learned are given in Table 4 Table 5: Average parses, recall and precision for text ARK after applying learned rules. Table 7 gives some additional statistical results at the sentence level, for each of the test texts. The columns labeled UA/C and A/C give the number and percentage of the sentences that are correctly disambiguated with one parse per token, and with more than one parse for at least one token, respectively. The columns labeled 1, 2, 3, and >3 denote the number and percentage of sentences that have 1, 2, 3, and >3 tokens, with all remaining parses incorrect. It can be seen that well 60% of the sentences are correctly morphologically disambiguated with very small number of ambiguous parses remaining.

Discussion of Results
We can make a number of observations from our experience: Hand-crafted rules go a long way in improving precision substantially, but in a language like Turkish, one has to code rules that allow no, or only carefully controlled derivations, otherwise lots of things go massively wrong. Thus we have used very tight and conservative rules in hand-crafting.
Although the additional impact of choose and rules that are induced by the unsupervised learning is not substantiM, this is to be expected as the stage at which they are used is when all the "easy" work has been done and the more notorious cases remain. An important class of rules we explicitly have avoided hand crafting are rules for disambiguating around coordinating conjunctions. We have noted that while learning choose rules, the system zeroes in rather quickly on these contexts and comes up with rather successful rules for conjunctions. Similarly, the delete rules find some interesting situations which would be virtually impossible to enumerate. Although it is easy to formulate what things can go together in a context, it is rather impossible to formulate what things can not go together. We have also attempted to learn rules directly without applying any hand-crafted rules, but this has resulted in a failure with the learning process getting stuck fairly early. This is mainly due to the lack of sufficient unambiguous contexts to bootstrap the whole disambiguation process.
From analysis of our results we have noted that trying to choose one correct parse for every token is rather ambitious (at least for Turkish). There are a number of reasons for this: There are genuine ambiguities. The word o is either a personal or a demonstrative pronoun (in addition to being a determiner). One simply can not choose among the first two using any amount of contextual information.
A given word may be interpreted in more than one way but with the same inflectional features, or with features not inconsistent with the syntactic context. This usually happens when the root of one of the forms is a proper prefix of the root of the other one. One would need serious amounts of semantic, or statistical root word and word form preference information for resolving these. where again with have a similar problem. It may be possible to resolve this one using subcategorization constraints on the object of the verb kur assuming it is in the very near preceding context, but this may be very unlikely as Turkish allows arbitrary adjuncts between the object and the verb.
Turkish allows sentences to consist of a number of sentences separated by commas. Hence locating a verb in the middle of a sentence is rather difficult, as certain verbal forms also have an adjectival reading, and punctuation is not very helpful as commas have many other uses.
The distance between two constituents (of, say, a noun phrase) that have to agree in various morphosyntactic features may be arbitrar- Choosing among the last three is rather problematic if the corresponding genitive form to force agreement with is outside the context.
Among these problems, the most crucial is the second one which we believe can be solved to a great extent by using root word preference statistics and word form preference statistics. We are currently working on obtaining such statistics.

Conclusions
This paper has presented a rule-based morphological disambiguation approach which combines a set of hand-crafted constraint rules and learns additional rules to choose and delete parses, from untagged text in an unsupervised manner. We have extended the rule learning and application schemes so that the impact of various morphological phenomena and features are selectively taken into account. We have applied our approach to the morphological disambiguation of Turkish, a free-constituent order language, with agglutinative morphology, exhibiting productive inflectional and derivational processes. We have also incorporated a rather sophisticated unknown form processor which extracts any relevant inflectional or derivational markers even if the root word is unknown.
Our results indicate that by combining these hand-crafted, statistical and learned information sources, we can attain a recall of 96 to 97% with a corresponding precision of 93 to 94% and ambiguity of 1.02 to 1.03 parses per token, on test texts, however the impact of the rules that are learned is not significant as hand-crafted rules do most of the easy work at the initial stages.