{'fraction_non_alphanumeric': 0.0310391363022942, 'fraction_numerical': 0.012627723154038943, 'mean_word_length': 4.744739756367664, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Simple Parser for Indian Languages in a Dependency Framework

This paper is an attempt to show that an intermediary level of analysis is an effective way for carrying out various NLP tasks for linguistically similar languages. We describe a process for developing a simple parser for doing such tasks. This parser uses a grammar driven approach to annotate dependency relations (both inter and intra chunk) at an intermediary level. Ease in identifying a particular dependency relation dictates the degree of analysis reached by the parser. To establish efficiency of the simple parser we show the improvement in its results over previous grammar driven dependency parsing approaches for Indian languages like Hindi. We also propose the possibility of usefulness of the simple parser for Indian languages that are similar in nature.


Introduction and Related Work
Broad coverage parsing is a challenging task. For languages such as the Indian languages, it becomes all the more difficult as these languages are morphologically richer and the word order for these languages is relatively variable and less bound. Although dependency grammar driven parsing is much better suited for such type of languages (Hudson, 1984;Mel"Cuk, 1988), robust broad coverage parsing (Bharati et. al, 2008) still involves extensive analysis. Achieving good results in parsing for these languages may require large amount of linguistic resources such as annotated corpora, verb frames, lexicon etc. On the other hand, pure shallow parsing techniques (PVS and Gali, 2007) are not enough for providing sufficient information for applications such as machine translation, query answering etc.
It is here that the notion of a simple parser is born where the idea is to parse a sentence at a coarser level. One could go to a finer level of parse depending on the ease with which such a parse can be generated. The simple parser that we describe here is a grammar oriented model that makes use of linguistic features to identify relations. We have modeled the simple parser on the Paninian grammatical model (Begum et al., 2008;Bharati et al., 1995) which provides a dependency grammar framework. Paninian dependency grammar works well for analyzing Indian languages (Bharati et al., 1993). We have followed karaka 1 based approach for parsing. An effort has been previously made in grammar driven parsing for Hindi by us (Gupta et al., 2008) where the focus was not to mark relations in a broad coverage sense but to mark certain easily identifiable relations using a rule base. In this paper, we show improvements in results over our previous work by including some additional linguistic features which help in identifying relations better. Our previous work focused only on inter-chunk annotation. In this paper, however, we have worked on both inter as well as intra chunk annotation. We later show their effectiveness and results at different levels of dependency annotation. We also propose how useful the simple parser is for Indian languages which are similar in nature.

Paninian Dependency Annotation Scheme at Various Levels
Paninian dependency scheme is based on a modifier-modified relationship (Bharati et al., 1995). The modified chunk (or group) is classified on the basis of its part of speech category. A hierarchy of dependency relations is thus established on the basis of this category. For example, all those relations whose parent (modified group) is a verb are classified under the verb modifier (vmod) category. Subsequent levels further classify these relations (or labels). Depth of a level in the hierarchy reflects the fineness of the dependency relations/labels. There are five labels at the coarsest level namely, vmod, nmod (noun modifier), jjmod (adjective modifier), advmod (adverbial modifier) and ccof (conjunct of). Athough, ccof is not strictly a dependency relation (Begum et al., 2008). Figure 1 shows the hierarchy of relations used in the scheme. The next level comprises of varg (verb argument), vad (verb adjunct) and vmod_1 2 labels under vmod. Under the nmod label, nmod_adj (adjective), r6 (genitive) are classified. At the most fine grained level, varg and vad further branch out into labels like k1, k2, k3, k5, k7 and rh, rt, rd, k1s etc. The relations under varg are the six karakas that are the most essential participants in an action. All the other dependency labels 3 are non-karakas (for a more detailed explanation see Begum et al. (2008) and Bharati et al. (1995)).
Languages often have constructions that are ambiguous, owing to similar feature and context distribution. Thus, in such cases, it is appropriate to under-specify the relations (labels) or group some of them together. Also, some labels have very less frequency of occurrence in the corpus and it is thus appropriate to leave them out for marking by the simple parser. One can later, on the availability of more information, try to identify and mark such instances with appropriate labels.
The dependency tagset described in this section is used to mark inter-chunk relations. For marking relations between words within a chunk (intra-chunk), a similar tagset has been developed.
2 vmod_1: A dependency relation in the vmod category, that exists between a non-finite verb and its parent verb. It has been under-specified for simplicity. 3 A comprehensive list of the dependency tagset can be found at http://ltrc.iiit.ac.in/MachineTrans/research/tb/deptagset.pdf

Procedure
Our approach is corpus based where rules have been crafted after studying the corpus. We used the Hyderabad Dependency Treebank (HyDT) for development and testing our rules. The treebank consists of about 2100 sentences in Hindi, of which 1800 were part of the development set and 300 were used as test data. Each sentence is POS tagged and chunked (Bharati et al., 2006) in SSF format (Bharati et al., 2005).

Approach
The simple parser we propose here is a language independent engine that takes a rule file specific for a particular language (Gupta et. al, 2008). Indian languages are similar in various respects (Emeneau 1956;1980). Hence, rules made for one language can be efficiently transferred for other similar languages. However, there can be cases where rules for one language may not work for another. These cases can be handled by adding some new rules for that particular language. The relative closeness among such languages, determines the efficiency of transference of rules from one language to another. We have taken Hindi and Punjabi, as example languages to support our proposal. 1(a) below is in Hindi, 1(a). raama ko mithaaii acchii nahii "Ram -dat" "sweets" "good" "not" lagatii. "appear" "Ram does not like sweets." Its corresponding Punjabi sentence, 1(b). raama nuu mitthaai changii nii "Ram -dat" "sweets" "good" "not" lagadii. "appear" "Ram does not like sweets." Now, the rules for identifying k1 4 and k2 in Hindi are similar to that of Punjabi. For instance, in both the cases, the noun chunk possessing a nominative case marker (chunks take the properties of their heads) and the TAM (tense, aspect and modality of the main verb) should agree in GNP for the noun to be a k2. It is easy to see how rules made for identifying certain relations in Hindi can be transferred to identify the same relations in Punjabi and similarly for other languages. However, not all rules can be transferred from one language to another.

Intra-chunk Relations
We also mark intra-chunk dependency relations. The procedure of marking intra-chunk labels is also rule based. Rules have been crafted using a common POS 5 tagset for Indian languages (Bharati et al., 2006). Rules can be applied to other languages. However, some rules may not work. In those cases we need to add some rules specific to the language. The rule format is a five-tuple containing the following fields, 1. Modified word 2. Modified constraints 3. Modifier word 4. Modifier constraints 5. Dependency relation Rules for marking intra-chunk relations have been marked studying the POS tagged and chunked corpus. Commonly occurring linguistic patterns between two or more nodes are drawn out in the form of statistics and their figures are collected. Such patterns are then converted into robust rules.

Experiments and Results
We conducted experiments using the simple parser to establish its efficacy in identifying a particular set of relations explained in section 2. Experiments were conducted on gold standard test data derived from HyDT. The experiments were carried out on Hindi.

Marking Relations at Various Levels
We marked dependency labels at various levels described above using the proposed simple parser. The results are shown below We report two measures for evaluation, labeled (L) and labeled attachment (LA). Table 1 shows results for marking relations at the top most level (cf. Figure 1).
It should be noted that we have not marked relations like jjmod and advmod because the frequency of their occurrence in the treebank is quite low. The focus is only on those relations whose frequency of occurrence is above a bare minimum (>15). The frequency of labels like jjmod and advmod is not above that threshold 5 POS: Part of Speech value (Relations like k1 and k2 occur more than 1500 times in the treebank).  In section 1, improvement in marking certain relations over our previous attempt (Gupta et. al, 2008)

Intra-chunk Experiments
We also carried out some experiments to determine the efficiency of the simple parser with respect to annotating intra-chunk relations for Hindi. Results shown below were obtained after testing the simple parser using gold standard test data of about 200 sentences.

Conclusion
We introduced the notion of a simple parser for Indian languages which follows a grammar driven methodology. We compared its performance against previous similar attempts and reported its efficiency. We showed how by using simple yet robust rules one can achieve high performance in the identification of various levels of dependency relations. The immediate tasks for the near future would be to identify relative clauses in order to reduce labeled attachment errors and hence to come up with rules for better identification of clauses. We also intend to thoroughly test our rules for Indian languages that are similar in nature and hence evaluate the efficiency of the simple parser.