{'fraction_non_alphanumeric': 0.027174603174603174, 'fraction_numerical': 0.008317460317460317, 'mean_word_length': 4.857567869096318, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Improving Dependency Parsing with Interlinear Glossed Text and Syntactic Projection

Producing annotated corpora for resource-poor languages can be prohibitively expensive, while obtaining parallel, unannotated corpora may be more easily achieved. We propose a method of augmenting a discriminative dependency parser using syntactic projection information. This modification will allow the parser to take advantage of unannotated parallel corpora where high-quality automatic annotation tools exist for one of the languages. We use corpora of interlinear glossed text—short bitexts commonly found in linguistic papers on resource-poor languages with an additional gloss line that supports word alignment—and demonstrate this technique on eight different languages, including resource-poor languages such as Welsh, Yaqui, and Hausa. We find that incorporating syntactic projection information in a discriminative parser generally outperforms deterministic syntactic projection. While this paper uses small IGT corpora for word alignment, our method can be adapted to larger parallel corpora by using statistical word alignment instead.


Introduction
The development of large-scale treebanks has significantly improved the performance of statistical parsers (e.g. Klein and Manning, 2001;Collins and Koo, 2005;de Marneffe et al., 2006) Unfortunately, resources such as these typically only exist for a handful of languages, because building large treebanks is very labor-intensive and often cost-prohibitive. As thousands of other languages have no such resources, other techniques are required to attain similar performance.
One way in which natural language tools might be created for resource-poor languages is by using resources containing translations between resource-rich and resource-poor languages and use the alignment information to transfer information to the resource-poor ones. Syntactic projection takes advantage of existing tools used to annotate a resource-rich language in a corpus, and transfers the analysis to the resource-poor language by means of syntactic projection using word-to-word alignment (Yarowsky and Ngai, 2001;Hwa et al., 2002).
While little annotated data typically exists for resource-poor languages, some work has been done examining the utility of interlinear glossed text (IGT) (Xia and Lewis, 2007;Lewis and Xia, 2008), a format used for illustrative examples in linguistic papers. An IGT instance includes a language line which is a phrase or sentence in a foreign language, a gloss line that shows word-to-word translation, and a translation line which is normally in English. We chose IGT for this study because the gloss line can serve as a bridge for aligning the words in the language line and the translation line. The method proposed in this paper can be easily extended when IGT is replaced by bitexts of sufficient size to train a high-quality word aligner.
In this paper, we investigate the possibility of using small corpora of interlinear text and syntactic projection to bootstrap a dependency parser and improve the resulting parses over projection alone.

Background
Before presenting our system, we will first describe previous studies on syntactic projection. Next, we will describe the phenomenon of linguistic divergence, and explain why this can affect projection performance. Finally, we will describe interlinear glossed text (IGT) in more detail and highlight the ways in which it is well-suited to this task.

Syntactic Projection
Syntactic projection via word alignment has shown promise in adapting resources between languages. Merlo et al. (2002) demonstrated a technique of classifying verb types via projection, while Yarowsky and Ngai (2001) worked on projecting POS taggers and NP bracketers. Hwa et al. (2004) bootstrapped both phrase and dependency parsers. While these systems did not match the performance of supervised systems, they do succeed in demonstrating that even a small amount of information can significantly boost the performance of a baseline system. Syntactic projection, however, suffers from a major flaw-using word alignment to transfer analyses between languages assumes that the language pair represents the similar sentences using similar structures. Hwa et al. (2002) referred to this as the Direct Correspondence Assumption, or DCA. As useful as this assumption may be, Dorr (1994) discussed how languages may be divergent in their representations of similar sentences in semantic, syntactic, or lexical representations.    Dorr (1994) defined several types of translation divergence. These variations, or linguistic divergence, can be problematic for projection algorithms, as the assumptions that projection relies on may not hold. One example which may affect projections is the light-verb construction. Take as an example the Hindi sentence in Figure 1, where an English verb ("inaugurated") is represented in Hindi as a light verb ("did") plus a noun ("inaugurate"). As a result, the dependency structures in English and Hindi are not identical, as illustrated in Figure 1

Linguistic Divergence
In order to address some of the noise inherent in projection results, Ganchev et al. (2009) took an approach of using "soft" constraints to improve projection results. Rather than commit to a single parse, Ganchev et al. used statistical methods to disambiguate between multiple parse options, and showed significant improvements over a purely deterministic approach.
In our previous study, (Georgi et al., 2012), we looked at measuring forms of alignment between dependency structures to quantify the amount of divergence between languages. In this paper we will look at how performance varies among a set of typologically different languages. We hope to investigate the correlation between performance in these languages and these quantitative measures in future work.

Interlinear Glossed Text (IGT)
IGT is a resource well-suited to the task of adapting dependency parsing to new languages. As seen in Figure 1a, an IGT instance contains a foreign-language line, an English translation, and a gloss line, which provides additional annotation about the foreign language line. Xia and Lewis (2007) demonstrated that interlinear glossed text can be used to obtain high quality word-to-word alignments with a relatively small amount of data. Xia  that these heuristics can be used to augment statistical methods to produce higher-quality alignments over statistical methods alone, suggesting that a small corpus of IGT may be able to provide a bootstrap for alignment on much larger parallel corpora.
Where does one find IGT corpora? The Online Database of INterlinear text (ODIN) (Lewis and Xia, 2010) is an online resource of IGT instances for that contains approximately two hundred thousand instances for 1274 languages. Lewis and Xia (2008) use ODIN data for 97 languages to perform syntactic projection and determine basic word order. They found that the languages in this sample with 40 or more instances could be used to predict basic word order with 99% accuracy. With such broad language coverage, ODIN is an ideal resource for providing information for resource-poor languages for which little other data exists.

Methodology
Given a set of IGT instances, our system works as follows (see Figure 2 If the input is a set of parallel sentences instead of IGT, the process will be exactly the same except that word alignment in step (1) will be done by training a statistical word aligner, instead of using the gloss line in IGTs as a bridge between the language lines and the translation lines. (a) Flowchart illustrating the training procedure for a given L F -L E language pair, where L F is the foreign (non-English) language, and L E is English.   In this study, we will focus the last two steps. Therefore, for the first two steps, we use word alignment and English parse trees from the gold standard. The last two steps and the evaluation corpora are explained below.

Corpora
We used two different sources of language data for our experiment. The first was a set of guideline sentences for the Hindi treebank (Bhatt et al., 2009). These sentences were provided both in the IGT format, as well as with gold-standard dependency trees in both English and Hindi. The second set was the IGT data used in Xia and Lewis (2007), which includes IGTs for seven languages, plus manually annotated word alignment and dependency trees for both English and the foreign language. In all, eight languages were used for our experiments: Hindi, German, Irish, Hausa, Korean, Malagasy, Welsh, and Yaqui. The size of each data set is given in Table 1. We use the pre-existing, manually annotated parse trees in these two data sets as our gold standard for evaluation.
For the Hindi data, which did not include word alignment, we first automatically aligned the Hindi sentences and the English translation via the gloss line as in Xia and Lewis (2007), then manually corrected the alignment errors. While this manual correction step creates alignments that are not fully automated, the amount of effort required to make these modifications are minimal, and still greatly reduces the costs involved in creating parallel annotated corpora.

The Projection Algorithm
For the projection algorithm, we follow the dependency projection algorithm described in Xia and Lewis (2007), an example of which can be seen in Figure 3. Given the word alignment L E and L F and the parse tree T E for L E , the projection algorithm works as follows. For each English node e i that aligns with foreign word f i , we replace the node for e i with the foreign word. If a single English node e i aligns with multiple foreign words ( f i , f j ), we make multiple copies of e i as siblings in the tree for each source word, then replace the English words with those from the source. If multiple English nodes align to a single foreign word, the node highest up in the tree is kept, and all others removed. Finally, remaining unaligned words in L F are attached heuristically, following (Quirk et al., 2005).

The Parser
Due to linguistic divergence, projected trees are error-prone. Instead of making hard decisions based on projection, we use information from projected trees as a feature in a discriminative parser. This feature will be highly predictive, but not result in a strictly deterministic method like projection alone. We modified the MST Parser (McDonald et al., 2006) by adding features that check whether certain edges considered by the parser appear in the projected tree. We define two types of features: BOOL, and TAG.
The first feature type, BOOL, looks at the current parent→child edge being considered by the parser and returns true if the edge matches one in the projected tree. While this feature was a logical starting point, we also wondered if certain word classes of English projected better than others, and so the second feature type, TAG, creates a feature for each POS par ent , POS child pair such that the feature is true if the current parent→child edge being considered matches the one in the projected tree, and the POS tags of the parent and child are POS_par ent and POS child , respectively.

Experiments
As shown in Figure 2a, the L F parser is trained with two kinds of input: (1) projected L F trees from which BOOL and TAG features are extracted, and (2) "Gold" trees for L F from which the standard features used by the MST parser are extracted.
We ran two sets of experiments. In the first, the "Gold" trees are the same as the projected L F trees. This is to replicate the case when no gold standard is available for L F . In the second set of experiments, the "Gold" trees are indeed the manually-corrected trees for the L F sentences. The results are shown in Tables 2a and 2b, respectively.
In each set, there are several individual experiments. The "projection" row shows the results of evaluating the projected L F trees directly without training a parser. The "MST baseline (B)" row shows the results when we train the MST parser without part-of-speech (POS) tag features and features from projected L F trees. The third to seventh rows show the results when features from the projected L F trees are added when training the MST parser. Finally, we want to see how well the system works if the projected L F trees are perfect, so in the first two rows, we replace the projected L F trees with the L F trees from the gold standard. Because our data sets are small, we ran ten-fold cross validation. The highest results in each column for rows 3-9 are shown in bold.

Discussion
There are several observations to make from Table 2. First, comparing Tables 2a and 2b, it is clear that using the gold standard trees for L F improves performance. Second, the projected trees are error-prone, as shown by the last row of the two tables, indicating linguistic divergence is very common. Third, in Table 2a, adding projection-derived features helps the MST baseline, but the results are not better than the "projection" row because the parser is trained on the projected trees only. In contrast, when the parser is trained on the correct parse trees with  "POS" represents the experiments where part-of-speech tags were projected from the English side, and "Bool" and "Tag" are features as explained in §3.3. The best result for the non-oracle runs is shown in bold.
additional features derived from projection (see Table 2b), the results are much better than both the MST baseline and projection, and often outperforms the heuristics-based projection algorithm as well. This indicates that, although projected trees are error-prone, using features from them indeed improves parsing performance.

Conclusion and Future Work
Building large-scale treebanks is very labor-intensive and often cost-prohibitive. As thousands of languages do not have such resources, syntactic projection has been proposed to transfer syntactic structure from resource-rich languages to resource-poor languages and the projected structure is used to bootstrap NLP tools. However, the projected structure is error-prone due to linguistic divergence.
We propose to augmenting a discriminative dependency parser by adding features extracted from projected structures, and have evaluated our system on eight typologically diverse languages, most of which are extremely resource-poor. The experiments show that the augmented parser outperforms both the original parser without the projection-derived features and the parse trees produced by the projection algorithm only. While the corpora used here are very small, they nonetheless are working examples and show that despite the linguistic divergence, parsing performance can be improved by using features extracted from the dependency trees produced by syntactic projection. Using the IGT data found in the ODIN database and elsewhere on the web, it is conceivable that our method can be applied to bootstrap dependency parsers for hundreds of languages at a very low cost.
For future work, there are several avenues we would like to investigate further. First, the results in this study have all been obtained from very small sets of data. Extending one or more of these languages out to a larger data set may give us a better understanding of the effects of incorporating IGT information. Second, we plan to extend our system to take advantage of a large amount of bitext if it is available. For that, we will train statistical word aligners and test how word alignment errors affect system performance. Finally, while the features used in this study look solely at the result of the projected trees, we would like to add features that look at different divergence types as discussed in Georgi et al. (2012).