{'fraction_non_alphanumeric': 0.032489610880241784, 'fraction_numerical': 0.007933509633547412, 'mean_word_length': 5.057589626239512, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Identifying and Handling Cross-Treebank Inconsistencies in UD: A Pilot Study

The Universal Dependencies treebanks are a still-growing collection of treebanks for a wide range of languages, all annotated with a common inventory of dependency relations. Yet, the usages of the relations can be categorically different even for treebanks of the same language. We present a pilot study on identifying such inconsistencies in a language-independent way and conduct an experiment which illustrates that a proper handling of inconsistencies can improve parsing performance by several percentage points.


Introduction
Merging several available treebanks for the same language is a simple way of obtaining more dataeither for training language-specific tools or for systematic corpus studies on the language. This strategy presupposes that each of the merged treebanks adheres to the same annotation scheme; when treebanks to be merged originate from distinct research contexts (which is typically the case in practice), a harmonization process is required. The UD initiative (whose main motivation has been to obtain comparable syntactic annotations across languages) has the welcome side effect that whenever several corpora for the same language are included in the UD collection, the amount of consistently annotated treebank data for this language increases -at least in theory. In practice, there can be serious inconsistencies in the way the dependency relations are used in the respective treebanks. This can for instance be the effect of distinct strategies in conversion from a non-UD treebank format, or of idiosyncrasies in the linguistic treatment of a certain construction pursued in one treebanking team. Often such inconsistencies may go unnoticed -in particular when researchers do experiments on large samples of languages from UD -and may lead to problematic conclusions.
Some previous efforts have focused on discovering errors or inconsistencies within a treebank, by measuring the patterns based on partially lexicalized dependency relations (Boyd et al., 2008;de Marneffe et al., 2017) or calculating the fitness scores of the dependency relations in a tree (Alzetta et al., 2017). In contrast to these studies, we focus on finding inconsistencies between treebanks that are supposed to be annotated in the same way according to a common guideline. Our method builds on the assumption that each individual treebank is already largely consistent within itself, and seeks to identify dependency patterns with contradicting statistics across treebanks.
So far, annotation inconsistencies have been reported for some specific languages, e.g. Korean (Noh et al., 2018) and Russian (Droganova et al., 2018), but there seems to be no comprehensive study addressing these cross-treebank inconsistencies within the UD collection. In our pilot study, we experiment with a dependency-based measure for inconsistency detection that makes no specific language-typological assumptions and appears to be quite effective for detecting various types of inconsistencies. In two experiments, we demonstrate that inconsistencies detected this way can be used as the trigger for a comparatively simple, but effective conversion process on one of the treebanks: a parser trained on the converted treebank achieves higher accuracy on the treebank that is closer to the UD guidelines.

Methodology
We became aware of inconsistent usages of dependency relations when extracting relative branching directions for another task (Dönicke et al., 2020;Dönicke, 2020) and we reuse this measure here to find potential cross-treebank inconsistencies, without making any prior assumptions what could be a problematic construction in a language under consideration. The measure takes advantage of the fact that the frequency distribution of the directionality of dependencies is by and large independent of domainspecific factors (which will of course differ across treebanks). Moreover, many of the problematic inconsistencies found across treebank schemes go along with the choice of head vs. dependency status and will thus be reflected in the directionality of the arcs for a particular label -even when the label is used fairly infrequently.
The relative frequency of right-branching instances for a dependency relation R in a treebank t is Here, an expression of the form #[ * ](t) returns how often the construction * appears in the treebank t. The value ranges between 0 and 1; for example, v t amod = 0 indicates that adjectival modifiers never follow (i.e. always precede) their governor in t, 1 indicates that they always follow their governor in t, and 0.5 indicates that both orderings are equally common. This measure is very similar to the directional dependency distance of Chen and Gerdes (2017). Our measure only quantifies the average direction of a relation whereas their measure quantifies average direction and distance.
We found that the relative branching direction of a relation sometimes greatly differs among the treebanks of the same language. Therefore, we calculate the maximum branching direction difference (MBD) for a language as MBD(R, ) = max where T ( ) is the set of all treebanks for . A higher value indicates that there are at least two treebanks for in which R shows a different branching behaviour.

Inconsistencies among treebanks
We use the MBD to search for inconsistent usages of dependency relations in UD, version 2.5 (Zeman et al., 2019). Table 1 shows languages and relations which have high differences in branching direction between treebanks. 2 We reduced all dependency relations to their base forms (e.g. "csubj:pass" to "csubj") and excluded dependency relations that are very infrequent and tend to have very dissimilar occurrences. 3 We then manually inspected the first 20 cases to evaluate whether they are a real inconsistency or merely a false positive. For the analysis, we mainly used the statistics for individual relations on the treebanks pages (e.g. https://universaldependencies.org/treebanks/en_partut/en_ partut-dep-compound.html) in combination with the Grew-match tool 4 , maintained by Inria, to inspect treebank-specific examples.
We show three examples in the following, two real inconsistencies and one case in which a higher MBD is caused by both consistent and inconsistent use of a relation. The rest of our analysis is provided in the appendix. We are not familiar with all languages of which we analysed treebanks and it is possible that we missed something, making our analysis possibly incomplete. This is also the reason for why we could not analyse the potential inconsistencies for some of the languages (marked by a question mark in the table).  Table 1: Top 20 relations with largest differences in branching direction within a language. The abbreviations written above numbers are the treebank shortcuts. The last column shows whether the difference is a true (T) or false (F) inconsistency or whether the case in unclear (?).
Inconsistent Chinese classifiers (clf) Figure 1 shows that the annotation for classifiers is categorically different among the Chinese treebanks. The treebanks CFL and HK follow the UD guidelines 5 and attach the classifier to the numeral whereas the other treebanks GSD and PUD attach it to the noun. Note that the annotation of numerals is also different; however, since the branching direction stays the same this cannot be detected by our method.  Inconsistent Korean auxiliaries (aux) All three Korean treebanks have a different usage of the "aux" relation ( Figure 2), which is already documented by Noh et al. (2018): Kaist is most closely to the UD guidelines, using the "aux" relation together with the "AUX" tag in around half of its instances (but with a lot of different tags such as "NOUN" and "SCONJ" in the other half), whereas GSD generally uses "flat" and "VERB" to connect auxiliary verbs with their main verbs. In both treebanks, "aux" connects main and auxiliary verbs from left to right. PUD on the other side uses "aux" to connect main and auxiliary verbs from right to left and tags both with "VERB".  Figure 2: Korean annotation scheme for auxiliary verbs in Kaist (left), GSD (middle) and PUD (right).
In-/Consistent English compounds (compound) The use of "compound" is only partially inconsistent in English. "compound" connects common-noun compounds right-to-left, whereas "compound:prt" connects verbs and their particles left-to-right, and the ParTUT contains a greater proportion of "compound:prt" instances than the other treebanks. This difference between the treebanks is detected by our method but not an inconsistency. However, the difference is additionally increased by many proper-name compounds in ParTUT that are connected left-to-right with "compound". In the other English treebanks, similar compounds usually show a "flat" relation. Figure 3 shows examples for nominal compounds.   Parsing performance of a model trained on one treebank of a language and tested on another treebank of the same language is generally affected negatively by the domain difference, but it will suffer even more seriously in case some of the arc labels are used inconsistently across the treebanks (even though the labels are taken from the fixed UD label set). To empirically demonstrate the impact of annotation inconsistency and verify that our method could alleviate the problem, we perform a pilot experiment on classifiers in Chinese and auxiliaries in Korean, as they are the top two in Table 1.
We designed a set of rules to convert the inconsistent relations in the training treebanks towards the UD guidelines 6 . The target test sets are left as they are, including the ones that are against the general UD guideline (the PUD Treebanks for both languages), since we cannot guarantee the correctness of the automatic conversion. For Chinese classifiers and Korean auxiliaries, there are almost perfect one-to-one correspondences between the annotation schemes which allows a simple conversion of ZH_GSD/GSDSimp and KO_GSD, which is shown in Figure 4. Figure 4: Conversion rules for instances of "clf" with modifier (left) and without modifier (middle) in Chinese, and for instances of "flat" to "aux" in Korean. X, Y, Z and a are variables for any POS tag or dependency relation, respectively.
We use the UDPipe parser (Straka et al., 2016) in our experiments with the default settings. It is based on the Arc-Standard projective transition system using a neural network classifier. It uses predicted POS tags and morphological features by the tagger. Table 2 shows the Labeled Attachment Scores (LAS) on the test sets by models trained on the original and converted training sets, noted by the suffixes ".orig" and the respective converted relations. For traditional Chinese, converting the classifier relation brings 2.11 points improvements on ZH_HK, while decreases by 3.56 points on ZH_PUD, since ZH_PUD annotates the classifier similar to the original ZH_GSD. For simplified Chinese, the converted model also outperforms the original by 3.42. For Korean, PUD is improved by 1.03 and Kaist by 3.17.
We then break down the parser accuracy on each individual relation. As expected, the F1 score of "clf" in the Chinese treebanks improves from 0 to 69.96 for HK, from 0 to 84.34 for CFL, and from 65.05 to 0 for PUD (since PUD is annotated the same as the original GSD). For Korean, the F1 score of "aux" changes from 0.93 to 63.53 for Kaist, but remains 0 for PUD, since PUD annotates "aux" different from both GSD and Kaist, as explained in Section 3.
Generally, it is clear that by identifying and converting the inconsistencies, the parser model can learn to handle the relations consistent to the target treebank. It improves the parsing accuracy on the particular relations as well as the overall performance.

Conclusion and future work
In this pilot study, we show that the measure of relative branching direction applied on two UD treebanks for the same language can be used as a simple, but effective means for detecting candidates for inconsistencies in the usage of UD relation labels -independent of prior assumptions of which relations are particularly problematic. We manually inspected a list of relations with high divergence of branching direction, and as a pilot experiment converted some identified relations and analyze the performance of a parser in the out-of-domain scenario. The experiment confirms the benefit of our identification and conversion of the inconsistent annotation.
Concurrently with this work, Aggarwal and Zeman (2020) propose another method that addresses the identification of cross-treebank inconsistencies by measuring the differences of POS-tag trigrams. However, it focuses on a different type of inconsistency than what our method can identify. Both types of inconsistency can occur independently and a joined approach of the two complementary methods could make further contribution to error detection/correction in the UD treebanks.  Figure 6: English annotation scheme for clausal subjects with expletive (above) and without expletive (below).

Consistent Old Russian expletives (expl)
The only expletives in the Old Russian treebanks are forms of sebe/sja 'oneself'. The TOROT contains over 1,000 examples and in 93% of the cases the expletive follows the verb; the RNC treebank addresses those forms with the subtype relation "expl:rv" and contains only nine examples, in all but one of them the expletive precedes the verb. Furthermore, in the RNC treebank the base "expl" relation is used to connect particles (PART) with their governors and the particle usually precedes its governor.
Inconsistent Spanish compounds (compound) First of all, the PUD treebank only contains "compound:prt" whereas the GSD treebank and the AnCora treebank only contain "compound". The usage varies in several points, for example with temporal words and reflexive words:  Inconsistent Chinese compounds (compound) All Chinese treebanks use the base "compound" relation and the direction is generally from right to left. However, the HK treebank and the CFL treebank also use "compound:dir", "compound:ext", "compound:vo" and "compound:vv" where the direction is generally from left to right. The extended use of "compound" (by subtypes) in some but not all Chinese treebanks is an inconsistency.
Consistent French clausal subjects (csubj) In French, the clausal subjects show the same expletive alternation as in English (see above).  Figure 9: Galician annotation scheme for numeral modifiers in TreeGal (left) and CTG (right).
Consistent French copulae (cop) The FQB contains a lot of questions where fronted forms of quel 'which' are the governor of the copula, i.e. the fronted forms change the average direction of the "cop" relation.

Inconsistent Latin coordinating conjunctions (cc)
The conjunction is categorically attached to different conjuncts in the Latin treebanks, analogous to Galician (see above).
In-/Consistent French compounds (compound) The "compound" relation is consistently used for common-noun compounds in the PUD treebank, the Spoken treebank and the ParTUT. An inconsistency, however, which cannot be detected by dependency direction difference, is that the other French treebanks mark compounds by other relations, e.g. "nmod".
Inconsistent Portuguese compounds (compound) The Portuguese treebanks show a different use of "compound" and "compound:prt", similar to the Spanish compounds.
Consistent German compounds (compound) The German treebanks differentiate between "compound" and "compound:prt". The former is used for (hyphenated) nominal compounds and usually spans from right to left; the latter is used for verbal particles and mainly goes the opposite direction, from left to right.  Figure 10: German annotation scheme for nominal compounds (left) and particle verbs (right).