{'fraction_non_alphanumeric': 0.0313545475764413, 'fraction_numerical': 0.01322648408931767, 'mean_word_length': 4.8453842655752615, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Using the Penn Treebank to Evaluate Non-Treebank Parsers

This paper describes a method for conducting evaluations of Treebank and non-Treebank parsers alike against the English language U. Penn Treebank (Marcus et al., 1993) using a metric that focuses on the accuracy of relatively non-controversial aspects of parse structure. Our conjecture is that if we focus on maximal projections of heads (MPH), we are likely to find much broader agreement than if we try to evaluate based on order of attachment. We hope that this method may find wider acceptance and be useful in establishing a generally applicable framework for evaluation in natural language parsing. We employ this method in an evaluation of NLPWin (Heidorn, 2000), a parser developed at Microsoft Research without reference to the Penn Treebank, and, for comparison, the well-known statistical Treebank parser of Charniak (2000).


Introduction
In recent years, the literature includes compelling evaluation results for several natural language parsers that train and test on the U. Penn Treebank (Marcus et al., 1993), including the work of Magerman (1995), Collins (1999), Collins & Duffy (2002), Charniak (2000), and Klein & Manning (2003). Since the publication of the Treebank, published evaluations of non-Treebank parsers (i.e., parsers not designed with the conventions of the English language U. Penn Treebank in mind) are rarely encountered; recent exceptions include an evaluation of the Xerox LFG fstructure parser (Riezler, 2002) and the dependency parser of Lin (1995Lin ( , 1998. When such evaluations do appear, the community does not regularly cite them and appears to be at a loss as to how to compare these results with the results published for the well-known Treebank parsers. We introduce a method for conducting evaluations of both Treebank and non-Treebank parsers using a metric that focuses on the accuracy of relatively non-controversial aspects of parse structure.

Data
We are using the English language University of Pennsylvania Treebank v.3 from the LDC. The Treebank contains parse trees annotated according to the Treebank annotation guidelines (Bies et al., 1995). Included in the Treebank are two datasets of interest to us: the Wall Street Journal (WSJ) and the Brown Corpus. The former has been divided by the parsing community into a standard training set (sections 02-21), a development test set (section 24), a blind test set (section 23), and some remainder sections. Charniak (2000), Collins (1997), and others use this standard division. To date, there is no standard division of the Brown Corpus Treebank, so we have provided one for this evaluation. We will elaborate on the division of the Brown Corpus Treebank after discussing the metric.

Non-Treebank Parsers
Natural language parsers not explicitly designed or trained to follow the conventions of the Penn Treebank may differ from the Treebank in any number of ways. Differences such as tokenization, part-of-speech labels, granularity of non-terminal constituents, and nonterminal constituent labels can usually be handled by ignoring labels and resorting to the "crossing brackets" metric, one of the Parseval metrics (Black, 1991). However, irreconcilable differences often remain, particularly with respect to whether left or right modifiers should attach to heads first. In situations where a parser has been designed to make attachments in an order different from that of the Treebank, large numbers of apparent crossing-bracket errors can result.
Consider an example from NLPWin, a parser developed at Microsoft Research (Heidorn, 2000) and the Treebank. Figure 1 is a tree produced by NLPWin, where each non-terminal represents a maximal projection of a single head, and each non-punctuation terminal is the immediate daughter of a non-terminal for which it is the head. Figure 2 contains the corresponding tree for the same sentence from the Treebank. There are several differences in tokenization, part-of speech tags, non-terminal constituents, and nonterminal labels.

Maximal Projections of Heads
Faced with this lack of comparability between non-Treebank parses and Treebank parses, how can we proceed? Our conjecture is that if we focus on maximal projections of heads (MPH), we are likely to find much broader agreement than if we try to evaluate based on order of attachment or on the granularity of intermediate projections.
While there are some theoretical differences that are not normalized by looking only at MPH brackets (e.g., whether "small clauses" exist or not), we proceed under the assumption that if only MPH brackets are evaluated, the remaining systematic differences are few enough to be dealt with on a case-by-case basis. In particular, to address tokenization issues, we added a post-process to NLPWin to tokenize in the manner of the Treebank, and we exclude part-of-speech tags and constituent labels from our evaluation. Differences with regard to the granularity of constituents, as well as order of attachment issues, are handled by focusing on maximal projections of heads.
A challenge posed by focusing on maximal projections of heads is the absence of annotations of heads and head inheritance in the Treebank itself. 1 To rectify this, we will employ a set of head-labeling rules and compute maximal projections of heads for the reference or "gold" Treebank trees. We can then have a comparable reference set containing only maximal projections against which to compute bracket precision 1 We observe that if we leave the Treebank trees untouched and compute only the precision of MPH trees against the untouched Treebank, we introduce a scoring bias in favor of high PP attachment; i.e., we would give credit to false high attachments. and recall (Black, 1991), as well as crossing brackets. We elaborate on this procedure in the next section.

The Metrics and Evaluation Procedure
To compute MPH trees from the Treebank, we use a three-step process. First, we employ Charniak's headlabeling rules. There will inevitably be discrepancies between the resulting heads and the heads chosen by the parser to be evaluated. In general, one head assignment or the other produces strictly fewer MPH brackets; therefore, we always resolve such disagreements in favor of the head choice leading to fewer brackets, because the more detailed bracketing is not necessarily recoverable from the other source. We construct a small set of additional rules capturing the regularities in head reassignment. After heads are assigned, we compute the MPH brackets.
When evaluating a Treebank-style parser, we employ that parser's choice of heads, or if the parser does not produce head annotations, we can employ Charniak's rules. To facilitate a comparison with a non-Treebank parser, we then complete the annotation by applying the set of additional head reassignment rules constructed above.
It is worth noting that in computing maximal projections of heads for a given tree, no brackets are added; we keep only the brackets that enclose maximal projections of the specified heads. In fact, when changing head choices, we are simply focusing on a set of brackets that allows for a more direct comparison between a Treebank-style parser and a non-Treebank parser. 2 After computation of MPH, to avoid giving credit for trivially recoverable brackets, we ignore brackets containing only a single terminal (as is standard) as well as the top level brackets around the entire sentence. In addition, in order to avoid free credit for re-labeling non-terminals, we flatten all non-branching nonterminals. It is standard to eliminate punctuation from the hypothesis and reference trees just before evaluating. Consequently, we ignore terminals with punctuation tags by employing the appropriate options in the EvalB evaluation tool (Sekine & Collins, 1997). With these refinements, our metrics are, therefore, precision and recall of unlabeled brackets enclosing maximal projections of heads (MPH), defined as follows: Unlabeled MPH Precision # of correct constituents in proposed MPH tree = # of constituents in proposed MPH tree Unlabeled MPH Recall # of correct constituents in proposed MPH tree = # of constituents in Trebank MPH tree 2 The set of additional head reassignment rules for one non-Treebank parser is likely to be different than the rule set for another such parser. A comparison between two non-Treebank parsers would thus require constructing a set of head reassignment rules capturing the common-ground among the Charniak-head-labeled Treebank and the two parsers.
We recommend a comparison of our approach with the proposals of Gaizauskas et al. (1998aGaizauskas et al. ( , 1998b; the principal contrast lies in our employment of head labeling rules and MPH.

Comparison System
As a point of comparison with NLPWin and to provide a connection with past Treebank evaluations, we use Eugene Charniak's state of the art statistical Treebankstyle parser (Charniak, 2000). Charniak's head-driven parser produces trees with heads. Figure 3 depicts the tree from Charniak's parser for the sentence used in the example trees above. Asterisks (*) mark the heads or daughters through which heads are inherited. For the sake of a direct comparison with NLPWin, we apply the set of head reassignment rules (found as the commonground between NLPWin and the Charniak-headlabeled Treebank) and compute the MPH trees using the process defined in section 2 to arrive at the tree in Figure 4. The primary changes are in the structure of the PP "in narrowing the trade gap" and in the head for the final S constituent "… have already been made".

WSJ Treebank
We evaluated NLPWin MPH trees and MPH trees from Charniak's parser side-by-side. For each result, we include the statistics listed in the first column of Table  1, most of them computed by EvalB in the standard ways. Columns 2-3 contain the primary results on the WSJ development test set (section 24), and columns 4-5 are the blind test set (section 23) results. Every sentence is included in these results without regard to sentence length.
The number of NLPWin HYP brackets and the number of Charniak HYP brackets are quite close to one another and to the number of REF brackets, which is one source of our confidence that we are indeed evaluating the two different systems in a comparable way against comparable reference brackets.
In brief, the results are as follows: on the blind test section (section 23) of the Wall Street Journal portion of the Treebank, NLPWin achieves precision (P) of 74.32 and recall (R) of 74.25. For Charniak: P=86.18, R=85.72.

Restricted Brown Corpus Treebank
The standard WSJ sections provide a benchmark, but since Charniak's parser was trained on WSJ material and NLPWin was not, we sought a data set that was equally novel to both parsers. We are using part of the Brown Corpus section of the Treebank (hereafter "the Brown Corpus Treebank" or BCTB) to play this role. A small amount of the original Brown Corpus (Francis & Kucera, 1967) material had been used in developing NLPWin, so we removed that subset from the BCTB, calling the remainder the "Restricted BCTB". We randomized the Restricted BCTB 3 , divided it into sections of 2000 sentences, and named a development test section (sec. BR0) and a blind test section (sec. BR1). The data that had been used in development of NLPWin was provided to Charniak, and he retrained his parser incorporating the used data.
The results of evaluating both NLPWin and Charniak's parser on BR0 are in columns 6-7, and results on BR1 are in columns 8-9 of Table 1. As before, every sentence is included in these results without regard to sentence length. Briefly, for the blind test section (BR1), the two systems scored as follows: for NLPWin: P=69.54, R=70.56; for Charniak: P=77.71, R=77.73, a narrower performance gap than we observed on the WSJ. Unlike the WSJ results, the number of NLPWin HYP brackets in column 6 is somewhat larger than both the number of REF brackets in that column and the number of HYP brackets in column 7; we do not observe a regular trend responsible for this difference.

Conclusions
We have introduced a method for conducting evaluations of both Treebank and non-Treebank parsers using a metric that focuses on the accuracy of relatively non-controversial aspects of parse structure, namely unlabeled precision and recall of maximal projections of heads. We hope that this method may find wider acceptance and be useful in establishing a generally applicable framework for evaluation in natural language parsing. Note that since a simple dependency tree (with ordered dependents) is isomorphic to an MPH tree (with heads labeled), this work also opens the door to evaluating dependency parsers (such as those described in the work of Riezler and of Lin cited above) against the Treebank.