{'fraction_non_alphanumeric': 0.02614999240929103, 'fraction_numerical': 0.005503264004858054, 'mean_word_length': 5.02400548696845, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Why Nitpicking Works: Evidence for Occam's Razor in Error Correctors

Empirical experience and observations have shown us when powerful and highly tunable classifiers such as maximum entropy classifiers, boosting and SVMs are applied to language processing tasks, it is possible to achieve high accuracies, but eventually their performances all tend to plateau out at around the same point. To further improve performance, various error correction mechanisms have been developed, but in practice, most of them cannot be relied on to predictably improve performance on unseen data; indeed, depending upon the test set, they are as likely to degrade accuracy as to improve it. This problem is especially severe if the base classifier has already been finely tuned.In recent work, we introduced N-fold Templated Piped Correction, or NTPC ("nitpick"), an intriguing error corrector that is designed to work in these extreme operating conditions. Despite its simplicity, it consistently and robustly improves the accuracy of existing highly accurate base models. This paper investigates some of the more surprising claims made by NTPC, and presents experiments supporting an Occam's Razor argument that more complex models are damaging or unnecessary in practice.


Introduction
The investigation we describe here arose from a very commonly discussed experience, apparently triggered by the recent popularity of shared task evaluations that have opened opportunities for researchers to informally compare their experiences "with a common denominator", so to speak.
Among the perennial observations which are made during the analysis of the results is that (1) methods designed to "fine-tune" the high-accuracy base classifiers behave unpredictably, their success or failure often appearing far more sensitive to where the test set was drawn from, rather than on any true quality of the "fine-tuning", and consequently, (2) the resulting system rankings are often unpredictable, especially as they are typically conducted only on a single new test set, often drawn from a single arbitrary new source of a significantly different nature than the training sets. One could argue that such evaluations do not constitute a fair test, but in fact, this is where computational linguistics modeling diverges from machine learning theory, since for any serious NLP application, such evaluations constitute a much more accurate representation of the real world.
We believe one primary reason for this common experience is that the models involved are typically already operating well beyond the limits of accuracy of the models' assumptions about the nature of distributions from which testing samples will be drawn. For this reason, even "sophisticated" discriminative training criteria, such as maximum entropy, minimum error rate, and minimum Bayes risk, are susceptible to these stability problems. There has been much theoretical work done on error correction, but in practice, any error correction usually lowers the performance of the combined system on unseen data, rather than improving it. Unfortunately, most existing theory simply does not apply. This is especially true if the base model has been highly tuned. For the majority of tasks, the performance of the trained models, after much fine tuning, tend to plateau out at around the same point, regardless of the theoretical basis of the underlying model. This holds true with most highly accurate classifiers, including maximum entropy classifiers, SVMs, and boosting models. In addition, even though data analysis gives us some general idea as to what kinds of feature conjunctions might help, the classifiers are not able to incorporate those into their model (usually because the computational cost would be infeasible), and any further post-processing tends to degrade accuracy on unseen data. The common practice of further improving accuracy at this point is to resort to ad hoc classifier combination methods, which are usually not theoretically well justified and, again, unpredictably improve or degrade performance-thus consuming vast amounts of experimental resources with relatively low expected payoff, much like a lottery.
There are a variety of reasons for this, ranging from the aforementioned validity of the assumptions about the distribution between the training and test corpora, to the absence of a well justified stopping point for error correction. The latter problem is much more serious than it seems at first blush, since without a well-justified stopping criterion, the performance of the combined model will be much more dependent upon the distribution of the test set, than on any feature engineering. Empirical evidence for this argument can be seen from the result of the CoNLL shared tasks (Tjong Kim Sang, 2002)(Tjong Kim Sang and Meulder, 2003), where the ranking of the participating systems changes with the test corpora.
Inspired by the repeated observations of this phenomenon by many participants, we decided to stop "sweeping the issue under the rug", and undertook to confront it head-on. Accordingly, we challenged ourselves to design an error corrector satisfying the following criteria, which few if any existing models actually meet: (1) it would leverage off existing base models, while targeting their errors; (2) it would consistently improve accuracy, even on top of base models that already deliver high accuracy; (3) it would be robust and conservative, so as to almost never accidentally degrade accuracy; (4) it would be broadly applicable to any classification or recognition task, especially high-dimensional ones such as named-entity recognition and word-sense disambiguation; and (5) it would be template-driven and easily customizable, which would enable it to target error patterns beyond the base models' representation and computational complexity limitations.
Our goal in this undertaking was to invent as little as possible. We expected to make use of relatively sophisticated error-minimization techniques. Thus the results were surprising: the simplest models kept outperforming the "sophisticated" models. This paper attempts to investigate some of the key reasons why.
To avoid reinventing the wheel, we originally considered adapting an existing error-driven method, transformation-based learning (TBL) for this purpose. TBL seems well suited to the problem as it is inherently an error corrector and, on its own, has been shown to achieve high accuracies on a variety of problems (see Section 4). Our original goal was to adapt TBL for error correction of high-performing models (Wu et al., 2004a), with two main principles: (1) since it is not clear that the usual assumptions made about the distribution of the training/test data are valid in such extreme operating ranges, empirical observations would take precedence over theoretical models, which implies that (2) any model would have to be empirically justified by testing on a diverse range of data. Experimental observations, however, increasingly drove us toward different goals.
Our resulting error corrector, NTPC, was instead constructed on the principle of making as few assumptions as possible in order to robustly generalize over diverse situations and problems. One observation made in the course of experimentation, after many attempts at fine-tuning model parameters, was that many of the complex theoretical models for error correction often do not perform consistently. This is perhaps not too surprising upon further reflection, since the principle of Occam's Razor does prefer simpler hypotheses over more complex ones.
NTPC was introduced in (Wu et al., 2004b), where the controversial issues it raised generated a number of interesting questions, many of which were were directed at NTPC's seeming simplicity, which seems in opposition to the theory behind many other error correcting models. In this paper, we investigate the most commonly-asked questions. We illuminate these questions by contrasting NTPC against the more powerful TBL, presenting experiments that show that NTPC's simple model is indeed key to its robustness and reliability.
The rest of the paper is laid out as follows: Section 2 presents an introduction to NTPC, including an overview of its architecture. Section 3 addresses key questions related to NTPC's architecture and presents empirical results justifying its simplicity.

N-fold Templated Piped Correction
N-fold Templated Piped Correction or NTPC, is a model that is designed to robustly improve the accuracy of existing base models in a diverse range of operating conditions. As was described above, the most challenging situations for any error corrector is when the base model has been finely tuned and the performance has reached a plateau. Most of the time, any further feature engineering or error correction after that point will end up hurting performance rather than improving it.

The architecture of NTPC is surprisingly simple
One of the most surprising things about NTPC lies in the fact that despite its simplicity, it outperforms mathematically much more "sophisticated" methods at error correcting. Architecturally, it relies on a simple rulelearning mechanism and cross-partitioning of the training data to learn very conservative, cautious rules that make only a few corrections at a time. Figure 1 illustrates the NTPC architecture. Prior to learning, NTPC is given (1) a set of rule templates which describe the types of rules that it is allowed to hypothe-size, (2) a single base learning model, and (3) an annotated training set.
The NTPC architecture is essentially a sequentially chained piped ensemble that incorporates crossvalidation style n-fold partition sets generated from the base model. The training set is partitioned n times in order to train n base models. Subsequently the n heldout validation sets are classified by the respective trained base models, with the results combined into a "reconstituted" training set. The reconstituted training set is used by Error Corrector Learner, which learns a set of rules. Rule hypotheses are generated according to the given set of allowable templates: is a sequence of reference labels y i for each example respectively,Ŷ is a sequence of labelsŷ i as predicted by the base model for each example respectively, H is the hypothesis space of valid rules implied by the templates, and τ min is a confidence threshold. Setting τ min to a relatively high value (say 15) implements the requirement of high reliability. R is subsequently sorted by the τ i value of each rule r i into an ordered list of rules R * = (r * 0 , . . . , r * i−1 ). During the evaluation phase, depicted in the lower portion of Figure 1, the test set is first labeled by the base model. The error corrector's rules r * i are then applied in the order of R * to the evaluation set. The final classification of a sample is then the classification attained when all the rules have been applied.

NTPC consistently and robustly improves
accuracy of highly-accurate base models In previous work (Wu et al., 2004b), we presented experiments on named-entity identification and classification across four diverse languages, using Adaboost.MH as the base learner, which showed that NTPC was capable of robustly and consistently improving upon the accuracy of the already-highly-accurate boosting model; correcting the errors committed by the base model but not introducing any of its own. Table 1 compares results obtained with the base Adaboost.MH model (Schapire and Singer, 2000) and the NTPC-enhanced model for a total of eight different named-entity recognition (NER) models. These experiments were performed on the CoNLL-2002 and CoNLL-2003 shared task data sets. It can be seen that the Adaboost.MH base models clearly already achieve high accuracy, setting the bar very high for NTPC to improve upon. However, it can also be seen that NTPC yields further F-Measure gains on every combination of task and language, including English NE bracketing (Model M2) for which the base F-Measure is the highest.
An examination of the rules (shown in the Appendix) can give an idea as to why NTPC manages to identify and correct errors which were overlooked by the highly tuned base model. NTPC's advantage comes from two aspects: (1) its ability to handle complex conjunctions of features, which often reflect structured, linguistically motivated expectations, in the form of rule templates; and (2) its ability to "look forward" at classifications from the right context, even when processing the sentence in a left-to-right direction. The base classifier is unable to incorporate these two aspects, because (1) including complex conjunctions of features would raise the computational cost of searching the feature space to a point where it would be infeasible, and (2) most classifiers process a sentence from left-to-right, deciding on the class label for each word before moving on to the next one. Rules that exploit these advantages are easily picked out in the table; many of the rules (especially those in the top 5 for both English and Spanish) consist of complex conjunctions of features; and rules that consider the right context classifications can be identified by the string "ne <num>", where <num> is a positive integer (indicating how many words to the right).

Experiments
The most commonly-raised issues about NTPC relate to the differences between NTPC and TBL (though the conceptual issues are much the same as for other errorminimization criteria, such as minimum error rate or minimum Bayes risk). This is expected, since it was one of our goals to reinvent as little as possible. As a result, NTPC does bear a superficial resemblance to TBL, both of them being error-driven learning methods that seek to incrementally correct errors in a corpus by learning rules that are determined by a set of templates. One of the most frequently asked questions is whether the Error Corrector Learner portion of NTPC could be replaced by a transformation-based learner. This section will investigate the differences between NTPC and TBL, and show the necessity of the changes that were incorporated into NTPC.
The experiments run in this section were performed on the data sets used in the CoNLL-2002 andCoNLL-2003 Named Entity Recognition shared tasks. The high-performing base model is based on AdaBoost.MH (Schapire and Singer, 2000), the multi-class generalization of the original boosting algorithm, which implements boosting on top of decision stump classifiers (decision trees of depth one).

Any Error is Bad
The first main difference between NTPC and TBL, and also what seems to be an extreme design decision on the part of NTPC, is the objective scoring function. To be maximally certain of not introducing any new errors with its rules, the first requirement that NTPC's objective function places onto any candidate rules is that they must not introduce any new errors ( (r) = 0). This is called the zero error tolerance principle.
To those who are used to learners such as transformation-based learning and decision lists, which allow for some degree of error tolerance, this design principle seems overly harsh and inflexible. Indeed, for al- most all models, there is an implicit assumption that the scoring function will be based on the difference between the positive and negative applications, rather than on an absolute number of corrections or mistakes. Results for eight experiments are shown in Figures 2  and 3. Each experiment compares NTPC against other variants that allow relaxed (r) ≤ max conditions for various max ∈ {1, 2, 3, 4, ∞}. The worst curve in each case is for max = ∞ -in other words, the system that only considers net performance improvement, as TBL and many other rule-based models do. The results confirm empirically that the (r) = 0 condition (1) gives the most consistent results, and (2) generally yields accuracies among the highest, regardless of how long training is allowed to continue. In other words, the presence of any negative application during the training phase will cause the error corrector to behave unpredictably, and the more complex model of greater error tolerance is unnecessary in practice.

Rule Interaction is Unreliable
Another key difference between NTPC and TBL is the process of rule interaction. Since TBL allows a rule to use the current classification of a sample and its neighbours as features, and a rule updates the current state of the corpus when it applies to a sample, the application of one rule could end up changing the applicability (or not) of another rule. From the point of view of a sample, its classification could depend on the classification of "nearby" samples. Typically, these "nearby" samples are those found in the immediately preceding or succeeding words of the same sentence. This rule interaction is permitted in both training and testing.
NTPC, however, does not allow for this kind of rule interaction. Rule applications only update the output classification of a sample, and do not update the current state of the corpus. In other words, the feature values for a sample are initialized once, at the beginning of the program, and not changed again thereafter. The rationale for making this decision is the hypothesis that rule interaction is in nature unreliable, since the high-accuracy base model provides sparse opportunities for rule application and thus much sparser opportunities for rule interaction, making any rule that relies on rule interaction suspect. As a matter of fact, by considering only rules that make no mistake during the learning phase, NTPC's zero error tolerance already eliminates any correction of labels that results from rule interaction-since a label correction on a sample that results from the application of more than one rule necessarily implies that at least one of the rules made a mistake.
Since TBL is a widely used error-correcting method, it is natural to speculate that NTPC's omission of rule interaction is a weakness. In order to test this question, we implemented an iterative variation of NTPC that allows rule interaction, where each iteration targets the residual error from previous iterations as follows: Here, incremental rule interaction is a natural consequence of arranging the structure of the algorithm to observe the right context features coming from the base model, as with transformation-based learning. In Step 5 of the algorithm, the current state of the corpus is updated with the latest rule on each iteration. That is, in each given iteration of the outer loop, the learner considers the corrected training data obtained by applying rules learned in the previous iterations, so the learner has access to the labels that result from applying the previous rules. Since these rules may apply anywhere in the corpus, the learner is not restricted to using only labels from the left context.
The time complexity of this variation is an order of magnitude more expensive than NTPC, due to the need to allow rule interaction using nested loops. The ordered list of output rules r * 0 , . . . , r * i−1 is learned in a greedy fashion, to progressively improve upon the performance of the learning algorithm on the training set.
Results for eight experiments on this variation, shown in Figures 4 and 5, demonstrate that this expensive extra capability is rarely useful in practice and does not reliably guarantee that accuracy will not be degraded. This is yet another illustration of the principle that, in highaccuracy error correction problems, at least, more simple modes of operation should be preferred over more complex arrangements.

NTPC vs. N-fold TBL
Another question on NTPC that is frequently raised is whether or not ordinary TBL, which is after all, intrinsically an error-correcting model, can be used in place of NTPC to perform better error correction. Figure 6 shows the results of four sets of experiments evaluating this approach on top of boosting. As might be expected from extrapolation from the foregoing experiments that investigated their individual differences, NTPC outperforms the more complex TBL in all cases, regardless of how long training is allowed to continue.

NTPC vs. Partition-Based Voting
Another valid question would be to ask if the way that NTPC combines the results of the n-fold partitioning is oversimplistic and could be improved upon. As was previously stated, the training corpus for the error corrector in NTPC is the "reconstituted training set" generated by combining the held-out validation sets after they have labeled with initial classifications by their respective trained base models. To investigate if NTPC could benefit from a more complex model, we employed voting, a commonly-used technique in machine learning and natural language processing. As before, the training set was partitioned and multiple base learners were trained and evaluated on the multiple training and validation sets, respectively. However, instead of recombining the validation sets into a reconstituted training set, multiple error corrector models were trained on the n partition sets. During the evaluation phase, all n error correctors were evaluated on the evaluation set after it had been labeled by the base model, and they voted on the final output. Table 2 shows the results of using such an approach for the bracketing + classification task on English. The empirical results clearly show that the more complex and time-consuming voting model not only does not outperfom NTPC, but in fact again degrades the performance from the base boosting-only model.

Experiment Summary
In our experiments, we set out to investigate whether NTPC's operating parameters were overly simple, and whether more complex arrangements were necessary or desirable. However, empirical evidence points to the fact that, in this problem of error correction in high accuracy ranges, at least, simple mechanisms will suffice to produce good results-in fact, the more complex operations end up degrading rather than improving accuracy.
A valid question is to ask why methods such as decision list learning (Rivest, 1987) as well as transformation-based learning benefit from these more complex mechanisms. Though structurally similar to NTPC, these models operate in a very different environment, where many initially poorly labeled examples are available to drive rule learning with. Hence, it is possibly advantageous to trade off some corrections with some mistakes, provided that there is an overall positive change in accuracy. However, in an error-correcting situation, most of the samples are already correctly labeled, errors are few and far in between and the sparse data problem is exacerbated. In addition, the idea of error correction implies that we should, at the very least, not do any worse than the original algorithm, and hence it makes sense to err on the side of caution and minimize any errors created, rather than hoping that a later rule application will undo mistakes made by an earlier one.
Finally, note that the same point applies to many other models where training criteria like minimum error rate are used, since such criteria are functions of the tradeoff between correctly and incorrectly labeled examples, without zero error tolerance to compensate for the sparse data problem.

Boosting and NER
Boosting (Freund and Schapire, 1997) has been successfully applied to several NLP problems. In these NLP systems boosting is typically used as the ultimate stage in a learned system. For example, Shapire and Singer (2000) applied it to Text Categorization while Escudero et al.(2000) used it to obtain good results on Word Sense Disambiguation. More closely relevant to the experiments described here in, two of the best-performing three teams in the CoNLL-2002 Named Entity Recognition shared task evaluation used boosting as their base system (Carreras et al., 2002) (Wu et al., 2002).
However, precedents for improving performance after boosting are few. At the CoNLL-2002 shared task session, Tjong Kim Sang (unpublished) described an experiment using voting to combine the NER outputs from the shared task participants which, predictably, produced better results than the individual systems. A couple of the individual systems were boosting models, so in some sense this could be regarded as an example. Tsukamoto et al.(2002) used piped AdaBoost.MH models for NER. Their experimental results were somewhat disappointing, but this could perhaps be attributable to various reasons including the feature engineering or not using cross-validation sampling in the stacking.
The AdaBoost.MH base model's high accuracy sets a high bar for error correction. Aside from brute-force en masse voting of the sort at CoNLL-2002 described above, we do not know of any existing post-boosting models that improve rather than degrade accuracy. We aim to further improve performance, and propose using a piped error corrector.

Transformation-based Learning
Transformation-based learning (Brill, 1995), or TBL, is one of the most successful rule-based machine learning algorithms. The central idea of TBL is to learn an ordered list of rules, each of which evaluates on the results of those preceding it. An initial assignment is made based on simple statistics, and then rules are greedily learned to correct the mistakes, until no net improvement can be made.
Transformation-based learning has been used to tackle a wide range of NLP problems, ranging from part-ofspeech tagging (Brill, 1995) to parsing (Brill, 1996) to segmentation and message understanding (Day et al., 1997). In general, it achieves state-of-the-art performances and is fairly resistant to overtraining.

Conclusion
We have investigated frequently raised questions about N-fold Templated Piped Correction (NTPC), a generalpurpose, conservative error correcting model, which has been shown to reliably deliver small but consistent gains on the accuracy of even high-performing base models on high-dimensional NLP tasks, with little risk of accidental degradation. Experimental evidence shows that when error-correcting high-accuracy base models, simple models and hypotheses are more beneficial than complex ones, while the more complex and powerful models are surprisingly unreliable or damaging in practice.