{'fraction_non_alphanumeric': 0.03637474541751527, 'fraction_numerical': 0.01584521384928717, 'mean_word_length': 4.612940100594422, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Learning part-of-speech taggers with inter-annotator agreement loss

In natural language processing (NLP) annotation projects, we use inter-annotator agreement measures and annotation guidelines to ensure consistent annotations. However, annotation guidelines often make linguistically debatable and even somewhat arbitrary decisions, and interannotator agreement is often less than perfect. While annotation projects usually specify how to deal with linguistically debatable phenomena, annotator disagreements typically still stem from these “hard” cases. This indicates that some errors are more debatable than others. In this paper, we use small samples of doublyannotated part-of-speech (POS) data for Twitter to estimate annotation reliability and show how those metrics of likely interannotator agreement can be implemented in the loss functions of POS taggers. We find that these cost-sensitive algorithms perform better across annotation projects and, more surprisingly, even on data annotated according to the same guidelines. Finally, we show that POS tagging models sensitive to inter-annotator agreement perform better on the downstream task of chunking.


Introduction
POS-annotated corpora and treebanks are collections of sentences analyzed by linguists according to some linguistic theory. The specific choice of linguistic theory has dramatic effects on downstream performance in NLP tasks that rely on syntactic features (Elming et al., 2013). Variation across annotated corpora in linguistic theory also poses challenges to intrinsic evaluation (Schwartz et al., 2011;Tsarfaty et al., 2012), as well as for languages where available resources are mutually inconsistent (Johansson, 2013). Unfortunately, there is no grand unifying linguistic theory of how to analyze the structure of sentences. While linguists agree on certain things, there is still a wide range of unresolved questions. Consider the following sentence: (1) @GaryMurphyDCU of @DemMattersIRL will take part in a panel discussion on October 10th re the aftermath of #seanref . . .
While linguists will agree that in is a preposition, and panel discussion a compound noun, they are likely to disagree whether will is heading the main verb take or vice versa. Even at a more basic level of analysis, it is not completely clear how to assign POS tags to each word in this sentence: is part a particle or a noun; is 10th a numeral or a noun?
Some linguistic controversies may be resolved by changing the vocabulary of linguistic theory, e.g., by leaving out numerals or introducing ad hoc parts of speech, e.g. for English to (Marcus et al., 1993) or words ending in -ing (Manning, 2011). However, standardized label sets have practical advantages in NLP (Zeman and Resnik, 2008;Zeman, 2010;Das and Petrov, 2011;Petrov et al., 2012;McDonald et al., 2013).
For these and other reasons, our annotators (even when they are trained linguists) often disagree on how to analyze sentences. The strategy in most previous work in NLP has been to monitor and later resolve disagreements, so that the final labels are assumed to be reliable when used as input to machine learning models.

Our approach
Instead of glossing over those annotation disagreements, we consider what happens if we embrace the uncertainty exhibited by human annotators when learning predictive models from the annotated data.
To achieve this, we incorporate the uncertainty exhibited by annotators in the training of our model. We measure inter-annotator agreement on small samples of data, then incorporate this in the loss function of a structured learner to reflect the confidence we can put in the annotations. This provides us with cost-sensitive online learning algorithms for inducing models from annotated data that take inter-annotator agreement into consideration.
Specifically, we use online structured perceptron with drop-out, which has previously been applied to POS tagging and is known to be robust across samples and domains (Søgaard, 2013a). We incorporate the inter-annotator agreement in the loss function either as inter-annotator F 1-scores or as the confusion probability between annotators (see Section 3 below for a more detailed description). We use a small amounts of doublyannotated Twitter data to estimate F 1-scores and confusion probabilities, and incorporate them during training via a modified loss function. Specifically, we use POS annotations made by two annotators on a set of 500 newly sampled tweets to estimate our agreement scores, and train models on existing Twitter data sets (described below). We evaluate the effect of our modified training by measuring intrinsic as well as downstream performance of the resulting models on two tasks, namely named entity recognition (NER) and chunking, which both use POS tags as input features.

POS-annotated Twitter data sets
The vast majority of POS-annotated resources across languages contain mostly newswire text. Some annotated Twitter data sets do exist for English. Ritter et al. (2011) present a manually annotated data set of 16 thousand tokens. They do not report inter-annotator agreement. Gimpel et al. (2011) annotated about 26 thousand tokens and report a raw agreement of 92%. Foster et al. (2011) annotated smaller portions of data for cross-domain evaluation purposes. We refer to the data as RITTER, GIMPEL and FOSTER below.
In our experiments, we use the RITTER splits provided by Derczynski et al. (2013), and the October splits of the GIMPEL data set, version 0.3. We train our models on the concatenation of RITTER-TRAIN and GIMPEL-TRAIN and evaluate them on the remaining data, the dev and test set provided by Foster et al. (2011) as well as an inhouse annotated data set of 3k tokens (see below).
The three annotation efforts (Ritter et al., 2011;Gimpel et al., 2011;Foster et al., 2011) all used different tagsets, however, and they also differ in tokenization, as well as a wide range of linguistic decisions. We mapped all the three corpora to the universal tagset provided by Petrov et al. (2012) and used the same dummy symbols for numbers, URLs, etc., in all the data sets. Following (Foster et al., 2011), we consider URLs, usernames and hashtags as NOUN. We did not change the tokenization.
The data sets differ in how they analyze many of the linguistically hard cases. Consider, for example, the analysis of will you come out to in GIM-PEL and RITTER (Figure 1, top). While Gimpel et al. (2011) tag out and to as adpositions, Ritter et al. (2011) consider them particles. What is the right analysis depends on the compositionality of the construction and the linguistic theory one subscribes to.
Other differences include the analysis of abbreviations (PRT in GIMPEL; X in RITTER and FOS-TER), colon (X in GIMPEL; punctuation in RIT-TER and FOSTER), and emoticons, which can take multiple parts of speech in GIMPEL, but are always X in RITTER, while they are absent in FOS- Since the mapping to universal tags could potentially introduce errors, we also annotated a data set directly using universal tags. We randomly selected 200 tweets collected over the span of one day, and had three annotators tag this set. We split the data in such a way that each annotator had 100 tweets: two annotators had disjoint sets, the third overlapped 50 items with each of the two others. In this way, we obtained an initial set of 100 doubly-annotated tweets. The annotators were not provided with annotation guidelines. After the first round of annotations, we achieved a raw agreement of 0.9, a Cohen's κ of 0.87, and a Krippendorff's α of 0.87. We did one pass over the data to adjudicate the cases where annotators disagreed, 3 Computing agreement scores Gimpel et al. (2011) used 72 doubly-annotated tweets to estimate inter-annotator agreement, and we also use doubly-annotated data to compute agreement scores. We randomly sampled 500 tweets for this purpose. Each tweet was annotated by two annotators, again using the universal tag set (Petrov et al., 2012). All annotators were encouraged to use their own best judgment rather than following guidelines or discussing difficult cases with each other. This is in contrast to Gimpel et al. (2011), who used annotation guidelines. The average inter-annotator agreement was 0.88 for raw agreement, and 0.84 for Cohen's κ. Gimpel et al. (2011) report a raw agreement of 0.92. We use two metrics to provide a more detailed picture of inter-annotator agreement, namely F 1-scores between annotators on individual parts of speech, and tag confusion probabilities, which we derive from confusion matrices.
The F 1-score relates to precision and recall in the usual way, i.e, as the harmonic mean between those two measure. In more detail, given two annotators A 1 and A 2 , we say the precision Figure 2: Inter-annotator F 1-scores estimated from 500 tweets.
of A 1 relative to A 2 with respect to POS tag T in some data set X, denoted Prec T (A 1 (X), A 2 (X)), is the number of tokens both A 1 and A 2 predict to be T over the number of times A 1 predicts a token to be T . Similarly, we define the recall with respect to some tag T , i.e., Rec T (A 1 (X), A 2 (X)), as the number of tokens both A 1 and A 2 predict to be T over the number of times A 2 predicts a token to be T . The only difference with respect to standard precision and recall is that the gold standard is replaced by a second annotator, A 2 . Note that Prec T (A 1 (X), A 2 (X)) = Rec T (A 2 (X), A 1 (X)). It follows from all of the above that the F 1-score is symmetrical, i.e., The inter-annotator F 1-scores over the 12 POS tags in the universal tagset are presented in Figure 2. It shows that there is a high agreement for nouns, verbs and punctuation, while the agree- ment is low, for instance, for particles, numerals and the X tag.
We compute tag confusion probabilities from a confusion matrix over POS tags like the one in Figure 3. From such a matrix, we compute the probability of confusing two tags t 1 and t 2 for some data point x, i.e. P ({A 1 (x), A 2 (x)} = {t 1 , t 2 }) as the mean of P (A 1 (x) = t 1 , A 2 (x) = t 2 ) and P (A 1 (x) = t 2 , A 2 (x) = t 1 ), e.g., the confusion probability of two tags is the mean of the probability that annotator A 1 assigns one tag and A 2 another, and vice versa.
We experiment with both agreement scores (F 1 and confusion matrix probabilities) to augment the loss function in our learner. The next section describes this modification in detail.

Inter-annotator agreement loss
We briefly introduce the cost-sensitive perceptron classifier. Consider the weighted perceptron loss on our ith example x i , y i (with learning rate α = 1), L w ( x i , y i ): In a non-cost-sensitive classifier, the weight function γ(y j , y i ) = 1 for 1 ≤ i ≤ N . The end for 10: end for 11: return w * / = (N × I) two cost-sensitive systems proposed only differ in how we formulate γ(·, ·). In one model, the loss is weighted by the inter-annotator F 1 of the gold tag in question. This boils down to where X is the small sample of held-out data used to estimate inter-annotator agreement. Note that in this formulation, the predicted label is not taken into consideration.
The second model is slightly more expressive and takes both the gold and predicted tags into account. It basically weights the loss by how likely the gold and predicted tag are to be mistaken for each other, i.e., (the inverse of) their confusion probability: γ(y j , y i )) = 1 − P ({A 1 (X), A 2 (X)} = {y j , y i }) In both loss functions, a lower gamma value means that the tags are more likely to be confused by a pair of annotators. In this case, the update is smaller. In contrast, the learner incurs greater loss when easy tags are confused.
It is straight-forward to extend these costsensitive loss functions to the structured perceptron (Collins, 2002). In Figure 4, we provide the pseudocode for the cost-sensitive structured online learning algorithm. We refer to the cost-sensitive structured learners as F 1-and CM-weighted below.

Experiments
In our main experiments, we use structured perceptron (Collins, 2002) with random corruptions using a drop-out rate of 0.1 for regularization, following Søgaard (2013a). We use the LXMLS toolkit implementation 1 with default parameters. We present learning curves across iterations, and only set parameters using held-out data for our downstream experiments. 2

Results
Our results are presented in Figure 5. The top left graph plots accuracy on the training data per iteration. We see that CM-weighting does not hurt training data accuracy. The reason may be that the cost-sensitive learner does not try (as hard) to optimize performance on inconsistent annotations. The next two plots (upper mid and upper right) show accuracy over epochs on in-sample evaluation data, i.e., GIMPEL-DEV and RITTER-TEST. Again, the CM-weighted learner performs better than our baseline model, while the F 1-weighted learner performs much worse.
The interesting results are the evaluations on out-of-sample evaluation data sets (FOSTER and IN-HOUSE) -lower part of Figure 5. Here, both our learners are competitive, but overall it is clear that the CM-weighted learner performs best. It consistently improves over the baseline and F 1weighting. The former is much more expressive as it takes confusion probabilities into account and does not only update based on gold-label uncertainty, as is the case with the F 1-weighted learner.

Robustness across regularizers
Discriminative learning typically benefits from regularization to prevent overfitting. The simplest is the averaged perceptron, but various other methods have been suggested in the literature.
We use structured perceptron with drop-out, but results are relatively robust across other regularization methods. Drop-out works by randomly dropping a fraction of the active features in each iteration, thus preventing overfitting. Table 1 shows the results for using different regularizers, in particular, Zipfian corruptions (Søgaard, 2013b) and averaging. While there are minor differences across data sets and regularizers, we observe that the corresponding cell using the loss function suggested in this paper (CM) always performs better than the baseline method.

Downstream evaluation
We have seen that our POS tagging model improves over the baseline model on three out-ofsample test sets. The question remains whether training a POS tagger that takes inter-annotator agreement scores into consideration is also effective on downstream tasks. Therefore, we evaluate our best model, the CM-weighted learner, in two downstream tasks: shallow parsing-also known as chunking-and named entity recognition (NER).
For the downstream evaluation, we used the baseline and CM models trained over 13 epochs, as they performed best on FOSTER-DEV (cf. Figure 5). Thus, parameters were optimized only on POS tagging data, not on the downstream evaluation tasks. We use a publicly available implementation of conditional random fields (Lafferty et al., 2001) 3 for the chunking and NER experiments, and provide the POS tags from our CM learner as features.

Chunking
The set of features for chunking include information from tokens and POS tags, following Sha and Pereira (2003).
We train the chunker on Twitter data (Ritter et al., 2011), more specifically, the 70/30 train/test split provided by Derczynski et al. (2013) for POS tagging, as the original authors performed cross validation. We train on the 70% Twitter data (11k tokens) and evaluate on the remaining 30%, as well as on the test data from Foster et al. (2011). The FOSTER data was originally annotated for POS and constituency tree information. We converted it to chunks using publicly available conversion software. 4 Part-of-speech tags are the ones assigned by our cost-sensitive (CM) POS model trained on Twitter data, the concatenation of Gimpel and 70% Ritter training data. We did not include the CoNLL 2000 training data (newswire text), since adding it did not substantially improve chunking performance on tweets, as also shown in (Ritter et al., 2011).
The results for chunking are given in Table 2. They show that using the POS tagging model (CM) trained to be more sensitive to interannotator agreement improves performance over  the baseline (BL) for the downstream task of chunking. Overall chunking F1 score improves.
More importantly, we report on individual scores for NP, VP and PP chunks, where we see consistent improvements for NPs and VPs (since both nouns and verbs have high inter-annotator agreement), while results on PP are mixed. This is to be expected, since PP phrases involve adpositionals (ADP) that are often confused with particles (PRT), cf. Figure 3. Our tagger has been trained to deliberately abstract away from such uncertain cases. The results show that taking uncertainty in POS annotations into consideration during training has a positive effect in downstream results. It is thus better if we do not try to urge our models to make a firm decision on phenomena that neither linguistic theories nor annotators do agree upon.

NER
In the previous section, we saw positive effects of cost-sensitive POS tagging for chunking, and here we evaluate it on another downstream task, NER. For the named entity recognition setup, we use commonly used features, in particular features for word tokens, orthographic features like the presence of hyphens, digits, single quotes, upper/lowercase, 3 character prefix and suffix information. Moreover, we add Brown word cluster features that use 2,4,6,8,..,16 bitstring prefixes estimated from a large Twitter corpus (Owoputi et al., 2013). 5 For NER, we do not have access to carefully annotated Twitter data for training, but rely on the crowdsourced annotations described in Finin et al. (2010). We use the concatenation of the CoNLL 2003 training split of annotated data from the Reuters corpus and the Finin data for training, as in this case training on the union resulted in a model that is substantially better than training on any of the individual data sets. For evaluation, we have three Twitter data set. We use the recently published data set from the MSM 2013 challenge (29k tokens) 6 , the data set of Ritter et al. (2011) used also by Fromheide et al. (2014) (46k tokens), as well as an in-house annotated data set (20k tokens) (Fromheide et al., 2014 Table 3 shows the result of using our POS models in downstream NER evaluation. Here we observe mixed results. The cost-sensitive model is able to improve performance on two out of the three test sets, while being slightly below baseline performance on the MSM challenge data. Note that in contrast to chunking, POS tags are just one of the many features used for NER (albeit an important one), which might be part of the reason why the picture looks slightly different from what we observed above on chunking.

Related work
Cost-sensitive learning takes costs, such as misclassification cost, into consideration. That is, each instance that is not classified correctly during the learning process may contribute differently to the overall error. Geibel and Wysotzki (2003) introduce instance-dependent cost values for the perceptron algorithm and apply it to a set of binary classification problems. We focus here on structured problems and propose cost-sensitive learning for POS tagging using the structured perceptron algorithm. In a similar spirit, Higashiyama et al. (2013) applied cost-sensitive learning to the structured perceptron for an entity recognition task in the medical domain. They consider the distance between the predicted and true label sequence smoothed by a parameter that they estimate on a development set. This means that the entire sequence is scored at once, while we update on a per-label basis.
The work most related to ours is the recent study of Song et al. (2012). They suggest that some errors made by a POS tagger are more serious than others, especially for downstream tasks. They devise a hierarchy of POS tags for the Penn treebank tag set (e.g. the class NOUN contains NN, NNS, NNP, NNPS and CD) and use that in an SVM learner. They modify the Hinge loss that can take on three values: 0, σ, 1. If an error occurred and the predicted tag is in the same class as the gold tag, a loss σ occurred, otherwise it counts as full cost. In contrast to our approach, they let the learner focus on the more difficult cases by occurring a bigger loss when the predicted POS tag is in a different category. Their approach is thus suitable for a fine-grained tagging scheme and requires tuning of the cost parameter σ. We tackle the problem from a different angle by letting the learner abstract away from difficult, inconsistent cases as estimated from inter-annotator scores.
Our approach is also related to the literature on regularization, since our cost-sensitive loss functions are aimed at preventing over-fitting to low-confidence annotations. Søgaard (2013b;2013a) presented two theories of linguistic variation and perceptron learning algorithms that regularize models to minimize loss under expected variation. Our work is related, but models variations in annotation rather than variations in input.
There is a large literature related to the issue of learning from annotator bias. Reidsma and op den Akker (2008) show that differences between annotators are not random slips of attention but rather different biases annotators might have, i.e. different mental conceptions. They show that a classifier trained on data from one annotator performed much better on in-sample (same annotator) data than on data of any other annotator. They propose two ways to address this problem: i) to identify subsets of the data that show higher inter-annotator agreement and use only that for training (e.g. for speaker address identification they restrict the data to instances where at least one person is in the focus of attention); ii) if available, to train separate models on data annotated by different annotators and combine them through voting. The latter comes at the cost of recall, because they deliberately chose the classifier to abstain in nonconsensus cases.
In a similar vein,  divide the instance space into easy and hard cases, i.e. easy cases are reliably annotated, whereas items that are hard show confusion and disagreement. Hard cases are assumed to be annotated by individual annotator's coin-flips, and thus cannot be assumed to be uniformly distributed . They show that learning with annotator noise can have deteriorating effect at test time, and thus propose to remove hard cases, both at test time  and training time ).
In general, it is important to analyze the data and check for label biases, as a machine learner is greatly affected by annotator noise that is not ran-dom but systematic (Reidsma and Carletta, 2008). However, rather than training on subsets of data or training separate models -which all implicitly assume that there is a large amount of training data available -we propose to integrate inter-annotator biases directly into the loss function.
Regarding measurements for agreements, several scores have been suggested in the literature. Apart from the simple agreement measure, which records how often annotators choose the same value for an item, there are several statistics that qualify this measure by adjusting for other factors, such as Cohen's κ (Cohen and others, 1960), the G-index score (Holley and Guilford, 1964), or Krippendorff's α (Krippendorf, 2004). However, most of these scores are sensitive to the label distribution, missing values, and other circumstances. The measure used in this paper is less affected by these factors, but manages to give us a good understanding of the agreement.

Conclusion
In NLP, we use a variety of measures to assess and control annotator disagreement to produce homogenous final annotations. This masks the fact that some annotations are more reliable than others, and which is thus not reflected in learned predictors. We incorporate the annotator uncertainty on certain labels by measuring annotator agreement and use it in the modified loss function of a structured perceptron. We show that this approach works well independent of regularization, both on in-sample and out-of-sample data. Moreover, when evaluating the models trained with our loss function on downstream tasks, we observe improvements on two different tasks. Our results suggest that we need to pay more attention to annotator confidence when training predictors.