{'fraction_non_alphanumeric': 0.02968832941533328, 'fraction_numerical': 0.011867748540229013, 'mean_word_length': 4.9753058450385135, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
The Hard-CoRe Coreference Corpus: Removing Gender and Number Cues for Difficult Pronominal Anaphora Resolution

We introduce a new benchmark task for coreference resolution, Hard-CoRe, that targets common-sense reasoning and world knowledge. Previous coreference resolution tasks have been overly vulnerable to systems that simply exploit the number and gender of the antecedents, or have been handcrafted and do not reflect the diversity of sentences in naturally occurring text. With these limitations in mind, we present a resolution task that is both challenging and realistic. We demonstrate that various coreference systems, whether rule-based, feature-rich, graphical, or neural-based, perform at random or slightly above-random on the task, whereas human performance is very strong with high inter-annotator agreement. To explain this performance gap, we show empirically that state-of-the art models often fail to capture context and rely only on the antecedents to make a decision.


Introduction
Coreference resolution is one of the best known tasks in the Natural Language Processing (NLP) community. Despite a large body of work in the area throughout the last few decades (Morton 2000; Bean and Riloff 2004;McCallum and Wellner 2005;Rahman and Ng 2009), the task remains challenging, as many resolution decisions require extensive world knowledge and understanding common points of reference (Pradhan et al. 2011). In the case of pronominal anaphora resolution, these requirements may become more important when cues such as gender or plurality do not by themselves indicate the correct resolution (Levesque, Davis, and Morgenstern 2011).
To date, most existing methods for coreference resolution (Raghunathan et al. 2010;Durrett, Hall, and Klein 2013;Lee et al. 2017) have been evaluated on a few popular datasets, including the CoNLL 2011 and 2012 shared coreference resolution tasks (Pradhan et al. 2011;Pradhan et al. 2012). These datasets were initially proposed as the first comprehensively tagged and large-scale corpora for coreference resolution, in the hope that they would contribute to improvements in state-of-the-art techniques. These improvements would ideally yield systems that rely on more than just shallow semantic features like *Equal contribution. number, gender, and semantic class, and more on world knowledge and the context of the mention. According to , these systems would contribute in the "uphill battle" to modelling not just syntax and discourse, but also semantic compatibility.
Despite the general success of these tasks, the question of what exactly current systems learn or exploit remains open, particularly with recent neural coreference resolution models that achieve high performance. For example, (Lee et al. 2017) astutely note that their state-of-theart model does "little in the uphill battle of making coreference decisions require world knowledge", listing a few examples in the CoNLL 2012 task that require external sources of knowledge with more complex inference. However, since these cases were infrequent in the tasks themselves, the best systems may still exploit the same cues whose avoidance originally inspired the CoNLL tasks. In addition, these systems have been observed to rely on societal stereotypes present in coreference datasets, which could significantly impact their performance for some demographics (Zhao et al. 2018).
There has been a recent trend to develop tasks that feature more challenging coreference problems. The most popular of these is the Winograd Schema Challenge (WSC), which has emerged as a popular alternative to the Turing test as a means to measure progress towards human-like artificial intelligence (Levesque, Davis, and Morgenstern 2011).
The WSC task is carefully controlled such that heuristics involving syntactic salience, the number and gender of the antecedent, or other simple syntactic and semantic cues become ineffective. Previous approaches to common sense reasoning, for instance based on logical formalisms (Bailey et al. 2015) or deep neural models (Liu et al. 2016), have solved only restricted subsets of the WSC with high precision. Others have developed systems for relaxed common sense datasets with looser constraints (Rahman and Ng 2012; Peng, Khashabi, and Roth 2015). These shortcomings can at least in part be attributed to the limited size of the WSC corpus, which is a natural side-effect of it being expertly handcrafted.
In general, endeavors to tackle coreference resolution reveal that, as much as the innovative strides in models and techniques are crucial to progress, the nature and de-mands of the tasks themselves guide and possibly constrain progress. As much as datasets like WSC are limited in size or naturalness, large-scale datasets like CoNLL are limited in their challenge level, exhibiting vulnerability to work-around tricks that both distinguish and confound neural models.
Accordingly, we propose the following desiderata for coreference resolution tasks. Such tasks should be • Challenging: in the two CoNLL shared tasks, most instances can be resolved using syntactic salience, the number and gender of the antecedent, or other simple syntactic and semantic cues.
• Large-scale: WSC is small (fewer than 1000 instances), resulting in difficulties in evaluation (i.e, obtaining a substantial train-test split) and for neural models that require much more training data.
• Based on natural text: Sentences in, e.g., WSC are written from scratch by experts, requiring substantial time and effort to produce and inducing a bias specific to the person/s who wrote them. It is important that sentences occur in text written for other purposes.
We present a coreference resolution dataset that aims to fulfill these desiderata. The main contributions of this paper can be summarized as follows: 1. We construct a preliminary dataset of 1275 samples comprising a binary-choice pronoun disambiguation task. These samples require significant common sense and background knowledge to solve. We generate samples semi-automatically (which makes the dataset easily scalable), automatically scraping candidate sentences from a single corpus, automatically altering the antecedents in the sentences to be identical as far as gender, number, and semantic class, and validating the resulting sentences manually (if they are easy to resolve with common sense).
2. We demonstrate the difficulty of our preliminary dataset by showing the failure of five state-of-the-art systems to outperform a random baseline, while measuring high human performance.
3. We analyze the behavior of the state-of-the-art methods and empirically demonstrate that they generally avoid using the context to make a decision.

Standard coreference resolution
Early automated techniques for standard coreference resolution, that is, the task of correctly partitioning the entity and events that occur in a document into resolution classes, date back decision trees and hand-written rules (Hobbs 1977;McCarthy and Lehnert 1995 Chinchor and Sundheim 2003) and the ACE (Doddington et al. 2004). These targeted noun phrases that were tagged with coreference information, but were either limited in size or had annotation that was restricted to a small subset of entities.
In light of these hindrances, the datasets of (Pradhan et al. 2011;Pradhan et al. 2012) were proposed as large-scale corpora that boasted high inter-annotator agreement, obtained by restricting the coreference to phenomena with annotations of high consistency, as well as being accompanied by a standard evaluation scenario to capture various ranges of performance on competing systems.
Largely due to the recognition and quality of these two tasks, many coreference resolution systems emerged, ranging from entirely hand-engineered systems to advanced learning approaches. The multi-pass sieve-based system of (Raghunathan et al. 2010) is fully deterministic and makes use of mention attributes like gender and number, boasting the best results on the CoNLL 2011 task for a number of years ). Later, highly lexical learning approaches took over as the new state-of-the-art , followed by more recent neural models (Wiseman, Rush, and Shieber 2016;Clark and Manning 2016). The current state of the art on the CoNLL 2012 task is the first end-to-end coreference resolution model that does not rely on a syntactic parser or handengineered mention detector (Lee et al. 2017).
Gender bias in standard coreference resolution (Zhao et al. 2018) observed that state-of-the-art methods for the standard coreference resolution are gender-biased, exploiting various stereotypes that appear in society. They study two hypotheses to explain this bias: the training data and the the external information sources (word embedding). Accordingly, they devise a new dataset called WinoBias that serves both as gender-bias test for coreference resolution models and which can also be used as a training set to compensate for the lack of diversity in previous corpora (i.e the 2 CoNLL tasks). This new dataset contains 3160 sentences and covers cases that require an understanding of both semantics and syntax. The sentences contain entities corresponding to people referred to by occupation. The following example is representative: (1) The physician hired the secretary because he was overwhelmed with clients.
(2) The physician hired the secretary because she was overwhelmed with clients.
As illustrated, the goal is to obtain a balanced dataset that would ultimately remove gender stereotypes. Experiments were conducted using three different models: the Stanford coreference resolution system (Raghunathan et al. 2010), the coreference resolution system of , and the recent system that tackles this task end-to-end (Lee et al. 2017). The results show that the end-to-end neural model maintains the same performance without the gender bias when trained partially on both the previous datasets and WinoBias.
A concurrent work, (Rudinger et al. 2018), also proposed an empirical study of the biases existent in current coreference resolution systems. As in the previous paper, they focus on gender bias by occupation. In contrast to (Zhao et al. 2018), however, who attribute the bias in part to the datasets, they conjecture that the gender bias comes primarily from the models themselves. They study three different methods: the rule-based Stanford coreference model (Raghunathan et al. 2010), statistical methods (Björkelund and Kuhn 2014;Durrett and Klein 2013) with hand-crafted features and a neural model (Clark and Manning 2016). Based on statistics from the Bureau of Labor, they show that the three models contains gender bias that are present in today's society.
The work on gender stereotypes already provides some insight into the behavior of current models. In the example above, if she is predicted as being the secretary, it is because the model learned a representation for this profession that contains gender information and based its prediction on that. The tested models do not capture the context and the relation between was overwhelmed and hired. Our work does not focus specifically on gender stereotypes. Unlike Wino-Bias, our task is composed of sentences that occur naturally in text. Moreover, we devise an experiment to show empirically that current state-of-the-art models often rely on the entity itself and ignore the context when making a decision.

"Difficult" coreference resolution
As the creators of the CoNLL tasks themselves note, most techniques rely primarily on surface-level features, such as the proximity between mentions, as well as shallow semantic features such as number, gender, named entities, semantic class, etc. They posit that a reduction of this knowledge gap is essential for the advancement of coreference resolution systems and that an important step towards that includes their large, standardized corpora (Pradhan et al. 2011).
A manually constructed dataset of challenging pronoun disambiguation problems called the Winograd Schema Challenge was proposed by (Levesque, Davis, and Morgenstern 2011). It was designed with the goal that any successful system would necessarily have reduced this knowledge gap. Although the WSC has shown itself to be an important means to evaluate systems en route to achieving human-like intelligence, as far as in the task of pronoun disambiguation, the size and quality of the dataset has become a crucial bottleneck for progress. A Winograd-like expanded corpus was proposed by (Rahman and Ng 2012) in order to partially address the WSC's size limitation. However, systems that perform well on the expanded dataset (Rahman and Ng 2012; Peng, Khashabi, and Roth 2015) do not report similar results on the original WSC, likely due to loosened constraints in the former.
The task that we propose distinguishes itself from the WSC (written from scratch by few experts) by being constructed from sentences that occur naturally in text. This contributes to much more diverse and pervasive problem instances. It is particularly important that, as well as being challenging, these tasks contain cases that are widely representative, so that improvements are more likely to transfer to the full coreference resolution task. The process by which we construct the dataset is also largely automatic, and so is amenable to significant scaling.

Hard-CoRe Coreference Task
We have developed a task called Hard-CoRe (a play on the expression "Hard Co-Reference") that features 1275 difficult pronoun disambiguation sentences. Specifically, each task instance is a short passage containing a target pronoun that must be correctly resolved to one of two possible antecedents. As an example: (3) Yifei goes about to woo Kishori, so that [she] can get the letters possessed by her. Who is [she]? (Answer: Yifei) Problem instances are controlled to not give away the pronoun's antecedent through cues involving syntactic salience, the number and gender of the antecedent, or other simple syntactic and semantic cues. Systems must make common sense inferences; i.e., that someone who woos another may be doing so to acquire something.
In the following section, we will describe the methodology used to construct the task dataset, provide a glimpse of a few of its instances and resolution rationale, outline the evaluation criteria used, and describe the task characteristics.

Dataset Creation
We create the dataset using a semi-automatic process that performs the following steps: First, a collection of documents (in this case, the 2015 English Wikipedia) is processed to remove markup, non-ASCII characters, parenthetical expressions, headings and lists. Next, we pass each sentence through a pipeline that filters out sentences that either do not fit the structure presented in the previous section or where the pronoun could be easily resolved based on surface features such as entity type. Finally, the remaining sentences are manually inspected to ensure that a system can not rely on shallow syntactic cues such as gender and number and that they are not ambiguous. The details of the pipeline are described below.
In a first step, we use regular expressions to ensure the presence of connectives 1 . We keep sentences which have between 9 and 33 words after naïve tokenization, start with an upper case letter and contain no math. We use a regular expression to ensure that there is only one connective cluster (e.g. ", and though"), and that there are at least two non-stopwords before this connective and a pronoun after the connective. As a final check, we ensure that no pronoun occurs before the connective, which tends to remove sentences which are not self-contained.
On the remaining set, we use the fast Stanford's Maxent tagger (Toutanova et al. 2003) to infer a flat part-of-speech (POS) tag labelling. Using the POS tags, we ensure that there are two nouns before the connective which do not reoccur after the connective. A noun re-occurrence after the connective indicates that the pronoun likely refers to the non-repeated noun.  In the final, most expensive check, we use Stanford CoreNLP parser (Manning et al. 2014) to determine number, type and gender of the noun phrases (NPs). We ensure that there are exactly two NPs before the connective, and that we can use the pronoun to distinguish between them (e.g. "they" indicates plural, "he/him" indicates male person). The mentioned checks resulted in 72,243 sentences. At least some of these remaining sentences have similar properties to Winograd schema sentences; that is, the two noun phrases (NPs) and the pronoun share the same type. From here, we kept only sentences where the type indicates that both NPs correspond to persons, which further filtered the remaining sentences to a total of roughly 5931 sentences. We do this because NPs which denote people are often named entities or can easily be replaced by named entities without loss of information. In addition, since our hypotheses relate largely to how current resolution systems use gender cues, the bulk of the gendered pronouns can be found when they refer to such NPs and so we sought to efficiently target these instances.
For the remaining 5931 sentences, we examined/adjusted them in a series of five steps to control their quality and difficulty: 1. We examine the resolution provided by (Raghunathan et al. 2010)'s system (via CoreNLP) for the sentence. The sentence, often structured so that only one of the entities shares the same gender as the pronoun, is correctly resolved by the resolution system anyways, but we check that the resolution is correct for each sentence regardless. The resolutions serve as the gold label for our task. 2. Then, we address the gender giveaways by replacing one of the named entities so that both entities and the pronoun match in terms of gender (e.g., in a sentence with "James", "Jessica", and "she" as the NPs and pronoun, we replace "James" with "Jane"). 3. Next, we ensure there are no structural giveaways (e.g., replacing "Bob defeats and stakes Malik , but fails to kill [him]", with "Bob defeats and stakes Malik , but *he* fails to kill [him]") 4. We remove the sentence if, after the above three checks/modifications, the resolution is not straightforward or logical.
After applying the above rules, there remain 1275 sen-tences whose pronoun disambiguation cannot involve shallow semantic features like gender, number, and type, but instead requires varying degrees of external knowledge. These sentences constitute the HardCoRE dataset. Examples of some of its instances are provided in Table 1. As the examples reveal, each passage may require a unique instance of common sense knowledge to resolve. In the first example, our understanding that death (by way of killing, for example) causes a disappearance helps us conclude that Radu, the victim of the Brother Paulo's murder, is the one that reappearance most plausibly refers to.
In the next example, we quickly infer that accept is related to accepting the apology. Therefore, she refers to the one that is receiving apologies, which is Rose.
For the third example, out understanding that being tied is related to being deprived of freedom allows us to conclude that it is Alex who has come free.
In the fourth example, we first see that Peter is realized to be innocent by Carl and that the second clause is there to justify the realization. In reading the second clause, a candidate person (either Peter or Carl) is said to have genuinely believed the other to be guilty. An important trait of an innocent person can be their genuine belief that they are indeed innocent, which, in turn, may also result in their genuine belief that someone else is guilty. Accordingly, it fits better that it was Peter who had this genuine belief, and this is what gives Carl the justification he needs to realize Peter's innocence.

Sentence Characteristic % of Data
Masculine target pronouns 51.5 Feminine target pronouns 44.5 Non-Gendered target pronouns 4.0 First Antecedent Correct 50.7 Second Antecedent Correct 49.2 Table 2: Characteristics of data, in terms of pronoun distribution and correct label.
In Table 2, we report the characteristics of the data, which suggests a roughly equal distribution of feminine and masculine target pronouns (he/him/his vs. she/her) as well as  an equal distribution of the two labels, which keeps random baseline performance at 50% expected accuracy. The significantly lower proportion of non-gendered target pronouns may be due to our final filtering step, where we kept only sentences in which the antecedents correspond to persons.

Evaluation
Our task requires the model to choose between two candidates, but classical coreference models build clusters of expressions that refers to the same entity. With respect to our setting, several errors can be made by these existing models: the two entities and the pronoun share a similar cluster (Both Antecedents Predicted), none of the two candidates shares a cluster with the pronoun (No Decision) or a cluster with the wrong candidate and the pronoun is created (Incorrect Decision). To obtain a score specific to our task, where only one candidate can be linked to the pronoun, we compute a Task Specific Accuracy that counts the number of correct decisions when only one of the two candidates is chosen. Given that there may exist a degree of subjectivity to the answers, we also determine a human performance baseline on the task. This equates accuracy to the percentage of problem instances for which a majority of people make the correct prediction. We explain this procedure in detail in the next section.

Experiments and Results
In this section, we compare the performance of five representative systems on our task: Stanford's rule-  (Lee et al. 2017)'s end-to-end neural system (E2E). We also report human performance (Human).
Human Performance: We determined the human performance by collecting the predictions of six native English speakers on a randomly generated sub-sample of 100 problem instances; we consider correct those predictions that agreed with the majority decision and matched the gold label derived from the original sentence. To determine the agreement statistics, only predictions that amounted to a strong majority decision (at least 5/6 agreement) among all six speakers were considered to agree.
We report the performance of the five coreference systems and the human baseline in Table 3. The human performance of 0.92 attests to the viability of the task. The performance of the automatic systems, random or slightly above random, demonstrates that state-of-the-art coreference resolution systems are unable to solve the task. Moreover, the end-to-end neural model struggles to distinguish the two candidates in 13% of the sentences by merging them into the same cluster.

Analysis by Switching Entities
We hypothesize that the current systems mostly rely on gender and number cues that are related to the entity itself and not the context. This causes performance to suffer when sentences contain gender-unbiased entities. We propose an experiment to validate our hypothesis. We use the current dataset and switch the two candidates every-time they appear in the sentence. If the coreference models rely on the context, the predicted candidate should change as well. If it relies on the entity itself, the output should stay the same.  Table 4 shows the proportion of predicted candidates that remain the same after switching. The rule-based system (Raghunathan et al. 2010) always resolves the same entity, suggesting that the decision is only based on the entity itself and that the context is entirely ignored. The hard decision behind this model mostly relies on a gender and number dictionary (Bergsma and Lin 2006). This dictionary is a countbased approach that assigns a masculine, feminine, neutral and plural score to each word. If the pronoun is his, the candidate with the higher masculine score is likely to be linked to the pronoun.  The other models, Stat, Deep-RL, E2E, and Latent, are much more robust to this experiment, demonstrating their use of context cues to make their decision. Yet, it is surprising that the model proposed by (Martschat and Strube 2015), which relies only on the previous and next token of each candidate as a context feature, outperforms the end-to-end model. The latter uses a Bidirectional LSTM to build a context-dependent representation of each candidate. It is likely that the model adapts to the training set, OntoNotes 5.0, by focusing on gender/number cues which represent the majority of pronoun disambiguation.
While an ideal coreference model would make use of common-sense reasoning and context cues, we show that current state-of-the-art methods for coreference resolution fail to perform well on the proposed task. The additional experiment demonstrates that the decision is often motivated by the entity itself and not by the context, especially when using rule-based methods. Yet, the performance of the method proposed by (Martschat and Strube 2015) is above random and suggests that there is space for improvement.

Conclusion
We presented a coreference task that features high human performance and inter-agreement but random or slightly above-random performance for various state-of-the-art resolution systems, including rule-based, graphical, and neural models. In addition, we demonstrated that the antecedents of a sentence seem to have the greatest impact on the resolution decision of these models, a behaviour that suggests their reliance on shallow semantic features like number and gender. In turn, models that exploit these features may perform very well on certain coreference tasks, but not on those for which a significant degree of common sense and world knowledge is required.
In the future, we plan to develop a larger corpus of at least 100,000 problem instances, by using additional text-data sources (besides Wikipedia) and by developing a crowdsourcing protocol to accept or reject candidate sentences processed by our the automatic modifications and checks. This would represent a corpus amenable to powerful neural models, which could serve as an important training set for models that have thus far performed poorly on difficult coreference problems like the Winograd Schema Challenge.