{'fraction_non_alphanumeric': 0.03634361233480176, 'fraction_numerical': 0.013325991189427313, 'mean_word_length': 5.011585567692817, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Resolving Entity Coreference in Croatian with a Constrained Mention-Pair Model

Being able to identify that different mentions refer to the same entity is beneficial for applications such as question answering and text summarization. In this paper, we propose the first model for entity coreference resolution for Croatian. We enforce transitivity constraints with integer linear programming on top of pairwise decisions produced by the supervised mention-pair model. Experimental results show that the proposed model significantly outperforms two different rule-based baselines, reaching performance of 74.4% MUC score and 77.6% B3 score.


Introduction
Entity coreference resolution, the task of recognizing mentions in text that refer to the same realworld entity, has been one of the central tasks of natural language processing (NLP) for decades (Grosz et al., 1983;Connolly et al., 1997;Ponzetto and Strube, 2006). Coreference resolution owes this attention to numerous applications that could greatly benefit from the ability to identify different mentions of the same entity, such as relation extraction (Shinyama and Sekine, 2006), question answering (Vicedo and Ferrández, 2000;Zheng, 2002), and text summarization (Bergler et al., 2003;Steinberger et al., 2007).
Despite being easy to define, coreference resolution is considered to be a rather difficult task, primarily because it heavily relies on external knowledge (e.g., for resolving "U.S. President" and "Barack Obama", one needs to know that Obama is the president of the USA) (Markert et al., 2003;Durrett and Klein, 2014). Although machine learning-based approaches to anaphora and coreference resolution for English appeared almost two decades ago (Connolly et al., 1997), for many languages, including the majority of Slavic languages, no coreference resolution systems exist, mainly due to the lack of annotated corpora required for developing such systems.
In this paper, we present a coreference resolution model for Croatian. Our model enforces transitivity of coreference relations via integer linear programming (ILP) optimization over a set of binary coreference decisions made by the supervised mention-pair model (McCarthy and Lehnert, 1995). To the best of our knowledge, this is the first work on coreference resolution for Croatian, and one of the first efforts in coreference resolution for Slavic languages in general.

Related Work
Early computational approaches to coreference resolution for English were rule-based and heavily influenced by computational theories of discourse such as focusing and centering (Sidner, 1979;Grosz et al., 1983). As annotated coreference corpora became available, primarily within the Message Understanding Conferences (MUC-6 and MUC-7), research focus shifted towards supervised machine learning models. The first learningbased coreference resolution approach dates back to Connolly et al. (1997).
The mention-pair model is essentially a binary coreference classifier for pairs of entity mentions, introduced by Aone and Bennett (1995) and Mc-Carthy and Lehnert (1995). It is still at the core of most coreference resolution systems, despite its obvious inability to enforce the transitivity inherent to the coreference relation and the fact that it requires an additional clustering algorithm to build the coreference clusters. Interestingly enough, more complex models such as entity-mention model (Mc-Callum and Wellner, 2003;Daumé III and Marcu, 2005;Yang et al., 2008a) and ranking models (Iida et al., 2003;Yang et al., 2008b), designed to remedy for the shortcomings of the mention-pair model, failed to demonstrate a significant performance improvements over the simple mention-pair model.
On the other hand, research on coreference resolution for Slavic languages has been quite limited, mainly due to the non-existence of manually annotated corpora. The exceptions are the work done for Polish, (Marciniak, 2002;Matysiak, 2007;Kopec andOgrodniczuk, 2012), Czech (Linh et al., 2009), and Bulgarian (Zhikov et al., 2013). In particular, Kopec and Ogrodniczuk (2012) demonstrate that a rule-based coreference resolution system for Polish significantly outperforms state-of-the-art machine learning models for English, suggesting that the coreference resolution model benefits from morphological complexity of Polish.
In this work, we present a mention-pair coreference resolution model for Croatian. Our model accounts for transitivity of coreference relations by encoding transitivity constraints as an ILP optimization problem. Our constrained mention-pair model reaches a performance of 77.6% B 3 score, which is significantly above the state-of-the-art performance for English. This supports the claim that rich morphological information facilitates coreference resolution.

Dataset Annotation
Supervised coreference models require a manually annotated dataset. We next describe how we compiled a coreference resolution dataset for Croatian.

Annotation Guidelines
Although coreference in most cases relates to both mentions referring to exactly the same real-world entity (i.e., identity relation), coreference may also relate to several near-identity relations between two mentions (Recasens et al., 2010); e.g., one mention may be referring to part of the entity to which the other mention refers. Arguably the most important step prior to annotating the coreference resolution dataset is to determine the identity and near-identity relations that hold between different mentions of the same real-world entity. Considering that Croatian is a highly inflectional language, we adopt the coreference relation type scheme for inflectional languages proposed by Ogrodniczuk et al. (2013). This scheme includes the following coreference relation types (an instantiation of each of the relation types is given in Table 1): • IDENTITY relation covers the most common case of coreference where both mentions refer to exactly the same real-world entity; • HYPER-HYPONYM relation refers to cases where one mention is a hypernym of the other mention (but both mentions still refer to the same entity); • MERONYMY relation is present where one mention refers to the part of the entity to which the other mention refers; • METONYMY is a relation in which one of the mentions, although referring to the same entity as the other mention, is expressed via a phrase that typically denotes a different entity; • ZERO ANAPHORA is a relation where one of the mentions is expressed implicitly in the form of a hidden subject. Annotators were instructed to annotate instances of all of the aforementioned coreference relation types. They were instructed to link each mention to its closest previous coreferent mention in the text. Entity mentions that are not being part of at least one coreference relation were ignored.

Annotation Workflow
Six annotators participated in the annotation task. The corpus used for annotation comprised of articles from the Croatian news collection "Vjesnik". Annotators used an in-house developed annotation tool and were provided detailed annotation guidelines. We first asked the annotators to annotate a calibration set consisting of 15 news articles. We then discussed the disagreements and resolved them by consensus.
After calibration, we conducted two rounds of annotation. In each of the rounds we paired the annotators (pairings were different between the rounds), so that we have each document annotated by exactly two annotators. In both rounds, each pair of annotators was assigned 45 news articles, but each annotator annotated the documents independently. After each of the two annotations rounds, we measured the average pairwise agreement and observed that it reached 70% of accuracy. The fol- The entire annotation procedure yielded a dataset consisting of 270 news articles (a total of 147,000 tokens), annotated with almost 13,000 coreference relations. 1 Expectedly, the IDENTITY relation is by far the most frequent one in the dataset, accounting for 87% of all coreference annotations, followed by MERONYMY (7%) and ZERO ANAPHORA (4%). Given the prevalence of the IDENTITY relation in our dataset, in this work we focus on extracting only coreference relations of that particular type.

Constrained Mention-Pair Model
At the core of our approach is a mention-pair model, i.e., a binary classifier that, given two entity mentions, predicts whether they corefer. To produce clusters of coreferent mentions, a mention-pair model needs to be coupled with two additional components: (1) a heuristic for the generation of mention-pair instances (as forming all possible pairs of mentions would result in a dataset that would be heavily skewed towards the negative class) and (2) a method for ensuring the transitivity of the coreference relation and the clustering of coreferent mentions (as the set of individual binary decisions may conflict the transitivity property of the coreference relation).

Creating Training Instances
In this work, we generate training instances using the heuristic proposed by Ng and Cardie (2002), which is, in turn, the extension of the approach by Soon et al. (2001). We thus create a positive 1 A part of this dataset is freely available; cf. Section 5.
instance between a mention m j and its closest preceding mention m i , and negative instances between m j and all the mentions in between m i and m j (m i+1 , . . . , m j−1 ). However, if the mention m j is non-pronominal and m i is pronominal, then we create the positive instance by pairing m j with its closest preceding non-pronominal mention, instead of with m i .

Mention-Pair Model
Our mention pair model is a supervised classifier that predicts whether an IDENTITY coreference relation holds for a given pair of mentions. The classifier is based on a set of binary and numeric features, each comparing two entity mentions. Most of these features or their variants have been proposed in previous work for English and other languages. The features can be roughly grouped into four categories: string-matching features, overlap features, grammatical features, and distance-based features.
String-matching features compare the two entity mentions on the superficial string level (without any linguistic preprocessing of the mentions): • Indication whether the two mention strings fully match (f 1 ); • Indication whether one mention string contains the other (f 2 ); • Length of the longest common subsequence between the mentions (f 3 ); • Edit distance (i.e., Levenshtein distance) between the mentions (f 4 ).
Overlap features quantify the overlap between the mentions in terms of tokens these mentions share: • Indications whether there is at least one matching word, lemma, and stem between the tokens of the two mentions (f 4 , f 5 , and f 6 ); • Relative overlap between the mentions, mea-sured as the number of content lemmas (nouns, adjectives, verbs, and adverbs) found in both mentions, normalized by the token length of both mentions (f 7 ). Grammatical features encode some grammatical properties and aim to indicate grammatical compatibility of the two mentions: • Indication whether the first and second mentions are pronominal mentions, respectively (f 8 and f 9 ); • Indication whether the mentions match in gender (f 10 ). Morphosyntactic descriptors for Croatian content words, including the information on gender and number, are obtained with the lemmatization tool for Croatian (Šnajder et al., 2008); • Indication whether the mentions match in number (f 11 ). Distance-based features indicate how far apart the two mentions are in the text (the pronominal references cannot be too far from the closest coreferent noun-phrase mention): • Distance between the mentions in the number of tokens (f 12 ); • Distance between the mentions in the number of sentences (f 13 ); • Indication whether the two mentions are in the same sentence (f 14 ); • Indication whether the two mentions are adjacent, i.e., whether there are any other entity mentions in between them (f 15 ); • Number of other mentions in between the mentions at hand (f 16 ). Given that our original feature space is relatively small (i.e., several orders of magnitude smaller than the number of instances in the training set), we chose as the learning algorithm the support vector machines (SVM) with the radial-basis function (RBF) kernel that maps the training instances into a high-dimensional feature space.

Enforcing Transitivity
The IDENTITY coreference relation is inherently transitive. However, by making only the local pairwise decisions, the mention-pair model does not guarantee global (i.e., document-level) coherence of its decisions with respect to the transitivity of the IDENTITY coreference relation. Thus, we need a separate mechanism to ensure that the transitivity between individual pairwise decisions holds. In this work, we enforce transitivity as a set of linear constraints in the integer linear programming (ILP) optimization setting. We aim to maximize the objective function, which is a linear combination of mention-pair classifier confidences for individual pairwise decisions, by taking into account the linear transitivity constraints at the same time.
Objective function. Let M = {m 1 , . . . , m n } be the set of all entity mentions in a single news article, let P be the set of all mention pairs considered by the pairwise classifier, P = {(m i , m j ) | m i , m j ∈ M, i < j}, let r(m i , m j ) be the mention-pair classifier's decision for mentions m i and m j , so that r(m i , m j ) ∈ {−1, 1}), and let C(m i , m j ) be the confidence of the binary mention-pair classifier (0.5 ≤ C(m i , m j ) ≤ 1). The objective function is then defined as follows: where x ij is the binary label variable indicating whether the mentions m i and m j corefer.
Transitivity constraints. For all triplets of entity mentions (m i , m j , m k ) for which all three pairs (m i , m j ), (m j , m k ), and (m i , m k ) exist, we enforce the following linear transitivity constraints: Clustering. After the ILP optimization, we obtain transitively coherent coreference relations, which allows us to derive the clusters of coreferent mentions simply by computing the transitive closure upon those relations.

Evaluation
We split the manually annotated dataset consisting of 270 documents into a train set containing 220 documents and a test set with 50 documents. 2 We optimized the hyperparameters of our SVM mention-pair model (C and γ) by means of 10folded cross validation. We then trained the model with the optimal hyperparameters on the entire train set and evaluated that model on the test set.  Results. We show the performance of our mention-pair model, both without (MP) and with (MP+ILP) enforcing transitivity, along with the performance of both baselines in Table 2. We evaluate all models in terms of two standard evaluation measures for coreference resolution -MUC score and B 3 score. In order to evaluate the contribution of morphological features, we additionally evaluate the mention-pair model but excluding all features relying on morphological preprocessing (MP-MORPH). Results show that the supervised mention-pair model significantly outperforms both reasonable rule-based baselines. When morphological features are not used, the model exhibits a slightly lower performance, although the difference is not substantial. Enforcing transitivity in an ILP setting marginally improves the overall MUC score, but yields notable 2-point improvement in B 3 score. Precision is consistently higher than recall for all models and both evaluation metrics, which is consistent with the coreference resolution results for other languages (Lee et al., 2011;Kobdani and Schütze, 2011).
Overall, our results are over 10 points higher than the state-of-the-art performance for English (Lee et al., 2011) and comparable (higher MUC and lower B 3 score) to the best results obtained for Polish (Kopec and Ogrodniczuk, 2012), suggesting that coreference resolution may be easier task for morphologically complex languages.
Error analysis. In an attempt to identify the most common types of errors, we manually analyzed the errors made by the supervised mentionpair model. The vast majority of false negatives originate from mention pairs where external knowledge is necessary for inferring coreference, e.g.,željezni kancelar (iron chancellor) and Bismarck). Other common causes of false negatives include abbreviations, e.g., DS and Demokratski savez (Democratic Alliance), and distant pronominal anaphora (i.e., when an anaphoric pronoun is far away from its preceding coreferent mention). Most false positives stem from non-coreferent mentions with substantial lexical overlap, e.g., Društvo hrvatskih književnika (Croatian Writers' Association) and svečanosti u Društvu hrvatskih književnika (ceremonies at the Croatian Writers' Association). A significant number of false positives are due to a pronominal mention being close to some non-coreferent noun-phrase mention.

Conclusion
We presented the first coreference resolution model for Croatian. We built a supervised mention-pair model for recognizing identity coreference relations between entity mentions and augmented it with transitivity constraints enforced via ILP optimization. We demonstrated the effectiveness of the model by showing that it substantially outperforms two rule-based baselines. Enforcing transitivity improves the B 3 score.
Manual error analysis revealed that most errors are due to the lack of external knowledge necessary for inferring coreference. Thus, we plan to extend the model with knowledge-based features obtained from external knowledge sources like Wikipedia. Furthermore, as we currently use no syntactic information, we intend to incorporate dependency relations as features.
In this work we focused on resolving identity coreference between gold event mentions. With the goal of building an end-to-end coreference resolution system for Croatian, our future efforts will focus on the development of a mention detection model. We will also consider near-identity relations like meronymy and zero anaphora.