{'fraction_non_alphanumeric': 0.033129548762736535, 'fraction_numerical': 0.01304221251819505, 'mean_word_length': 4.685534591194968, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Partially-supervised Mention Detection

Learning to detect entity mentions without using syntactic information can be useful for integration and joint optimization with other tasks. However, it is common to have partially annotated data for this problem. Here, we investigate two approaches to deal with partial annotation of mentions: weighted loss and soft-target classification. We also propose two neural mention detection approaches: a sequence tagging, and an exhaustive search. We evaluate our methods with coreference resolution as a downstream task, using multitask learning. The results show that the recall and F1 score improve for all methods.


Introduction
Mention detection is the task of identifying text spans referring to an entity: named, nominal or pronominal (Florian et al., 2004). It is a fundamental component for several downstream tasks, such as coreference resolution (Soon et al., 2001), and relation extraction (Mintz et al., 2009); and it can help to maintain coherence in large text generation (Clark et al., 2018), and contextualized machine translation (Miculicich et al., 2018). Previous studies tackled mention detection jointly with named entity recognition (Xu et al., 2017;Katiyar and Cardie, 2018;Ju et al., 2018;Wang et al., 2018). There, only certain types of entities are considered (e.g., person, location), and the goal is to recognize mention spans and their types. In this study, we are interested in discovering entity mentions, which can potentially be referred to in the text, without the use of syntactic parsing information. Our long term objective is to have a model that keeps track of entities in a document for word disambiguating language modeling and machine translation.
Data from coreference resolution is suitable for our task, but the annotation is partial in that it contains only mentions that belong to a coreference chain, not singletons. Nevertheless, the missing mentions have approximately the same distribution as the annotated ones, so we can still learn this distribution from the data. Figure 1 shows an example from Ontonotes V.5 dataset (Pradhan et al., 2012) where "the taxi driver" is annotated in sample 1 but not in 2. Thus, we approach mention detection as a partially supervised problem and investigate two simple techniques to compensate for the fact that some negative examples are true mentions: weighted loss functions and soft-target classification. By doing this, the model is encouraged to predict more false-positive samples, so it can detect potential mentions which were not annotated. We implement two neural mention detection methods: a sequence tagging approach, and an exhaustive search approach. The first method is novel, whereas the other is similar to previous work (Lee et al., 2017). We evaluate both techniques for coreference resolution by implementing a multitask learning system. We show that the proposed techniques help the model increase recall significantly with a minimal decrease in precision. In consequence, the F1 score of the mention detection and coreference resolution improves for both methods, and the exhaustive search approach yields a significant improvement over the baseline coreference resolver.
Our contributions are: i We investigate two techniques to deal with partially annotated data.  ii We propose a sequence tagging method for mention detection that can model nested mentions.
iii We improve an exhaustive search method for mention detection.
iv We approach mention detection and coreference resolution as multitask learning and improve both tasks' recall.
The rest of the paper is organized as follows. Sections 2 and 3 describe the two mention detection approaches we use in our experiments. Section 4 presents the proposed methods to deal with partially annotated mentions. We use coreference resolution as a proxy task for testing our methods which is described in Section 5. Section 6 contains the experimental setting and the analysis of results. Section 7 contains related work to this study. Finally, the final conclusion is drawn Section 8.

Sequence tagging model
Several studies have tackled mention detection and named entity recognition as a tagging problem. Some of them use one-to-one sequence tagging techniques (Lample et al., 2016;Xu et al., 2017), while others use more elaborate techniques to include nested mentions (Katiyar and Cardie, 2018;Wang et al., 2018). Here, we propose a simpler yet effective tagging approach that can manage nested mentions.
We use a sequence-to-sequence model, which allows us to tag each word with multiple labels. The words are first encoded and contextualized using a recurrent neural network, and then a sequential decoder predicts the output tag sequence. During decoding, the model keeps a pointer into the encoder, indicating the word's position, which is being tagged at each time step. The tagging is done using the following set of symbols: {[, ], +, -} . The brackets "[" and "]" indicate that the tagged word is the starting or ending of a mention respectively, the symbol "+" indicates that one or more mention brackets are open, and "-" indicates that none mention bracket is open. The pointer into the encoder moves to the next word only after predicting "+" or "-"; otherwise, it remains in the same position. Figure 2 shows a tagging example indicating the alignments of words with tags.
Given a corpus of sentences X = (x 1 , ..., x M ), the goal is to find the parameters Θ which maximize the log likelihood of the corresponding tag sequences Y = (y 1 , ..., y T ): The next tag probability is estimated with a softmax over the output vector of a neural network: where W o , b o are parameters of the network, d t is the vector representation of the tagged sequence at time-step t, modeled with a long-short term memory (LSTM) (Hochreiter and Schmidhuber, 1997), and h i is the vector representation of the pointer's word at time t contextualized with a bidirectional LSTM (Graves and Schmidhuber, 2005).
At decoding time, we use a beam search approach to obtain the sequence. The complexity of the model is linear with respect to the number of words. It can be parallelized at training time, given that it uses ground-truth data for the conditioned variables. However, it cannot be parallelized during decoding because of its autoregressive nature.

Span scoring model
Our span scoring model of mention detection is similar to the work of Lee et al. (2017) for solving coreference resolution, and to Ju et al. (2018) for nested named mention detection, as both are exhaustive search methods. The objective is to score all possible spans m ij in a document, where i and j are the starting and ending word positions of the span in the document. For this purpose, we minimize the binary cross-entropy with the labels y: where Θ are the parameters of the model, y m ij ∈ [0, 1] is one when there is a mention from position i to j. If y m ij is zero when there is no mention annotated, this is the same as maximizing the log-likelihood. Nevertheless, we will consider models where this is not the case.
The probability of detection is estimated as: where V, W m , W h are weight parameters of the model, b m , b h are biases, and m ij is a representation of the span from position i to j. It is calculated with the contextualized representations of the starting and ending words h i , h j , and the average of the word embeddingsx ij : The complexity of this model is quadratic with respect to the number of words. However, it can be parallelized at training and decoding time. Lee et al. (2017) uses an attention function over the embeddings instead of an average. That approach is less memory efficient and requires the maximum length of spans as a hyperparameter. Also, they include embeddings of the span lengths which are learned during training. As shown in the experimental part, these components do not improve the performance of our model.

Partially annotated data
The partial annotation of coreference data for mention detection means that not labeled spans may be true mentions of entities. Thus, the approach of treating spans without mention annotations as true negative examples would be incorrect. On the other hand, the ideal solution of sampling all possible mention annotations, which are consistent with the given partial annotation, would be intractable. We want to modify the model's loss function in such a way that, if the system predicts a false-positive, the loss is reduced. This encourages the model to favor recall over precision by predicting more mention-like spans, even when they are not labeled. We assume that it is possible to learn the true mention distribution using the annotated mention samples by extrapolating the non-annotated mentions, and we propose two ways to encourage the model to do so.
Weighted loss function: We use a weighted loss function with weight w∈{0, 1} for negative examples only. The sequence tagging model makes word-wise decisions; thus, we consider words tagged as "out of mention", y t ="-", as negative examples, while the rest are positives. Although this simplification has the potential to increase inconsistencies, e.g., having non-ending or overlapping mentions, we observe that the LSMT-based model can capture the simple grammar of the tag labels with very few mistakes. For span scoring, the distinction between negative and positive examples is clear, given that the decisions are made for each span.
Soft-target classification: Soft-targets allow us to have a distribution over all classes instead of having a single class annotation. Thus, we applied soft-targets to negative examples to reflect the probability that they could actually be positive ones. For sequence tagging, we set the target of negative examples, y t ="-", to (ρ, ρ, ρ, 1 − 3ρ) corresponding to the classes ([, ], +, -). For span scoring, we change the target of negative examples to y neg =ρ. In both cases, ρ is the probability of the example being positive.

Coreference Resolution
We use multitask learning to train the mention detection together with coreference resolution. The weights to sum the loss functions of each task are estimated during training, as in Cipolla et al. (2018). The sentence encoder is shared, and the output of mention detection serves as input to coreference resolution. We use the coreference resolver proposed by Lee et al. (2017). It uses a pair-wise scoring function s between a mention m k and each of its candidate antecedents m a , defined as: where s c is a function that assesses whether two mentions refer to the same entity. We modified the mention detection score s m .
For the sequence tagging approach, the function s m serves as a bias value and it is calculated as: where y t i and y t j are the labels of the first and last words of the span, and v is a scalar parameter learned during training. At test time, only mentions in the one-best output of the mention detection model are candidate mentions for the coreference resolver. During training, the set of candidate mentions includes both the spans detected by the mention detection model and the ground truth mentions. The mention decoder is run for one pass with ground-truth labels in the conditional part of the probability function (Eq. 2), to get the mention detection loss, and run for a second pass with predicted labels to provide input for the coreferece task and compute the coreference loss.
For the span scoring approach, s m is a function of the probability defined in Eq. 8, scaled by a parameter v learned during training. Instead of the end-to-end objective of Lee et al. (2017), we use a multitask objective, which adds the loss function of mention detection. We do not prune mentions with a maximum length, nor impose any maximum number of mentions per document. We use the probability of the mention detector with a threshold of τ for pruning.

Experiments and Results
We evaluate our model on the English OntoNotes set from the CoNLL 2012 shared-task (Pradhan et al., 2012), which has 2802 documents for training, 343 for development, and 348 for testing. The setup is the same as Lee et al. (2017) for comparison purposes, with the hyper-parameters ρ, w, τ optimized on the development set. We use the average F1 score as defined in the shared-task (Pradhan et al., 2012) for evaluation of mention detection and coreference resolution.

Mention detection
First, we evaluate our stand-alone mention detectors. For this evaluation, all unannotated mentions are treated as negative examples. Table 1 show the results on the test set with models selected using the best F1 score with τ =0.5, on the development set. We can see that sequence tagging performs almost as well as span scoring in F1 score, even though the latter is an exhaustive search method. We also evaluate the span scoring model with different components from Lee et al. (2017). By adding the span size vector, the precision increases but the recall decreases. Replacing the average embeddingx with attention over the embeddings requires a limited span size for memory efficiency, resulting in decreased performance. Table 2 shows the results obtained for our multitask systems for coreference resolution and mention detection with and without the loss modification. The sequence tagging method obtains lower performance compared to span scoring. This result can be attributed to its one-best method to select mentions, in contrast to span scoring, where uncertainty is fully integrated with the coreference system. The span scoring method performs similarly to the coreference resolution baseline, showing that the naive introduction of a loss for mention detection does not improve performance (although we find it does decrease convergence time). However, adding the modified mention loss does improve coreference performance. For sequence tagging, the weighted loss results in higher performance, while for the span scoring, softtargets work best. In both cases, the recall increases with a small decrease in precision, which improves the F1 score of mention detection and improves coreference resolution. Figure 3 shows a comparison of the mention detection methods in terms of recall. The unmodified sequence tagging model achieves 73.7% recall, and by introducing a weighted loss at w=0.01, it reaches 90.5%. The lines show the variation of recall for the span scoring method with respect to the detection threshold of τ . The dotted line represents the unmodified model, while the continuous line represents the model with soft-targets at ρ=0.1, which shows higher recall for every τ . Lee et al. (2017) proposed the first end-to-end coreference resolution that does not require heavy feature engineering for word representations. Their mention detection is done by considering all spans in a  Figure 3: Recall of the mention scoring function with respect to the detection threshold τ . Values for the sequence tagging are referential document as the candidate mentions, and the learning signal is coming indirectly from the coreference annotation. Zhang et al. (2018) used a similar approach but introducing a direct learning signal for the mention detection, which is done by adding a loss for mention detection with a scaling factor as hyperparameter. This allows a faster convergence at training time. Lee et al. (2018) proposed a high-order coreference resolution where the mention representation are inferred over several iterations of the model. However, the mention detection part is same as in (Lee et al., 2017). The following studies proposed improvements over this work (Fei et al., 2019;Joshi et al., 2019;Joshi et al., 2020) but maintaining the same method for mention detection. Name entity recognition has been largely studied in the community. However, many of these models ignored the nested entity names. Katiyar and Cardie (2018) presents a nested named entity recognition model using a recurrent neural network that includes extra connections to handle nested mention detection. Ju et al. (2018) uses stack layers to model the nested mentions, and (Wang et al., 2018) use an stack recurrent network. Lin et al. (2019) proposed a sequence-to-nuggets architecture for nested mention detection.  uses pointer networks and adversarial learning. Shibuya and Hovy (2020) uses CRF with a iterative decoder that detect nested mentions from the outer to the inner tags. Yu et al. (2020) use a bi-affine model with a similar method as in (Lee et al., 2017).

Conclusion
We investigate two simple techniques to deal with partially annotated data for mention detection and propose two methods to approach it: a Weighted loss function and a soft-target classification. We evaluate them on coreference resolution and mention detection with a multitask learning approach. We show that the techniques effectively increase the recall of mentions and coreference links with a small decrease in precision, thus, improving the F1 score. In the future, we plan to use these methods to maintain coherence over long distances when reading, translating, and generating large text, by keeping track of abstract representations of entities.