{'fraction_non_alphanumeric': 0.026545960346087652, 'fraction_numerical': 0.018836787137232712, 'mean_word_length': 4.769586507072906, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
An Analysis of the Semantic Annotation Task on the Linked Data Cloud

Semantic annotation, the process of identifying key-phrases in texts and linking them to concepts in a knowledge base, is an important basis for semantic information retrieval and the Semantic Web uptake. Despite the emergence of semantic annotation systems, very few comparative studies have been published on their performance. In this paper, we provide an evaluation of the performance of existing systems over three tasks: full semantic annotation, named entity recognition, and keyword detection. More specifically, the spotting capability (recognition of relevant surface forms in text) is evaluated for all three tasks, whereas the disambiguation (correctly associating an entity from Wikipedia or DBpedia to the spotted surface forms) is evaluated only for the first two tasks. Our evaluation is twofold: First, we compute standard precision and recall on the output of semantic annotators on diverse datasets, each best suited for one of the identified tasks. Second, we build a statistical model using logistic regression to identify significant performance differences. Our results show that systems that provide full annotation perform better than named entities annotators and keyword extractors, for all three tasks. However, there is still much room for improvement for the identification of the most relevant entities described in a text.


Introduction
Semantic annotation is an important basis for realizing the Semantic Web vision (Dill et al. 2003, Shen et al. 2015, a vision of a Web of machine-understandable data, and an important foundation for retrieving semantic information. Semantic annotation involves the recognition of short text fragments called mentions in documents (aka spotting) and links them to URIs defined in a knowledge base, aka disambiguation. Originally, automatic semantic annotation has been implemented using well-defined and restricted ontologies and knowledge bases (Kiryakov et al. 2011). This led to several platform such as KIM (Kiryakov et al. 2011) or Apache Stanbol (Sinaci & Gonul 2012). The emergence of the linked data cloud has encouraged the development of several annotation services (Milne & Witten 2013a, Ferragina & Scaiella 2010a, Mendes et al. 2011) such as DBpedia Spotlight and Yahoo which exploit LOD datasets and especially DBpedia/Wikipedia (Bizer et al. 2009) as their background knowledge bases. These knowledge bases, with their wide coverage, their structured description of content and their dynamic nature, are well-suited for enriching almost all types of unstructured text. However, they also raise new challenges due to their size and their cross-domain nature. Thus, it is not surprising that, among the various services that appeared in the last few years, we see a great variation in terms of performance (Cornolti et al. 2013a, Ludovic Jean-Louis & Ensan 2014, Chen et al. 2013, Ruiz & Poibeau 2015, Derczynski et al. 2015, Gangemi 2013. Additionally, mentions in text might represent entities, concepts, keywords, multi-word expressions, events, etc. and depending on the task at hand, some types of mentions might be more appropriate. While the majority of linked data annotators are described as "Semantic Annotators" without any specific type of mention in mind, in practice, many are more geared towards named entities (e.g. organizations, people) than topics or keywords (e.g. Artificial intelligence) for instance. It is thus often difficult to distinguish the most adequate service among the plethora of available Web APIs. In this paper, our aim is to facilitate such a choice, formalize semantic annotation tasks as well as assess some of the available linked data semantic annotators' strengths and weaknesses for these tasks. Note that this paper does not aim at providing an exhaustive survey of existing annotation APIs but rather focuses on some prominent APIs and describes a methodology for the evaluation of Semantic annotators.
Based on our analysis of the state of the art, we identified that semantic annotators can be applied to three main tasks: These three tasks are used as a basis for evaluating and predicting the performance of some of the most prominent semantic annotators on similar datasets. While there are frameworks such as GERBIL (Usbeck et al. 2015) which handle the evaluation of semantic annotators, none has made the distinction based on the three tasks as described above. Additionally, our results show that it does not suffice to compare metrics' results to evaluate the interest of semantic annotators. As we will see in this paper, a finer statistical analysis indicates that some semantic annotators' results are indistinguishable.
The remainder of this paper is organized as follows: in Section 2, we briefly present the limited state of the art on semantic annotators' evaluation. Section 3 describes our research methodology, including our research question, datasets and evaluation metrics. We also provide a description of the evaluated annotators. The following three sections describe our experimental results, first by taking macro-averages over documents, then by considering annotations independently from documents (micro-average) and finally by considering only the three categories of systems instead of their individual performances. Section 7 discusses our findings and the limitations of this study, and concludes with further discussion on the evaluation strategies and results.

State of the Art
Due to the recent development of semantic annotation systems, very few comparative studies have been published on their performance (Cornolti et al. 2013b, Joksimovic et al. 2013, especially in the three tasks mentioned above. Existing evaluation results are mostly related to specific semantic annotation services (e.g. Mendes et al. (2011) and Ferragina & Scaiella (2010b)), and hence are based on diverse metrics and gold standards, different data gathering methodologies and a limited set of evaluation datasets. In general, these works do not include numerous annotation systems for their evaluation and comparison. Two significant exceptions are the works reported by Cornolti et al. (2013b) and Usbeck et al. (2015). In these works, the authors provide a framework for benchmarking semantic annotation systems and comparing their performance. They introduce a set of problems for which semantic annotation systems are usually employed (e.g. Annotate to Wikipedia (A2W) and Disambiguate to Wikipedia (D2W)) and provide metrics to evaluate systems in these contexts. However, these results might not be sufficient to distinguish the top performing annotators without a deeper statistical analysis, as we propose in this paper.
One of the main limitations of existing literature in semantic annotation evaluation (Cornolti et al. 2013b, Meij 2013) is that it does not take into account the fact that the performance of a system may vary according to a specific task. By contrast, our evaluation aims at providing experimental results on the performance of current semantic annotation systems for the three tasks (SA, NE, KW) with the objective of identifying annotators that are best suited for each of them. To achieve this, we rely on standard datasets that are experimentally selected for each task. We identify the statistical significance of systems' results using ANOVA (macro-average) and logistic regression (micro-average).

Research methodology
In this paper, we address the following research question: RQ: How do linked data annotators perform on the three tasks (SA, NE and KW)?
To answer this research question, we examine the overall spotting and disambiguation performance of linked data annotators in terms of macro and micro evaluation metrics.

Datasets
We selected three different groups of datasets in English (a dataset is a corpus where mentions are spotted and disambiguated according to a gold standard). Each dataset is focused on at least one of three tasks 1 and uses DBpedia/Wikipedia as a background knowledge base. These datasets include 1) AI and IITB for the semantic annotation task, 2) MSNBC, for the evaluation of named entity annotation, and 3) the SemEval and Inspec datasets, which are used for the evaluation of keyword extraction.

AI
The AI corpus is a small set of documents composed of Wikipedia articles related to the artificial intelligence domain, which was used in previous experiments for ontology learning from text [20]. The gold standard was created by running all selected semantic annotators and evaluating the returned annotations as correct or incorrect. This evaluation was performed by two authors of this paper (two postdoctoral researchers).

IITB
IITB is a dataset proposed by Kulkarni et al. (2009) which includes more than a hundred documents comprehensively annotated by human experts. Documents were collected from popular Websites on sport, entertainment, health and science. In the literature, IITB is often used for the evaluation of named entity annotation, but in our evaluation we associate it to the semantic annotation task, since it contains annotations that go beyond named entities (eg: sniper, militant, October 7, president of Afghanistan)

MSNBC
MSNBC is a small collection of news documents (18 documents) on different popular subjects such as sport, politics and technologies and was proposed by Cucerzan (2007). MSNBC is mainly focused on important named entities. However, an initial analysis revealed significant problems in the dataset, such as entities that are indicated in the gold standard, but not found in the documents, entities cited in the documents, but absent from the gold standard, and, less frequently, incorrect entities specified in the gold standard. For the purpose of this research, we completely re-annotated the documents of this corpus, to obtained a gold standard more accurate than the original one.

SemEval
The SemEval dataset (Kim et al. 2010) is a standard benchmark for keyword extraction that associates keyphrases to documents. It contains 244 scientific articles, usually composed of 6 to 8 pages. The articles cover different research areas of the ACM classification: Distributed Systems, Information Search and Retrieval, Distributed Artificial Intelligence, Multiagent Systems, Social and Behavioral Sciences and Economics. Most articles essentially cover the Computer Science domain (75% of the documents) and the other documents cover the Economy domain. The gold standard includes key-phrases assigned by annotators (75%) as well as key phrases assigned by the papers' authors (25%). The SemEval corpus is divided into a training dataset (144 articles, 2070 key-phrases) and testing dataset (100 articles, 1443 key-phrases). In our experiment, we consider the 244 articles as a single corpus.

Inspec
Inspec is a set of 2 000 documents and consists of abstracts from scientific journal papers. Each abstract has two sets of keywords assigned by a professional indexer. One is a set of controlled terms from the Inspec thesaurus, and the other one is an uncontrolled set of keywords that contain any suitable term identified by the indexer in texts. Both sets can contain keywords that are not found in the abstracts. In our evaluation, we used only the uncontrolled set of keywords. Table 1 shows some descriptive statistics about the number of documents and the number of mentions in each dataset. We can notice that the Inspec dataset contains the highest number of mentions (it also contains many more documents), but the fewest number of mentions per document (due to the smaller size of documents and due to the fact that the keyword extraction task identifies the most relevant keywords only). In the IITB and AI datasets, the average number of mentions per document is much higher than in the three other datasets. This is expected for the task of semantic annotation. Finally, the average number of words in SemEval is much higher than in other datasets, but this value is somehow misleading, since these documents contain a high number of tokens that are not words (for example, elements of mathematical formulas).

Semantic annotators
In this section, we briefly present the semantic annotators selected for this study. For the purpose of our evaluation, we selected academically or industrially prominent semantic annotators available through a Web API. Table 2 shows all the evaluated annotators categorized according to their best-suited task based on their description: semantic annotation (SA), named entity annotation (NE) or keyword annotation (KW). We also indicate if the service is commercial, if the result of the annotation process may contain external entities that are not found in the text and, finally, the knowledge base that is used to disambiguate the entities. Herafter, we describe the chosen semantic annotators. In some cases, the description is very brief due to the lack of published research on the semantic annotator.
Watson 2 APIs employ a set of deep linguistic parsing methods and statistical language processing techniques for performing semantic annotation. Various APIs are available, among which three are relevant to our research objectives: named entity extraction (Watson/NE), keyword extraction (Watson/KW), and concept extraction (Watson/Concepts). The named entity extractor (Watson/NE) is able to disambiguate the detected entities and resolve co-references. Entities are linked to various datasets on the Linked Open Data Cloud (LOD). Keyword extraction (Watson/KW) produces a list of key-phrases without any linkage to an external knowledge base (i.e. without disambiguation). Concept extraction (Watson/Concepts) produces a 2 https://www.ibm.com/watson/services/natural-language-understanding/ list of concepts, that is, topics that are not necessarily mentioned in the text, along with their corresponding links on the LOD.
Aylien 3 is another commercial product that offers two services that are relevant for our study. One is the concept extraction service and the other is the entity extraction service, which not only extracts named entities, but also keywords. Since these results correspond to different tasks in our framework, we analyzed them separately. Note that this second service does not provide any disambiguation for the annotated entities.
On its website 4 , Babelfy (Moro et al. 2014) is defined as "a unified, multilingual, graph-based approach to Entity Linking and Word Sense Disambiguation based on a loose identification of candidate meanings coupled with a densest subgraph heuristic which selects high-coherence semantic interpretations". Babelfy is based on BabelNet, a multilingual semantic network.
Dandelion 5 offers several text analysis services for many languages: entity extraction, text similarity, text classification, language detection and sentiment analysis. Only the first one is of interest for our study.
Open Calais 6 is a service offered by Thomson Reuters. It can detect different kinds of entities, which are disambiguated with a proprietary knowledge base. It can also detect events, relationships and topics. DBpedia Spotlight 7 (Mendes et al. 2011) is a configurable annotator that is linked to DBpedia (we used the default values). After the spotting phase, DBpedia Spotlight pre-ranks DBpedia concept candidates for each spotted key-phrase in text. It uses a similarity score to determine which candidate concept is the most relevant. The similarity score takes into account the context of the phrase (a window of words around the phrase) and the context of each candidate concept.
Tagme 8 is a semantic annotator mainly designed for analyzing short texts such as tweets (Ferragina & Scaiella 2010b), but it has also been reported to perform well on longer documents (Cornolti et al. 2013a). Tagme tokenizes a given text and finds candidate spots from token sequences of up to six words. It uses a set of heuristics and probability and coherence measures to decide which spotted candidates should be considered for disambiguation and which spot must be pruned from the result set. Tagme returns all annotations in a text plus their corresponding relevance scores according to the text topic.
Umbel 9 offers two tagging services. One tries to detect concepts from the Umbel ontology in texts, and the other is restricted to noun phrases. It is the latter that has been used in this study. Note that by default, Umbel does not apply any stemming.
Yahoo Content Analysis API 10 annotates entities and concepts and also provides a ranking of these entities and concepts, according to their overall relevance. Access to the service is achieved through the Yahoo Query Language (YQL), a SQL-like language that enables querying, filtering, and combining data across the web.
TextRazor 11 offers many services for the extraction of information from text. It also enables customization by using Prolog rules. In our study, we only use the entity recognition service.
Enrycher 12 provides deep and shallow text processing services. We used the two following services: named entity resolution and keyword detection.
Finally, MeaningCloud 13 offers several text analysis services for many languages: topic extraction, text classification, sentiment analysis and text clustering. The topic extraction service detects and disambiguates named entities, and it can also extracts keywords. MeaningCloud offers the possibility of adding your own dictionaries in its annotation services.

Evaluation metrics
The following metrics are used for the evaluation of the semantic annotators: • Precision: the ratio of the number of correct items returned by the annotator over the total number of items returned by the annotator.
• Recall : the ratio of the number of correct items returned by the annotator over the total number of items specified in the gold standard.
• F-measure: harmonic mean of precision and recall, 2 × P ×R P +R . There are two ways of averaging the performance values: macro-average and micro-average. The first one consists in computing the metrics for each document and then averaging these values. In the second one, we take all mentions, without considering their source document, and compute the ratio of hits among these mentions. In our case, both approaches are relevant. The recognition of an entity in a document may depend on the overall context of the document 14 . For example, the occurrence of Washington in a document may refer to many different entities, and the document helps disambiguate its meaning. In this case, where the document represents the relevant context, macro-average should be used. Nevertheless, in many systems, a limited context of few words around the mention is used. In this case, each mention may be taken individually, and micro-average is well-suited for the evaluation. An important advantage of the micro-average is that it is less sensitive to the number of documents and their length. In this paper, we evaluate the systems with both approaches.  Ambiverse  76  1716  173  12433  2791  3438  AylienCon  260  2022  152  11860  5241  3907  AylienEnt  68  1332  147  12283  2726  3311  AylienKW  143  2049  198  4607  37886  8977  BabelCon  1346  11853  932  44597  58843  23514  BabelNE  113  2395  213  5132  3398  2250  Dandelion  1135  5464  402  62580  33359  20588  EnrychKW  51  663  71  1119  14445  3270  EnrychNE  54  1031  111  2996  2253  1289  MCloudEnt  115  1988  169  16547  3820  4528  MCloudKW  656  4983  374  38754  25605  14074  OpCalais  159  2286  222  16952  5345  4993  Spotlight  377  2933  210  22821  10040  7276  Tagme  1974  15212  1079  120291  70616  41834  TextRazor  989  6284  624  67826  6498  16444  Umbel  444  3184  257  26529  17803  9643  WatsonCon  63  809  79  1835  12917  3141  WatsonKW  333  4710  393  10592  40346  11275  WatsonNE  84  2191  203  9507  3198  3037  Yahoo  68  932  95  2040  12979  3223  Average  425  3702  305  24565  18505 We ran all semantic annotators on all the available datasets. Table 3 shows the total number of annotations extracted by each semantic annotator. We can notice considerable variations across systems and across datasets. For example, Tagme returns 5.7 times more annotations than DBpedia Spotlight, on average (41834 vs 7276). Similarly, Babelfy/Concepts, Dandelion, MeaningCloud/KW, Tagme, TextRazor and Watson/KW return more than 10 000 annotations on the average, while other systems return a much lower number of annotations (1289 for Enrycher/NE, and 2250 for Babelfy/NE). We may expect here that systems with the largest number of annotations will exhibit a high recall. We can also note that much more annotations are extracted from SemEval compared to other datasets. In fact, in this corpus, there are much more extracted keywords on the average than the number of correct mentions in the gold standard (more than 20000 on average, compared to 3689 in the gold standard). We thus expect a very low precision for this corpus. AI and MSNBC are the smallest datasets in terms of numbers of annotations, which is expected, since the number of correct mentions in their gold standards is also very low compared to other datasets.

Evaluation using macro-average
In this section, we use the macro-average to evaluate the performance of the systems for the two main steps of semantic annotation, namely the spotting step and the disambiguation step. In each step, we analyze the precision and recall of semantic annotators for the SA and NE tasks. In the KW task, there is not any disambiguation step, so only spotting is evaluated. All mentions returned by annotators and all the ones provided in gold standards are stemmed using an implementation of the Porter stemmer. As an example, two key-phrases 'parallel processes' and 'parallel processing' are matched to the gold standard entry "parallel process" because all have the same stem. This approach seems reasonable for the spotting phase, as we consider all these alternatives as valid mentions. If an entity is spotted more than once in a text, it appears only once in the gold standard. This means that we evaluate the capability of spotting at least one occurrence of each relevant mention. Table 4 presents the precision values obtained for all datasets, considering all spotted entities returned by systems (without any filtering). We can see that the AI corpus cannot really be used to distinguish systems' results as there are only two performance groups based on ANOVA. This is probably due to the small number of documents (8 documents only). With the other corpora, the situation is clearer: Aylien/Concepts and DBpedia Spotlight are the most precise APIs on the IITB corpus, which corresponds to the NE task. On SemEval and Inspec, results are quite low, with Yahoo being the top performer in both cases. Some systems are among the best ones on almost all corpora: Yahoo, Aylien/Concepts and DBpedia Spotlight. We can also note that, as expected, the precision observed on SemEval and Inspec is very low for all systems (remember that the SemEval corpus is the one with the largest number of annotations returned by the systems, thus increasing the probability of an incorrect annotation). Overall, the results in Table 4 show us, for instance, that a precision of 0.54 is in the same group as a precision of 0.45 on the IITB corpus. A shallower observation of those results might have led to a false conclusion, i.e. that EnrychNE outperforms by far WatsonCon, which is not the case. Table 5 presents the recall values. We can see that the best performing systems are not the same as in Table 4. Interestingly, Tagme and Babelfy/Concepts are among the worst systems in terms of precision, but among the best ones if we consider recall. This is expected, considering the high number of annotations returned by these system, which necessarily favours recall over precision. Yahoo, which is the best API in both SemEval and Inspec in terms of precision, does not perform so well in terms of recall on these two same corpora. At the opposite, Watson/KW obtains low precision values on these corpora, but performs very well in term of recall. Note that a recall of 0.47 puts this system clearly in a dominant position compared to the other ones. The trade-off between precision and recall can be observed in Table 6. For example, considering the IITB corpus, Dandelion, which is not among the top systems in precision and recall, is clearly the best one if we consider its F-score. Looking at the results for the two corpora related to the KW task, we see that Yahoo and Watson/KW are the top systems despite low F-scores. Interestingly, we can see that for both SA and KW tasks, the top-performing systems are almost the same for both corpora (TextRazor, Dandelion, Spotlight, Babel/Concepts, Aylien/Concepts and Tagme for the SA task; Yahoo and Watson/KW for the KW task). Here we observe that a keyword extractor Watson/KW is among the second group of systems for the AI corpus, despite the fact that this corpus is related to the SA task. Now considering MSNBC corpus, most of the best performing systems are NE systems, with the notable exception of Ambiverse, the top API, which is also the only one that appears among the best ones for both precision and recall.

The spotting step
Considering the SA and NE tasks, we can conclude that globally, the results correspond to the expectations, that is, the best performing systems on each corpus are the ones whose task is related to this corpus. For the KW task, the situation is quite different. By looking at the two corpora associated to this task, we see that only Watson/KW appears among the best-performing systems, due mainly to its very good recall score. Keyword extraction seems to be a difficult task, with F-scores lower than the ones obtained in the other tasks. Interestingly, Yahoo, the best performing system on SemEval, and second-best on Inspec, is not specifically designed as a KW extractor.

The disambiguation step
In this section, we evaluate the disambiguation capability of semantic annotators. Three datasets are used for this task: AI, IITB and MSNBC. We also consider only full semantic annotators and named entity annotators, as keyword extractors usually do not return any disambiguation information with their keyphrases. Also we restrict the evaluation to the systems that output links to Wikipedia or DBpedia. An annotation is considered correct if the pair m, e , where m and e are the textual mention and the linked entity, respectively, is found as such in the Gold Standard. If the mention m is found in the gold standard but annotated with the wrong entity by the system, it is considered as a miss. If a mention returned by a system is not contained in the gold standard, it is ignored, since there is no way of determining whether the entity is correctly disambiguated. To check the validity of the disambiguated entity, we take into account Wikipedia redirect links. Thus, a system that would return London_Heathrow_Airport will be considered correct even if the entity in the gold standard is Heathrow_Airport. Put simply, we are computing the ratio of correctly disambiguated entities among the correctly spotted mentions, that is, we evaluate the probability P(Disambiguating | Spotting). Practically, this means that, for each annotator, mentions that are not correctly spotted by the annotator are removed from the gold standard, leading to a reduced gold standard. Here recall is equal to precision, since all tested systems provide at least one entity for each spotted mention and these mentions are the same in the reduced gold standard.  Table 7 shows the scores obtained for disambiguation where we obtained very few groups using ANOVA with post hoc Tukey analysis (3 for AI, 4 for IITB and 3 for MSNBC). We can note that Watson/Concept outperforms all other systems on IITB, and is among the leading systems for the two other corpora. We can also note that the systems are usually rather good at disambiguating an entity once the spot has been found. Interestingly, we observe that Tagme is in the first group for AI and MSNBC but in the last one for IITB. Let us now consider the full annotation process, that is, the combination of spotting and disambiguation. In this case, an annotation is considered as a hit if it has been both correctly spotted and correctly disambiguated, and a miss if has not been correctly spotted (and should be according to gold standard) or if it has been correctly spotted but incorrectly disambiguated. We see that the results are not as good (see Table 8). For example precision values are much lower, especially on the IITB and MSNBC corpora. Watson/Concepts, which performs well on all corpora for disambiguation, outperforms the other systems only on AI for the full annotation process. In fact, we see that there is no system that really outperforms the other ones in terms of precision. In terms of recall, Tagme and TextRazor are among the top-performing systems for all three corpora. We can also note that Spotlight, Ambiverse and Enrycher/NE, which perform poorly on AI and IITB corpora, perform much better on the MSNBC corpus. For Spotlight and Ambiverse, this is somehow surprising: we expect SA systems to perform better on the two other corpora, which correspond to their task. For the F-score, Table 8 shows that TextRazor and Dandelion are the only ones found in the top-performing systems for both SA corpora (AI and IITB) despite the fact that TextRazor is not classified as a SA system. For the NE task, results on the MSNBC corpus show that Ambiverse and Aylien/Concepts outperform the other systems.
To summarize, our results using macro-average metrics show that despite good disambiguation capabilities, semantic annotators poorly identify entity mentions in text (spots), which leads to a weak performance for the full annotation process. Additionally, the limited number of datasets impedes the capability to discriminate systems at a fine-grained level. In the next section, we present our results using micro-average.

Evaluation using micro-average
In this section, we aggregate all spotted mentions for each system, and thus we can estimate the overall precision and recall, independently of the documents. Basically, the precision could be estimated by computing the ratio of correctly detected mentions over the total mentions spotted by a system. Similarly, by taking the set M of all mentions in the gold standard, we can estimate a recall value for each system by computing the ratio of correctly detected mentions over the number of mentions in the set M . The problem with this way of estimating precision and recall is that it becomes difficult to identify statistically significant differences between systems, since we only have one ratio for each system.
Thus we rather chose to estimate the probability that a system behaves correctly for a specific mention in a document. More precisely, to estimate the precision, we take each mention spotted by a system, and we calculate the probability of this mention to be correct using logistic regression. Similarly, for recall, we take each mention of set M and we consider the probability for the system to detect this mention. In our data, since each correctly detected mention is associated to a value 1, and 0 otherwise, we can estimate these probabilities using logistic regression 15 .
Tables 9 and 10 show the estimated probability values for precision and recall obtained with a logistic regression model. By comparing the results to Tables 4 and 5, we observe that the best performing systems are globally the same ones. Note that we do not present the metrics' values computed directly on the datasets as they are similar to the logistic regression values.
On the AI corpus, two systems display significantly better precision than the others: Watson/Concept and Yahoo. Remember that in our macro-average evaluation, these two systems are not distinguishable from the six other systems. For recall, we get the same four best-performing systems. On IITB and MSNBC, the results are consistent with the results obtained with macro-average. For the KW task, we see that Yahoo is not the only top system, in contrast to the results obtained with macro-average. In fact, Yahoo shares the top position with Watson/Concepts on SemEval and with three other systems (Aylien/Concepts, Open Calais and Spotlight) on Inspec.

Semantic annotators ranking
Based on our previous results, we propose a global ranking score that combines the individual rankings computed separately for each dataset. These rankings are necessary to take into account groups rather than individual systems values, which are non-statistically distinguishable. Let C be one of our five datasets. A system s is attributed a local rank i = Rank(s, C), if s is the i th best group according to its estimated performance on dataset C. The global ranking score is the average of system local rank i over the five datasets. Figure 1 shows the relative position of annotators by combining their ranks for precision (x axis) and recall (y axis). Note that the best ranking position is 1.
If we consider precision only, Yahoo and Aylien/Concepts are the best annotators. In terms of recall, the best one is Tagme, closely followed by Dandelion. As expected, we can notice that annotators with good precision tend to have poor recall, and vice versa.
It is interesting to note that almost all NE and KW systems appear in the upper triangle, which means that their combined rankings in terms of precision and recall are not very good. At the opposite, most of the best systems, the ones situated in the left triangle, are SA systems.

Logistic regression for the disambiguation task
In this section, we repeat the same generalization process using logistic regression for the disambiguation task (Table 11). Remember that what is computed here is the probability P (Disambiguating|Spotting), that is, we estimate the probability of correctly disambiguating the entity when a mention is correctly spotted.
We can see that on AI, the performance of the systems is not distinguishable. On IITB, Watson/Concepts significantly dominates the other annotators, and it shares this position with five other systems on MSNBC. We also note that Tagme, which does not perform very well on IITB, is among the best ones on MSNBC. Table 12 compares the annotators on the combined task of spotting and disambiguation. Once again, we note that the obtained results are very similar to the ones with macro-average. One interesting difference is   Figure 2 shows the combined rankings in terms of precision and recall using the same formula as before. Overall, we can draw the same conclusion about the superiority of SA systems. Figure 2: Ranking based on precision and recall, for the full annotation process across tasks and datasets 6 Evaluation for the three tasks

Logistic regression for the full annotation process
In this section, we determine if there is some correlation between the task associated to an annotator (SA, NE or KW), and the task associated to each dataset (remember that each dataset is also associated to one of these three tasks). We computed a logistic regression where we consider all the systems associated to each of the three tasks. Note that we restrict here our analysis to spotting since, as we noted in our results, this is the most decisive step for the annotators' performance. Also, this enables the inclusion of KW systems in the analysis (remember that these systems do not have a disambiguation step). Results are shown in Table  13. The first observation is that SA annotators globally perform better than other systems on all datasets, if we consider both precision and recall. If we consider only the precision, KW systems do not perform as well as SA and NE systems. Another conclusion is that there is not any clear correspondence between the original task of a semantic annotator and the corresponding dataset. For example we cannot conclude that SA annotators perform better than others on the AI and IITB datasets, contrarily to our expectation. Similarly, NE annotators are not necessarily the best performing annotators on MSBNC and, finally, KW systems do not dominate on the last two datasets.

Discussion and conclusion
The work presented in this paper provides a comprehensive study over a wide range of semantic annotation systems for three different tasks. This kind of evaluation provides a basis for choosing a particular system depending on the task at hand, e.g. achieving a comprehensive annotation of documents, identifying named entities, or providing a small number of relevant keywords. We provide a statistical analysis of our results through ANOVA and a logistic regression model. Hereafter, we refer to our initial research question and provide some answers drawn from our experiments.
How do linked data annotators perform on the three tasks (SA, NE and KW)?
The first major observation is the weak spotting capability of annotators across all three tasks with an F-score for the top performing systems (macro average) ranging from 0.40 to 0.53 for the SA task, from 0.17 to 0.30 for the KW task, and with only one system for the NE task reaching 0.75. Once the spots are identified, the majority of annotators disambiguate correctly the spots with an F-score between 0.86 and 1. However, due to the low spotting performance, the F-measure for the full annotation process only ranges from 0.31 to 0.43 for the SA task and from 0.37 to 0.47 for the NE task.
The second major observation is that semantic annotators perform better than the other two types of systems on all three tasks in terms of F-score. Even when the identified task is a keyword extraction or a named entity recognition, SA systems are still the best choice, especially AlienConcept, Yahoo, DBpedia Spotlight and Dandelion. However, the full annotation process, even for the top SA system, remains a difficult task given an F-score lower than 0.5. This is even more noticeable for the KW task, whose results do not exceed 0.30 F-score. When we statistically analyze tasks (SA, NE, KW) independently from the individual systems, we observe that all three system's types obtain a very low performance on the keyword extraction datasets (SemEval and Inspec). In terms of precision, SA are indistinguishable from NE (they obtain a similar results) for the SA task but they outperform NE systems in recall. Finally, SA systems outperform NE systems for the NE task. We notice also that NE systems do not return a lot of spots and fail to identify many relevant named entities. This might be explained by the emergence of several named entity types on the linked data cloud in contrast to the more traditional named entity detection task. Based on our experimental results, one key insight is that semantic annotators are better at annotating all concepts in documents rather than extracting a limited set of key-phrases (keyword extraction), or relevant named entities, but are nevertheless the best performing systems for all three tasks.
There are some limitations to our study. One improvement to our work would be to report evaluation results based on a semantic approach. In our current work, we applied a lexical approach for matching key-phrases and entities returned by semantic annotators with those indicated in the gold standards. As an example, based on this approach, two key-phrases 'parallel processes' and 'parallel processing' are matched because both have the same stem. This approach seems reasonable for the spotting phase. However, in the disambiguation part of the evaluation, this might cause some problems as two phrases will match only if both have been assigned to exactly the same entity in Wikipedia or any other knowledge base. This approach limits the evaluation performance by disregarding possible partial matchings that may exist between keyphrases. For example, given a gold standard phrase 'parallel processing method', systems that retrieve 'parallel processing', 'parallel systems' or nothing will be considered as equally unsuccessful. Cornolti et al. (2013b) attempt to address this issue by providing a weak annotation match mechanism. Based on this mechanism, two key-phrases match if they overlap and refer to the same entity in the knowledge base. However, there are situations that cannot be handled by this partial matching approach. As an example, consider one gold standard key-phrase 'parallel computing', and two systems I and II that return 'parallel processing' and 'CPU' as key-phrases, respectively. We can assume that both systems work better than a system that returns nothing for instance, while it is obvious that System I finds a closer match to the original keyword than System II. Cornolti et al. (2013b) try to address some of these concerns by applying Milne-Witten's relatedness measure between phrases (Milne & Witten 2013b). Nonetheless, it does not provide a set of formalized performance measures on this basis. To address these issues, a more comprehensive approach would have to deal with the semantics of links between entities on the linked open data cloud, such as sub-class, broader, narrower and similar links to evaluate the disambiguation performance. Another important limit of our evaluation is due the nature of our datasets. It is well known that the elaboration of a suitable dataset for the evaluation of semantic annotators is a very difficult task. There are usually a large number of mentions that must be annotated in a document. This can hardly be achieved automatically, and thus results in datasets that contain few documents, and makes it difficult to obtain reliable statistics. Also, by inspecting the gold standards distributed with the datasets, we note that there are usually many incorrect and missing annotations (e.g. MSNBC, IITB). Finally, in our evaluation, we used only one or two datasets for each task, which may not be sufficient to avoid biases in the analysis. Still, we do think that our results provide some interesting hints on the behaviour of semantic annotators on the three tasks.

Acknowledgements
This work was partly funded by the Royal Military College of Canada Academic Research Program and sabbatical leave fund and the NSERC discovery grant program.