{'fraction_non_alphanumeric': 0.021122166041428806, 'fraction_numerical': 0.00391453270265862, 'mean_word_length': 5.005386875612145, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Knowledge Ratings in MetaboLights

This technical report presents an evaluation of the ontology annotations in the metadata of a subset of entries of MetaboLights, a database for Metabolomics experiments and derived information. The work includes a manual analysis of the entries and a comprehensive qualitative evaluation of their annotations, together with the evaluation guide and its rationale, that was defined and followed. The approach was also implemented as a software script that given a MetaboLights entry returns a quantitative evaluation of the quality of its annotations (available on request).


Motivation
Metabolomics is the systematic study of metabolites in a biological system, whose identification and quantification can provide insights to metabolic processes under certain conditions. These metabolite profiles can potentially act as biomarkers for certain diseases and their applications extend to toxicology, pharmaceutical research and nutrition. Repositories such as MetaboLights play a key role in data sharing and linking resources of that which is by definition a highly integrative field of study [2]. Therefore, since it is essential to ensure the quality of annotated information [1], we selected MetaboLights as our study subject that is exactly our topic of discussion.

Problem
Linking web resources implies using a standardized vocabulary in order to filter out ambiguity and subjectivity. However, it is frequent that this premise is overlooked when articles are submitted, as free text or no information at all is provided instead of ontology terms. It is unlikely that the data in MetaboLights proves to be an exception in this respect, and since there is not a comprehensive evaluation of annotated information in global terms, it is impossible to guarantee its quality beforehand.

Objectives
1. Produce a global evaluation of the annotations of a set comprising all public studies repositioned in MetaboLights by combining manual and an electronic approaches; 2. Identify and list any specific error or insufficiency found in the metadata file for each analyzed study; 3. Take into account and quantify all instantiated strings of text fielded in the descriptions section in place of ontology terms; 4. Run a statistical analysis of our results in order to support our conclusions regarding the global status of the database; 5. Make suggestions on how to improve the overall quality of stored information, based on our findings.

Framework
We now present a theoretical framework on information sources, methodologies and technologies we used for this project.

Information sources
Information on the structure and organization of MetaboLights as well as its content, was retrieved from the web page and the corresponding article in May 2015. Ontology structure and terms were retrieved directly from BioPortal and EMBL-EBI.

Methodologies and technologies
Our approach was centered around scoring the entries in MetaboLights. This required the retrieval of all annotations (identified by a PURL) listed in the metadata file for each of the public access studies in MetaboLights. For that, we designed two Python modules; the first one writes the links of the metadata files on a text file and the second one returns a list of PURLs by annotation type and writes all relevant information on a spreadsheet file. We then proceeded to map the appropriate ontology terms using Protégé visualizing tools and web resources. Each annotation was scored by dividing the terms' depth by the total length of the associated branch (values ranging from 0 to 1). Hence, the more specific a term is, the closer its score is to 1. The type scores were determined by summing up each annotation score and dividing the result by the number of annotations. The global score for an entry was determined by calculating the arithmetic mean of the scores for each type. For this we ruled out all types which had absolutely no annotations in the retrieved files and also study person type (5 studies with annotations). The individual scoring of each annotation was performed manually; for the type scores and global scores a Python module was designed and used. In this process we also took note of any irregularity discovered in the metadata. Additionally, we calculated the log global scores for a more realistic assessment of the quality, since we considered we should favor entries with relatively fewer annotations.
The scores' module also retrieves all unannotated strings under each type. Using the same method, we calculated new type scores taking the number of strings for that type into consideration instead of the number of annotations, which consequently led to different global scores, and compared the results.

Results
We now present a detailed description of our results as well as examples in which we demonstrate the algorithms we used.

Global Evaluation
The immediate intake from the global scores distribution is the fact that nearly a third (31/95) of the submitted articles have no annotated term ( Figure 1). Solely considering the goal of linking information, this represents a substantial flaw in the data. From our perspective, there are two possible solutions for this issue: either encouraging submitters to properly annotate their studies or implement an electronic annotation algorithm (which would be viable since none of the published studies lacks terms fielded as free text and most of them are fairly comprehensible in comparison to their respective ontology terms).
As to information which is in fact annotated, our results show that only a limited number of studies score above a reasonable threshold (log score¿70). The comparison between the average scores when considering either the total number of terms (annotated and un-annotated) or the total number of annotations (Table 1) suggests that while the number of free text descriptions has a significant impact on information quality (as it potentially means more of the terms could have been annotated), the existing annotations could possibly be more specific. However, it should also be noted that there are no extremely poor annotations in general terms (log score¡30)Be that as it may, this problem can be more easily addressed than the former, using the same methods we suggested and it would improve the quality of annotated information substantially.
Interestingly, the percentage of studies scoring above the mean is higher when we weight by the total number of terms (Table 1). While a lower average value was expected, this result suggests that the issue of unannotated terms is slightly more localized. The same can be inferred from the lower standard deviation value. Nevertheless, from our point of view, it remains a hindrance and should be fixed.
Our observations also identified a general trend in the data in which no submitted study is annotated for all types. Given the overall quality of the annotations, this is not entirely unsurprising. However, it is still noteworthy as even the best scoring studies fail in this aspect and it may reveal some problem the submitters may have in finding the appropriate ontology term.
Another situation which should be addressed is the fact that specific terms and links appear repeated in some metadata files and in consistent fashion. It is unlikely that this is intentional and could possibly be due a bug in the code of the database (as this is only valid for "metabolite profiling" and "mass spectrometry assay") or be a consequence of father-son relationships in the ontologies used.  Regarding our methodology, our choice of disregarding non-PURL links could have had a negative impact on our evaluation of annotation quality, as it would yield undoubtedly higher scores. While we lack statistical support to back this choice, while curating we deemed they were relatively irrelevant. On the other hand, had we not omitted the types for which there was no ontology term in any of the metadata files, scores would drop significantly lower.

Type score analysis
With respect to types scores, assay ranks consistently higher than the others ( Figure 2). However, this may be due to the fact that there are only three different annotations in all the files, which is remarkable giving the variety of terms we encountered while searching through several ontologies.
Protocol on the other hand scores extremely low, which in part can explained by a small number of annotations. We also noted a considerable amount of free-text descriptions containing multiple terms, which can be informative for a curator but can make electronic annotation that more difficult.
Looking at the average score differences (Figure 3) and the score distribution (Figure 2), we can conclude that while design terms are fairly well annotated (in comparison), that type also includes the majority of unannotated terms. As opposed to that, there is no unannotated term under assay. As stated previously, this can be due to the overall redundancy of the annotations.

Entry-specific considerations
Over the course of our manual approach, we noticed that entries MTBLS107 through MTBLS111 represented exactly the same study about phytohormones, with each entry pertaining to a specific compound (zeaxanthin, lutein, α-carotene, β-carotene and lycopene, respectively) but otherwise identical in terms of metadata. Given the fact that the annotation regarding the phythormones was unspecific, the intention behind the five submissions might have been a solution to specifying each one. For this and since it is fundamentally the same study, we assert that it is a poor organization of the data, both semantically and concerning the repository itself, and it skews database statistics by attributing extra submissions to the authors' study count.
Additionally, while scoring, we noted the PURL corresponding to the term "ovarian cancer" in MTBLS150 and MTBLS152 was broken (<error>Ontology not specified or not supported</error). Our conclusion is that either the authors annotated it incorrectly or the PURL was altered afterwards.
One other situation is that on a few cases terms are listed under two types, but only one of them has the corresponding annotation. This is true for MT-BLS81 and MTBLS147; for the former, the term "lipid droplets" is listed under factor type (twice) and design type, with a PURL being provided only for the first one; for the latter, the term "NMR spectroscopy" is listed under assay type and protocol type, and only the term under assay type is annotated. This incongruence is probably due to an error in assigning the string terms, as presumably they should be fielded just under one type, rather than them being unannotated. For example, for all other studies, "NMR spectroscopy" appears under assay type and not protocol type. Also, following the link in MTBLS81, no ontology term was returned ("The page you are looking for wasn't found. Please try again.")

Discussion
As per our initial objectives, we feel the result is satisfying. Nonetheless, given the volume of data we gathered, we may need to reorganize some topics covered by this report in order to thoroughly explore and describe our findings, otherwise, as it is now, we would exceed the page limit. Also, some of our groundings lack statistical support, even though we do not perceive this to have an effect on our conclusions. Perhaps we should have made to our scoring method, by allowing scores to be enriched by sheer number of terms, since the best scoring study only contains four annotations. However, we found no correlation between the number of terms and global scores and that should be explored first. Another aspect which should be analyzed, given the differences between type scores, is the possibility to assign different weights to each one.
Maybe the most interesting prospect we have is to perform a qualitative evaluation of the free-text descriptions and, eventually, an algorithm for electronic annotation if that does not prove to be overly ambitious.
One final suggestion for the database developers: a scoring method similar to that which we used could be employed to block submissions rating under a predetermined value. We feel this would encourage submitters to annotate their studies properly.