{'fraction_non_alphanumeric': 0.02688674348593377, 'fraction_numerical': 0.008720024914356897, 'mean_word_length': 4.897459442913989, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Combining rule-based and embedding-based approaches to normalize textual entities with an ontology

In this paper, we propose a two-step method to normalize multi-word terms with concepts from a domain-specific ontology. Normalization is a critical step of information extraction. The method uses vector representations of terms computed with word embedding information and hierarchical information among ontology concepts. A training dataset and a first result dataset with high precision and low recall are generated by using the ToMap unsupervised normalization method. It is based on the similarities between the form of the term to normalize and the form of concept labels. Then, a projection of the space of terms towards the space of concepts is learned by globally minimizing the distances between vectors of terms and vectors of concepts. It applies multivariate linear regression using the previously generated training dataset. Finally, a distance calculation is carried out between the projections of term vectors and the concept vectors, providing a prediction of normalization by a concept for each term. This method was evaluated through the categorization task of bacterial habitats of BioNLP Shared Task 2016. Our results largely outperform all existing systems on this task, opening up very encouraging prospects.


Introduction
An important part of knowledge is expressed in textual form, such as in scientific articles. Specialized literature is characterized by the presence of terms of interest which are often complex nominal groups (e.g. "epithelial cells of the intestine") and display a high variability in their forms. At the same time, life sciences are a field with many structured, albeit incomplete, representations of knowledge: ontologies. These representations have the advantage of being machine interpretable, which can greatly improve the ability of programs to extract and reuse information from texts. The biomedical/biological field is therefore a good candidate for the development of more efficient generic methods that make use of these structured representations.
To extract entity information from texts, two steps are commonly applied: recognition of named entities and normalization of these entities (also called entity linking in the general domain). The recognition step detects terms of interest (e.g. bacterial habitat references) while normalization identifies them precisely by linking them to specific concepts or categories of an ontology (e.g. "Tcells" is a bacterial habitat mention that can be identified by the "lymphocyte" labeled concept). We are focusing on this particular task.
Today's best-performing normalization methods commonly rely on supervised learning from data that is manually annotated by experts of the field. However in specialized fields, these annotations are rare because they are difficult to obtain. Moreover, given the large number of target concepts in these fields (e.g. the biomedical metathesaurus UMLS contains more than 3 million concepts, the Ontobiotope ontology contains more than 2 thousand concepts, etc.), it seems unlikely to obtain sufficient data to cover all the possibilities of learning. An efficient and distant supervised method (i.e. with learning based on the results of an unsupervised method) would therefore be of great interest.
In this work, we propose several methods, notably a distant supervised normalization method, which we evaluated through the BioNLP 2016 Shared-Task (Deléger et al., 2016) and its task of bacterial habitat categorization. Habitat entities are often designated by complex nominal groups with variable forms, offering a relevant case study.

Related work
In the biomedical domain, normalization approaches relying on dictionaries and similarities of form between terms and labels from a knowledge source (Hanisch et al., 2005;Schuemie et al., 2007) are historically the oldest approaches and are still much used. They often provide good precision but a low recall because they have difficulty dealing with important variations in the form of terms/labels (e.g. synonymy such as "T-cells" / "lymphocyte", hyperonymy such as "Chondrus crispus" / "algae", etc.). These types of approach also often combine dictionaries with manually defined heuristic rules (Gerner et al., 2010;Kang et al., 2013). However rules are timeconsuming to implement and highly dependent on the task and domain.
Statistical approaches aimed at learning directly the associations between terms and labels from corpora. Deep neural networks and word embeddings are those that have achieved the best performance at present (Mehryary et al., 2017), but the lack of training data and the large number of concepts seem to be a limitation on their potential for improvement.
Another recent method, CONTES (Ferré et al., 2017) is based on an approach that does not take into account the form of terms and concept labels, but only distributional information for terms, and hierarchical ontological information for concepts. It is based on the ability of the latest word embedding methods to generate relevant semantic spaces, as well as on building "concept embeddings" that preserves hierarchical information between concepts. This kind of method aims to overcome the problem of variability of forms of associated terms/labels, but does not otherwise take into account relevant morphological information.
In this work, we propose to improve the CONTES method by combining it with a rule-based approach, the ToMap method (Golik et al., 2011).

Material
Part of the data used is from the Bacteria Biotope categorization task of BioNLP Shared Task in 2016. The documents are MEDLINE references, consisting of titles and abstracts of scientific articles in the field of biology. The task is to assign concepts from the OntoBiotope ontology 1 to textual entities denoting bacterial habitats (entities are provided and do not have to be detected beforehand). The corpus is divided into two parts: a development corpus (combining the initial training and development corpora of the shared task) and a test corpus which we used to evaluate our method for the normalization task. The entities of each of these corpora have been annotated manually (see Table 1).

Dev.
Test  Table  2). The selection of this corpus was motivated by the need to use a corpus that is representative of the specific field of interest for this normalization task (i.e., habitats of bacteria). Considering that a high quantity of data for computing word embeddings does not guarantee a high quality in the biomedical field (Chiu et al., 2016), we chose to use this smaller targeted corpus as opposed to a larger and less relevant corpus.

Rule-based approach: ToMap
As first approach to normalize entities, we used the ToMap method (Golik et al., 2011). ToMap relies on the internal morpho-syntactic structure of entities and maps them based on their syntactic heads, the underlying assumption being that the head is often the most informative component. The system first looks for a match between the syntactic head of an entity and the syntactic heads of the ontology concepts. Then, the entity is assigned the concept(s) with matching head(s). As there may be multiple candidates for a given entity, a Jaccard index is also computed between the entity and each of the concepts, and the concept with the highest score is selected.
The core algorithm is complemented by a set of heuristics designed to handle specific cases. For instance, a list of uninformative syntactic heads is provided so that if a head belongs to this list, then the algorithm tries to match the modifiers instead (e.g., in "water sample", "sample" is not very informative so "water" will be chosen to perform the mapping). Other heuristics include disambiguation rules targeted at particularly ambiguous terms (e.g., "plant" which can designate either a processing factory or living things such as trees, flowers, etc.). These heuristics are dependent on the type of entities (in our experiments, heuristics are designed for habitat entities). In the remainder of the article, we refer to the core algorithm alone (without the specific heuristics) as "simple ToMap".
Not all entities can be matched to an ontology concept with the ToMap method. When the syntactic head of an entity has no equivalent in the ontology, the entity cannot be normalized to a precise concept and is simply assigned the root concept of the ontology (i.e., "Bacteria habitat").

Word embeddings with Word2Vec and Skip-Gram architecture
To train the word embedding method on the expanded corpus, the sentences were randomly shuffled and converted to lower case. Next, we applied the Word2Vec method with the Skip-Gram architecture (Mikolov et al., 2013). To evaluate our different methods, all based on embeddings, we chose to adopt the optimal parameters given in similar work on biomedical literature, which does not seem to drastically impact results (Chiu et al., 2016).  We chose to keep all words, even those appearing once in the whole corpus, because we want to normalize entities that contain words with a low frequency and it does not seem to have a real impact on the global vector space. The most impacting parameters are the size of the vectors and the size of the contextual windows. We chose the vector size following the initial work on CONTES (Ferré et al., 2017), which obtained the best results with an output vector size of 200. However, we choose a smaller size of the context window, because of their possible potential to affect the nature of semantic proximity in the embedding space (Turney, 2012). Indeed, a common hypothesis is that larger window emphasizes the learning of domain similarity between words, while a narrow context window leads to hyponymic gathering. Thus, a smaller size of context window could be more preferable in our task.
All remaining parameters are those by default. Table 3 gives the main parameters.

Term and concept embeddings
The CONTES method computes embeddings for each word of the expanded corpus as described in the previous subsection. Then, for each entity of the training corpus and of the test corpus, a term vector is calculated by computing the barycenter of the vectors of the words that compose the entity. In parallel, concept vectors are calculated for all concepts of the ontology of interest, thus generating an ontological space. Each concept is associated with a vector of equivalent size to the number of concepts in the ontology. Each dimension is associated with a fixed concept. The vector is initialized as a one-hot (i.e. all weights are set to zero except the weight associated with the current concept, which is set to one), then each of the weights of the dimensions associated with the ancestors of the current concept are also set to one. The method encodes the hierarchical information of the ontology: if we estimate the cosine similarity between ontology concepts, the parent/child concepts are always the closest. This hierarchical information has the advantage of being the most frequent semantic relation (is_a relation) in ontologies.
After the generation of the term vectors (that represent the textual entities) and of concept vectors (that represent the concepts from the ontology), the method performs a projection of the term vectors into the ontological space. We use the training corpus to learn the optimal projection. That is, the projection that globally minimizes the distance between the projection of terms in the ontological space and the vectors of the associated concepts is determined. The learning method we used is a multivariate linear regression in order to limit the overfitting risk, particularly with regard to a relatively small training corpus. This projection is then applied to the term vectors of the test corpus, allowing to obtain new term vectors in the ontological space, and to calculate a cosine similarity with the concept vectors. The closest concept is chosen to normalize a term. The method has been designed to address the problem of term variability, because it does not rely on the similarity of form between terms and concept labels.

Combining the two approaches: HONOR (Hierarchical Ontological NORmalization)
The aim of the HONOR method is to take advantage of the precision of ToMap and to complement it with the CONTES method which has the potential to address the problem of form variability. More specifically, cases that are not handled by ToMap will be normalized by CONTES. Our hypothesis is that CONTES, which is not based on 2 https://bibliome.github.io/alvisnlp/ form similarity, should have the potential to propose relevant normalization predictions in cases where ToMap cannot. The overall scheme of the method is shown in Figure 1.

Distant supervision version
Additionally, as ToMap is not supervised and potentially yields good precision results, we can also use it to generate a first prediction on a larger corpus, which will be used to train CONTES rather than using the gold standard. In this setting, the method becomes a distant supervised method, as it would not rely on manual annotation anymore to train its learning model. We chose to use the same corpus as the one used for the training of Word2Vec. We tested for many random subsamples of the full predictions of ToMap on this corpus and analyzed the impact of the number of selected examples on the global performance of these methods.
We  You need to download the module for the CONTES method 3 . AlvisNLP/ML allows to combine the two methods and to emulate HONOR.

Evaluation
We evaluated the performance of ToMap and CONTES individually and of our combined approach HONOR (both in its supervised and distant supervised settings), on the Bacteria Biotope normalization task of the BioNLP Shared Task 2016. The predicted concept identifiers are compared to the gold standard concepts according to the similarity measure of (Wang et al., 2007), with the weight parameter set to 0.65. The evaluation was performed by submitting our results to the evaluation server of the BioNLP-ST 2016 challenge site. We computed baseline results by assigning all terms to the concept "bacteria habitat", which is the root of the OntoBiotope ontology hierarchy. The baseline obtained a score of 32% (Table 6). For comparison, we also included results from other systems that proposed methods to normalize bacterial habitats and evaluated them on the shared task corpus (Grouin, 2016;Tiftikci et al., 2016;Mehryary et al., 2017).

ToMap and CONTES individually
ToMap can provide a normalization prediction for 54% of the terms. Even if there are still problems of ambiguity, it enables us to estimate the proportion of terms that have a form similar to an ontology label at least at the level of their syntactic head. Despite its limitation, ToMap obtained good performances on the test corpus and ranked above existing methods (Table 6). This shows that the method has a really good precision when it can provide a prediction.
As long as terms are composed with tokens from the initial vocabulary, which allows to calculate an embedding for the terms, CONTES can provide a normalization prediction for all terms to normalize. The method obtained a score of 61% (Table 6). Compared to previous work, this version of CONTES includes a few improvements (refined embedding parameters and extended training corpus). We also tested the impact of the hierarchical information encoded in concept vectors to evaluate the gain when using this information compared to a simple one-hot representation (Table 4). Indeed, a one-hot encoding does not use hierarchical information because all vectors are equidistant from each other. The improvement of 7% validates the usefulness of this information to improve the matching between the two vector spaces.
Taking into account hierarchical information 0.61 One-hot encoding 0.54 Table 4: Comparison between the CONTES method encoding of concepts and an approach with one-hot encoding of concepts

HONOR
We compared the results of HONOR to those of existing systems. We report all results in Table 5 and Table 6. In 2016, two teams participated in this task. The best method was the BOUN method (Tiftikci et al., 2016) which combined a form similarity approach and an information-3 https://github.com/ArnaudFerre/CONTES retrieval based approach (based on tf-idf) and obtained a 0.61 score. The method of LIMSI (Grouin, 2016), which is based on a form similarity approach, had the lowest performance (0.43). Since the 2016 shared task, the University of Turku has proposed an end-to-end neural network method (Mehryary et al., 2017) which has outperformed these methods with a 0.63 score.
Compared to these systems, our method performs well above by a 10 points increase compared to the Turku system. It also brings a significant gain to the ToMap method (+7 points in the supervised version and +6 points in the distant supervised version). There is only a one-point difference between the supervised and distant supervised HONOR methods, which shows that our method could enable to perform without manual annotated data and without a significant loss of performance.

Discussion and future work
The key hypothesis behind the efficiency of the CONTES and HONOR methods is that the semantic space of terms (based on distributional semantics) and the semantic space of the ontology (based on the specific hierarchical information of the current ontology) are homologous. Even if this work seems to indicate that there are at least some similarities between these two vector spaces, this hypothesis is most likely too strong. It would certainly be interesting to alter the space of embeddings to increase this similarity and then use a non-linear algorithm to find a better projection from the embedding space to the ontological space.
With this kind of word embedding-based approach, there is a problem with computing words that have not been met before (i.e. out-of-vocabulary words). Consequently, any new word encountered should require a complete recalculation of word embeddings to be taken into account.
Recently, a new method to calculate embeddings seems to overcome this difficulty (Bojanowski et al., 2016) and we plan to estimate its performance on our method in further work.
Like the CONTES method, a problem of high dimensional representation of the concept vectors persists. We plan to address this issue in further work.
The bacterial habitat normalization task includes the normalization of an entity by multiple concepts. Currently, no system has successfully addressed this issue. We would like to investigate this problem in the future.
Finally, beyond the benefits of using a multivariate linear regression, work is underway to explore the possibility that a non-linear learning method can provide a better projection between vector space of terms and vector space of concepts.

Conclusion
Our method seems to open up interesting perspectives for joint use of word embeddings, ontologies and form similarity based methods, particularly for the domain specific literature. This kind of synergy could also address challenges in fields where sufficient manual annotations are difficult to obtain.