{'fraction_non_alphanumeric': 0.031578947368421054, 'fraction_numerical': 0.01761265909872721, 'mean_word_length': 4.9185667752443, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
The impact of simple feature engineering in multilingual medical NER

The goal of this paper is to examine the impact of simple feature engineering mechanisms before applying more sophisticated techniques to the task of medical NER. Sometimes papers using scientifically sound techniques present raw baselines that could be improved adding simple and cheap features. This work focuses on entity recognition for the clinical domain for three languages: English, Swedish and Spanish. The task is tackled using simple features, starting from the window size, capitalization, prefixes, and moving to POS and semantic tags. This work demonstrates that a simple initial step of feature engineering can improve the baseline results significantly. Hence, the contributions of this paper are: first, a short list of guidelines well supported with experimental results on three languages and, second, a detailed description of the relevance of these features for medical NER.


Introduction
Named Entity Recognition (NER), such as the recognition of person names, organizations, locations or medical entities, has become a crucial task in any Natural Language Processing (NLP) application, as a first step to other types of processing as, for example, Relation Extraction (Oronoz et al., 2015). Several tools have been developed for this task, such as CRF++ (Kudo, 2013), SVM (Kudo and Matsumoto, 2001) or Perceptron (Collins, 2002). Using these tools, and training them with a set of annotated data, many people can obtain a NER system easily and apply it to the respective domain. In this paper the experiments will be performed with clinical texts, on the recognition of Medical entities such as disorder or drug brand names. The basic NER models make use of a sequence of (word form, features, tag) elements for training. For inference, the system will give the tag sequence with the highest score given a new text. Each model is defined by a set of features, taken from the surroundings of each word to be tagged, usually by means of a sequential tagging approach.
Many techniques have been developed in order to improve the NER results, such as the incorporation of additional information, in the form of lemmatization, POS tagging, dictionaries and ontologies (IHTSDO, 2016), or the inclusion of knowledge acquired by unsupervised techniques like Brown clusters (Brown et al., 1992;Clark, 2003), word2vec neural models (Agerri and Rigau, 2016) or deep neural network architectures (dos Santos and Guimarães, 2015) that yielded significant improvements.
However, this availability of tools and techniques has led to using only a limited set of predefined or standard models that were successful for a prototypical NER task, without any kind of time-consuming adjusting (Pradhan et al., 2014). Moreover, as most published papers center on novel techniques (Ratinov and Roth, 2009;Turian et al., 2010), sometimes less effort is devoted to data analysis or to filtering and tuning the models. Researchers rarely give the full details of feature engineering and they often present their best configurations, or otherwise they only study the impact of one or two specific types of feature. However, the benefits of sophisticated techniques would be better highlighted taking a stronger baseline as departure. In this sense, this paper may be useful to researchers that are new to the field of medical NER, showing the impact of simple feature engineering on medical texts in three languages.
As an example, looking at the systems presented at the Semeval 2014 Shared Task 7 on English Medical texts (Pradhan et al., 2014), we see that most of the system descriptions do not give a precise overview of the contribution of the simplest feature types (Ramanan et al., 2014;Leal et al., 2014;Kate, 2014;Attardi et al., 2014) and they give at most a list of the used features, but without a detailed account of each's performance. For example, while Attardi et al. (2014) describe word shape features, they do not describe the window of words used, while Parikh et al. (2014) use a window of three words ([-2,0]).
There exist several available systems for English, as cTAKES (Savova et al., 2010), which was used by some of the participants at Semeval 2014 or cLiner (Boag et al., 2010). However, for other types of languages, there is a scarcity of resources and information about the usefulness of the available features.
We will experiment the effect of using simple features on medical NER, giving a measure of the improvements that can be achieved without resorting to more sophisticated types of information. Although most of these techniques have been previously applied in many works (Pradhan et al., 2014), we think that their effectiveness has not always been clearly evaluated, and they are briefly described as a preprocessing step before applying other, more complex, techniques. The main contribution of this paper will be a thorough examination of simple features for the recognition of entities in the medical domain. To give a better account of the generalization across different languages, we will perform our experiments on English, Spanish and Swedish, hoping that these results will be useful for many researchers and will help them to follow the principle of doing the easy things first, before resorting to more complex models.

Experimental Setup
We will perform a set of experiments using different types of features, starting from the most basic type of information, the word form itself and its derivatives, and continuing with basic language processing tools as lemmatization, POS tagging and medical dictionaries and ontologies: Phase 1: using only word forms (plus lower-casing); Phase 2: using prefixes and suffixes of different length. For example, the four letter suffix -itis indicates an inflamatory disease, as in meningitis or bronchitis; Phase 3: using different patterns of capitalization of word forms (word starts with a letter, all letters are capitalized, or different types of numbers); Phase 4: using lemmas; Phase 5: using POS tags. Phase 6: using Snomed-CT tags.
With the objective of establishing measures of the contribution of several features corresponding to simple types of information to medical NER, we will examine three languages: • English (EN) We will use data from the SemEval-2014 Task 7 Analysis of Clinical Text Shared Task ShARe 1 . This corpus comprises annotations of disease entities (9,694 instances) over de-identified clinical reports from a US intensive care department (version 2.5 of the MIMIC II database) • Spanish (SP) The Spanish EHRs consist of patient records collected during 2008-2012 at the Galdakao-Usansolo Hospital leading to 141,800 documents, 52 million word-forms (Oronoz et al., 2015). The entire corpus was provided after anonymization, signing confidentiality agreements and passing the corresponding ethical committees. From this set of raw clinical text, a subset of 121 texts was randomly selected for manual annotation (3,362 instances of diseases and 1,406 drugs). • Swedish (SW) The Swedish clinical text 2 origins from patient records from over 500 different clinical units at Karolinska University Hospital. The texts were collected during 2009-2010 and are stored in HEALTH BANK (Dalianis et al., 2015). For this study, a supervised corpora was created, annotated with medical entities (4,000 entities corresponding to body parts, disorders and findings). Regarding the English corpus, we only had access to the train and development sets, because the test set was not public. This is not a problem, because from our experiments on the Semeval Shared Task datasets, the results on the test set increased by about 2 percent points (Pradhan et al., 2014), as using the train and development sets for training compensates the effect of evaluating on the unseen test set. For that reason, we will use the train set for training and will evaluate on the development set.
For the experiments, we will use our own implementation of the averaged structured perceptron (Freund and Schapire, 1999;Collins, 2002), a state of the art tagging model that relies on Viterbi decoding of training examples combined with simple additive updates. The algorithm is competitive to maximumentropy taggers or CRFs (Collins, 2002). For each experiment, we trained 25 iterations on different feature templates. Although not reported in this work, similar experiments have also been performed    with SVM and CRFs, obtaining results comparable but slightly lower than with the Perceptron. For English and Spanish, we used FreelingMed for lemmatization, POS tagging, and annotating Snomed CT concepts (Oronoz et al., 2013). For Swedish, we used Stagger (Östling, 2013). The experiments were tested following a greedy approach, taking at each phase the best model in the previous phase as a baseline. This approach can be debatable, as it could happen that the knowledge used in phase x+1 could not be useful when applied with the best model in phase x, but perhaps it produced improvements at phases earlier than x. We have also experimented the effect of applying each set of features independently, but our aim is to get an account of the benefits obtained by applying a simple yet coherent approach (from the simplest to more elaborated experiments), and we leave out of the scope of this work the development of more time-consuming tests, such as grid search. Table 1 shows the results (F-measure) with different values of the window size (WS). There is no use on trying a single WS for all the languages as it has different impacts on different languages. Note that lowercasing improved the results considerably for all three languages, specially for Spanish. We hypothesize that this can be due to the informal writing used in the Spanish medical reports, characterized by big differences in writing style and non-consistent use of casing (either lowercase, uppercase or mixed). The use of prefix/suffixes in Phase 2 (see Table 2) helps significantly for all the languages with respect to the best results from Phase 1 (above 5 absolute points in all cases). Lower casing does not seem useful for English and Swedish (0.5 improvement for English over the best result without lower casing, and no improvement for Swedish), but it gives an increase of 2 points on Spanish. Table 3 presents the effect of adding features to represent capitalization patterns (words formed only by capital letters and words that start with a capital letter) and number types 3 . The improvements are modest for Swedish and slightly better for English (adding 0.8 points) and Spanish (almost one point).   Using lemmatization, Table 4 shows that we get an improvement on English (+0.3) and a decrease for Spanish and Swedish. This seems surprising, as in principle lemmatization could be useful to normalize terms (e.g. singular/plural and feminine/masculine in Spanish). Note that, as we are performing a greedy approach, the number of features used grows from one phase to the next one, and this is the main reason why we limited the number of feature templates, because the gains are decreasing for each phase. It should be clear that our experiments do not conclude that lemmatization is not useful but, rather, they show that it is not useful after applying other features. Table 5 shows the results using POS features, helpful for English and Spanish, but not for Swedish. We hypothesize that it could be due to the poorer quality of the Swedish POS tagger (Dalianis et al., 2015). Finally, Table 6 presents the results using specialized medical dictionaries, giving the best results for English and Swedish, but no improvement for Spanish. This aspect deserves further work, because the Spanish Snomed has similar coverage to the English version regarding concepts (around 300,000), but less terms (660,000 compared to 480,000).

Conclusion
Standard and well-known features together with model tuning are frequently being left aside by researchers in favor of novel approaches, as though they were low-level or insignificant mechanisms. By contrast, we have showed that these simple techniques lead us to achieve significant improvements at really low computation expenses. As an example, looking at the Semeval 2014 Shared Task, we can say from our results that a simple system using only word forms and POS would outperform more than half of the presented systems 4 . It is not our aim to imply that other systems were poorly designed, as most of them had other objectives in mind, such as experimenting new approaches but, rather, our objective is to delve into the details of the simplest approaches, that are specially interesting for implemented systems, but are often neglected in scientific papers 5 . The results for our best performing systems for Swedish and Spanish are near to those obtained by more elaborated techniques like word embeddings, although they are still far from the best performing system on the Semeval English test.
To summarize, we experimented the NER task related to the biomedical domain in three languages: Semeval task in English, and EHRs in both Swedish and Spanish. The techniques presented tend to be of much benefit, particularly for domains that lack of big amounts of data, as it is the case of biomedicine: • It is recommendable to re-case the text and well-worthy trying different window-sizes on each language (not simply using the default parameters adopted from other languages). • While prefixes and suffixes have a different impact on each language, it seems as though taking all prefixes and suffixes of lengths 3 and 4 is a generally recommendable configuration. These techniques can be specially useful when analyzing non-formal text, as in the Spanish medical records. • Regarding other types of information (capitalization, numbers, lemmas and POS) we have seen that, although the features can be effective, they should be carefully tested on each language and corpus. • Overall, we see that there are important differences on the impact of different features with respect to each language. This fact opens an interesting research area for analyzing the effect of language and corpus types on the effectiveness of each feature. For future work we will take these results as a stronger baseline and delve into state-of-the art techniques e.g. word embeddings (Bengio et al., 2006;Mikolov et al., 2013) and recursive neural networks (Lample et al., 2016) to gain an insight on their impact on medical NER for these three languages.