{'fraction_non_alphanumeric': 0.03503556021919086, 'fraction_numerical': 0.01334965605689635, 'mean_word_length': 4.998251748251748, 'pattern_counts': {'":': 1, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Towards a Learning Approach for Abbreviation Detection and Resolution.

The explosion of biomedical literature and with it the -uncontrolled- creation of abbreviations presents some special challenges for both human readers and computer applications. We developed an annotated corpus of Dutch medical text, and experimented with two approaches to abbreviation detection and resolution. Our corpus is composed of abstracts from two medical journals from the Low Countries in which approximately 65 percent (NTvG) and 48 percent (TvG) of the abbreviations have a corresponding full form in the abstract. Our first approach, a pattern-based system, consists of two steps: abbreviation detection and definition matching. This system has an average F-score of 0.82 for the detection of both defined and undefined abbreviations and an average F-score of 0.77 was obtained for the definitions. For our second approach, an SVM-based classifier was used on the preprocessed data sets, leading to an average F-score of 0.93 for the abbreviations; for the definitions an average F-score of 0.82 was obtained.


Introduction
With the explosion of biomedical information, the number of biomedical abbreviations is growing rapidly. As there are no rules for the formation of new abbreviations, their detection and association to the correct full form becomes increasingly difficult. Abbreviation detection is especially useful for language technology applications like information retrieval (Byrd et al., 1994) and extraction (Maynard and Ananiadou, 1999;Roark and Charniak, 1998;Liu et al., 2002), NER and anaphora resolution. Yu et al. (2002b) claim that exploiting abbreviations in IR systems increases the number of relevant documents retrieved,  argue that not handling abbreviations in NLP is a major source of errors. In this paper, we describe the compilation of a monolingual corpus of Dutch medical texts which can be used as a gold standard for the detection of abbreviations. Chang and Schütze (2006) mention the lack of a suitable standard for English as one of the factors that hamper the creation and evaluation of efficient abbreviation detection systems. In section 3, we describe the creation of a Dutch gold standard for abbreviation detection and resolution and the guidelines used for the annotation of the corpus. Section 4 presents some challenges for the automatic detection of abbreviation patterns and in section 5, we refer to related work and we present our approach. Finally, in section 6, we present some conclusions and describe our future work.

Creation of a gold standard 2.1. Corpus
As the construction of a gold standard is both timeconsuming and expensive, such inventories are scarce, and even more so for a niche language like Dutch. Chang and Schütze (2006) list four criteria for a solid gold standard: size, breadth, accuracy and common use in the community. A bigger corpus generally leads to a larger coverage and thus to a higher accuracy. As we are focusing on a specific domain, namely the medical domain, the second criterium, i.e. breadth of the corpus, is of lesser importance to our research. There are two popular gold standards for English abbreviation detection: the AbbRE gold standard and the Medstract acronym gold standard. The corpus used for the evaluation of the AbbRE algorithm (Yu et al., 2002a) is claimed to be reliable, as it was annotated by three experts in the field. However, its size is next to negligible: it consists of only 10 articles. Medstract, a publicly available and commonly used standard, is somewhat larger but seems to be less reliable (Chang and Schütze, 2006). For Dutch, no such resources are available. For the construction of our corpus, we randomly selected abstracts from the Dutch journal Nederlands Tijdschrift voor Geneeskunde 1 (NTvG) and the Belgian journal Belgisch Tijdschrift voor Geneeskunde 2 (TvG), two important sources of information for Dutch-speaking experts in the biomedical sector in the Netherlands and in Belgium. This resulted in a corpus of 66,739 words, 29,978 (100 abstracts) of which are from the NTvG, and 36,757 (256 abstracts) from the TvG. The corpus was tokenized before the annotation.

Annotation guidelines
In order to be able to mark all abbreviations in a text, we needed to define what we consider abbreviations, and what distinguishes them from acronyms, clipping and other short forms. The definitions found in the literature vary, but generally, abbreviations are described as a shortened form of a word or phrase (Chang and Schütze, 2006).  distinguish six types of abbreviations: 1. Truncating the end, e.g. adm for administration (or administrator), also called clipping (Bloom, 2000) 2. First letter initialization, or acronyms, e.g. AAA for abdominal aortic aneurysm 3. Opening letter initialization, e.g. HeLa for Henrietta Lacks 4. Syllabic initialization, e.g. BZD for benzodiazepine 5. Combination initialization, e.g. e-mail for electronic mail 6. Substitution initialization e.g. ASD I for Primum atrial septal defect; Fe for iron. Liu et al. (2002) add a seventh type of abbreviation, i.e. the chemical abbreviation, which is a combination of letters and numbers. The kappa score (Carletta, 1996), which is an indication of the agreement between the annotators of the corpus, is 0.89. Table 1 shows the frequency of the labels used in our annotations.

Challenges for the automatic detection of abbreviations
Some "irregularities" in the formation or patterns of abbreviations can form a challenge to the automatic detection of abbreviations and their definitions.
First of all, many of the English abbreviations in Dutch texts which have a Dutch definition cannot be matched to the initial letters of the definition.
HAART krachtige antiretrovirale therapie (EN: highly active anti-retroviral therapy) Another challenge for the system will be the distinction between ordinary parenthetical patterns and parenthetical patterns which include an abbreviation and a definition, e.g.
(13) gunstige uitkomst (score 5) (EN: positive result (score 5)) Due to its different compounding rules, Dutch has more compounds which are written in one single word than English. Consequently, the letters in abbreviations have to be aligned to syllables or word parts rather than initial letters. Not all words are included in the abbreviation. Usually, function words are left out. In these cases, POS tagging or stop word lists can help the resolution process or the system can allow for words to be in the definition which are not initialized in the abbreviation (Liu et al., 2002). According to , the average number of different full forms for all abbreviations with six characters or less is 2.28. The longer the abbreviation, the less ambiguous it will be. This is especially challenging for the resolution of undefined abbreviations, i.e. abbreviations which have no definition in their local context, e.g.

Related research
Different methodologies have been applied to the detection and resolution of abbreviations. The most popular approach relies on the use of heuristics to detect patterns of uppercase words consisting of a limited number of letters, which occur in predefined constructions. Taghva and Gilbreth's (1999) system, for example, identifies words of 3-10 uppercase letters as candidate abbreviations. An algorithm matches the letters of the abbreviation against the initial letters of a candidate definition, which is situated in a window of 2*N (where N stands for the number of letters in the abbreviation) words next to the abbreviation. The authors report recall and precision rates of 86% and 98% for definitions respectively and 93% and 98% for the abbreviations. The Stanford Medical Abbreviation Method (Chang and Schütze, 2006) uses 3*N instead of 2*N. This is a hybrid system, using both rule-based and machine learning (cf. infra) techniques. The ARGH 3 (Acronym Resolving General Heuristics) method (Wren and Garner, 2002) uses a set of heuristics and refinement rules to identify the boundaries of acronym-definition pairs. It treats parenthetical constructions consisting of one word as candidate acronyms and attempts to match each acronym letter to letters within the words immediately to the left of it. Conversely, parenthetical patterns consisting of multiple words are considered as candidate definitions. The advantage of this system is that it is able to recognize word patterns that are not in the same order as the acronym letters (e.g. ACMV-NOg for African cassava mosaic virus isolate originating from Nigeria). The system achieves an estimated rate of 96.5% precision and 93% recall when tested on over 12 million MEDLINE records. Another example of a heuristics-based approach is presented by Park and Byrd (2001) who combine text markers (e.g. "(..)", "[...]" and "=") and linguistic cues (e.g. "short", "or" and "stand") with pattern-based recognition. In addition to heuristics, natural language processing tools can be used to refine the search space of the definition. Pustojevski et al. (2001), for example, use partof-speech information and only consider noun phrases as candidate definitions.
As an alternative to the heuristics-based approaches, a machine learning approach can be adopted. Chang et al. (2002) present a supervised learning algorithm for acronym identification, which only searches for "definition (acronym)" patterns, as this is the most frequently used pattern. Chang et al. use nine features to score the abbreviations (such as percentage of letters aligned at the beginning of a word, number of definition words that were skipped in the alignment, percentage of letters aligned on syllable boundary etc.). They report a 83% recall and 80% precision on the Medstract corpus composed of MEDLINE abstracts (Pustejovsky et al., 2002).

Pattern-based approach
Our pattern-based approach can be divided into two steps, as in most of the systems mentioned above: abbreviation detection and definition matching, based on some predefined patterns. In the first step, we tried to detect constructions which consist of capital letters, or combinations of capital letters with one to three lowercased letters and/or numbers. Parenthetical constructions and other text markers (e.g. "=", " " ", " ' ") (Park and Byrd, 2001) serve as indicators for abbreviation-definition combinations and are used to match the abbreviations to their full forms in the text in the second step. In the abbreviation detection stage, abbreviations are printed with their candidate definition, which is situated within a range of 3*N (Chang and Schütze, 2006) words preceding or following the abbreviation.
In the second step, i.e. filtering the definitions from the list of candidate definitions, we faced some of the challenges described in section 4: combinations of English abbreviations with Dutch definitions, which could not be matched on the basis of their initial letters, parenthetical patterns which did not contain an abbreviation-definition combination, the different compounding rules in Dutch, which made matching to the definition more difficult. We matched the first letter in the abbreviations against the words in the candidate definition and considered the matching word and the following words -until the end of the 3*N sequence-as a definition. In our future research, we intend to apply decompounding techniques to match all the letters in the abbreviations to word and/or word-part boundaries. The results of both the abbreviation detection and abbreviation resolution are presented in  Table 3: Results of the patern-based approach.

Learning approach
In addition to the pattern-based approach, we experimented with a machine learning approach for the detection and resolution of abbreviations in Dutch texts. Whereas the pattern-based approach relies on a set of pre-defined patterns which are applied to raw text, in the machine learning approach we start from a corpus which has undergone the following preprocessing steps, viz. POS tagging and NP chunking performed by a memory-based shallow parser (Daelemans and van den Bosch, 2005).
The following information was encoded in the feature vector both for the detection and resolution of abbreviations: token, part-of-speech of the token, binary features indicating whether the token is an initial, part of a URL or in sentence initial position, morphological binary features to indicate whether the token starts with a capital letter, is completely capitalized, is a roman number, contains internal capital letters, is completely lowercased, contains or is completely composed of digits, punctuation, hyphens, a feature indicating whether the token incorporates vowels, etc. Also prefix and suffix information (n = 4) was incorporated in the feature vector. A symbolic word shape feature was used to merge all information encapsulated in the morphological binary features. For the resolution of the abbreviations, a simple heuristic feature was added which matches the first letter in the detected abbreviations against the words in the local context (context of 3*N sequence). YamCha (Kudo and Matsumoto, 2003) (version 0.33), an open source text chunker using Support Vector Machines, was used for the learning experiments. The experiments were conducted with standard parameter settings and in a ten-fold cross-validation set-up. Table 4 gives an overview of the results both for abbreviation and definition detection. For the detection of abbreviations, the SVM-based classifier yields an performance increase of 10% and more over the pattern-based approach. For the definition detection, the learning-based approach has led to a large increase of precision, leading to an overall F-score improvement of 5% over the pattern-based approach. Given the basic features which were used for this definition detection, we believe that there is certainly room for improvement for the results reported in Table 4; a more mature approach would definitely benefit from features encoding decompounding information in order to detect word-internal mappings (e.g. CVS: chronische-vermoeidheidsyndroom), features which allow for a mapping between English abbreviations and Dutch definitions, etc.  Table 4: Ten-fold cross-validation results of the learning experiments.

Manual error analysis
In order to better understand the errors on the abbreviation detection and resolution, we performed a manual error analysis on the output of the more error-prone pattern-based approach. This analysis can help us both to define new patterns and to develop new features for the learner. For both detection tasks, we distinguished between false positive and false negative results.

Abbreviation detection
In our error analysis of the abbreviation detection, we distinguished five possible causes for false positive results, i.e. words that were detected as abbreviations, but which had not been annotated as such: 1. Titles which are printed in capital letters: In this example, the colon was seen as a text marker indicating the possible presence of an abbreviation. Our system detected "HET" as an abbreviation.
2. First name initials which were erroneously detected as an abbreviation: In this example, A, B and C are not abbreviations, but rather a type of the disease.
5. Numerals in compounds consisting of an abbreviation, a numeral and a word. In this case, the system was not able to decide whether "-1" belonged to the compound word "1-gen" or to the abbreviation "PS-1".
The false negatives, on the other hand, were caused by the lack of rules which cover specific abbreviation formation patterns, such word-internal capital letters (22), the use of numerals and hyphens in compounds (23), single (lowercase) letter abbreviations (24), etc. Also abbreviations which do not have the orthographical characteristics of an abbreviation (25) were not detected. Most of these false negatives, however, were solved in the classification-based approach.

Definition detection
The pattern-based definition detection step suffered from error percolation from the abbreviation detection, leading to both false positives (26, 27), indicated with "[]" in the examples below, and false negatives (28). Since both detection problems were handled completely independently as a binary classification task in the learning experiments, this error percolation was avoided in the classification approach. Other false positives in the definition detection can be attributed to: 1. Linking of the first letter of the abbreviation to another word in the sentence which starts with the same letter. The words following this mislinked word are also often erroneously labeled as parts of the definition. Since the pattern-based approach operates on raw text, function words could be detected as first word of a definition. This problem was handled in the classification-based approach by incorporating partof-speech information.
(29) het hepatitis-A-virus (HAV) (EN: the hepatitis-Avirus (HAV)) 2. Parenthetical patterns which contain an abbreviation that is not locally defined. In the example below, HLA-H and HFE are examples of a "hemochromatosegen":