{'fraction_non_alphanumeric': 0.037943393894450624, 'fraction_numerical': 0.010958678366909735, 'mean_word_length': 5.235294117647059, 'pattern_counts': {'":': 1, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Collection, Encoding and Linguistic Processing of a Swedish Medical Corpus - The MEDLEX Experience

Corpora annotated with structural and linguistic characteristics play a major role in nearly every area of language processing. During recent years a number of corpora and large data sets became known and available to research even in specialized fields such as medicine, but still however, targeted predominantly for the English language. This paper provides a description of the collection, encoding and linguistic processing of an ever growing Swedish medical corpus, the MEDLEX Corpus. MEDLEX consists of a variety of text-documents related to various medical text genres. The MEDLEX Corpus has been structurally annotated using the Corpus Encoding Standard for XML (XCES), lemmatized and automatically annotated with part-of-speech and semantic information (extended named entities and the Medical Subject Headings, MeSH, terminology). The results from the processing stages (part-of-speech, entities and terminology) have been merged into a single representation format and syntactically analysed using a cascaded finite state parser. Finally, the parsers results are converted into a tree structure that follows the TIGER-XML coding scheme, resulting a suitable for further exploration and fairly large Treebank of Swedish medical texts.


Introduction
A number of publicly available (bio)medical corpora and data sets have been come to light during the last years. These corpora vary considerably in size, quality, coverage, encoding and depth of linguistic and structural characteristics. The vast majority are monolingual English corpora, covering different medical genres, while a small number of text collections exist in other major languages, particularly French and German. According to Zweigenbaum et al. (2001), a corpus useful for testing and training biomedical Natural Language Processing (bio-NLP) tools must account for the variety of medical texts; diversity, (origin, genre etc.), must be obtained in addition to mere volume. Having the above statement in mind, we describe here the design, collection and development of a Swedish medical corpus, the MEDLEX Corpus. MEDLEX consists of a variety of text-documents related to various medical text genres, and it is annotated with different levels of structural and descriptive meta-information as well as linguistic information. All text samples are automatically fetched from heterogeneous web pages from the Internet and converted to text files. The corpus consists of: teaching material, guidelines, official documents, scientific articles from medical journals, conference abstracts, consumer health care documents, descriptions of diseases, definitions from on-line dictionaries, patient's FAQs etc. Our motivation in collecting and annotating a Swedish medical corpus initiated by the need to support lexical acquisition and further population of term databases, during our department's involvement in the EU-funded Network of Excellence: Semantic Interoperability and Data Mining in  This paper starts by providing some background notes on corpus collection and processing of biomedical corpora, Section 2. Section 3 provides the characteristics and details of the content of the MEDLEX Corpus. In Section 4 we describe the various annotations added to the corpus and provide some evaluation figures for some of the processing stages. In Section 5 we discuss how the results from the previous processes have been integrated into the parser, the parsing mechanism, as well as the conversion of the parser's output into the TIGER-XML format. Finally, conclusions and suggested ways to improve the various processes, the potential use of the corpus and directions for future research end the paper.

Background
The far more cited corpora within the area of biomedicine originate from the PubMed Central (www.pubmedcentral.org) and the MEDLINE database. Particularly, the GENIA corpus (Kim et al., 2003), which consists of 2000 MEDLINE abstracts, has been used in many bio-NLP related activities; e.g. Yakushiji et al. (2001), by applying a full parser for the extraction of argument structures. Other corpora include the GENETAG corpus (ftp.ncbi.nlm.nih.gov/pub/tanabe) Tanabe et al, 2005; the Yapex (www.sic.se/humle/projects/prothalt) Franzén et al. (2002) and the PennBioIE (bioie.ldc.upenn.edu) Bies et al. (2005). For a survey on some of the available linguistically processed (tagged) biomedical corpora see Cohen et al. (2005). Other known, but varying in length, annotation and availability bio-medical corpora, include the large corpus of electronic articles in the cardiology domain, 85 million words, by Teufel & Elhadad (2002); the smaller, 100,000 tokens of German clinical reports by Wermter & Hahn (2004), and the MUCHMORE (muchmore.dfki.de/resources/index.htm) parallel corpus of English-German scientific medical abstracts obtained from the Springer Link web site, one million tokens for each language.

Collection
The starting point for the collection of the MEDLEX corpus was to navigate the Internet and manually inspect through links or simple keywords, using own Google searches as input, a number of potentially relevant web-sites. However, during later progressing of the corpus collection and in order to reduce the time spend for finding new sites, we found useful to apply the BootCaT toolkit by Baroni & Bernardini (2004). BootCaT implements an iterative procedure to bootstrap specialized corpora and terms from the web, requiring a list of "seed terms", typical of the domain of interest. MEDLEX does not focus at a particular subfield due to the lack of very large resources within a particular subarea for Swedish, and as such, the texts range through many sub-domains, genres and specialised topics, including pharmacology. Thus, the target has been to locate a large number of potentially interesting, though heterogeneous, medically oriented Swedish sites and extract, as automatically as possible, textual information from them (e.g. by using Lynx and the perl LWP library).

Content Extraction
Most of the sampled documents were in (X)HTML format, while there were a number of documents in PDF and MS Word. However, various types of (X)HTML mark-up or other types of formatting information were removed from the original files and the whole body of texts was converted into text format (ISO Latin1). Important structural meta-information, particularly the title, the source of origin and the date of publication of each article (if available) were preserved.
The corpus includes: scientific articles from medical journals, teaching material, guidelines, official documents, conference abstracts, consumer health care documents, descriptions of diseases, definitions from on-line dictionaries/glossaries, patient's FAQs, editorial articles and encyclopaedic information.

Status of the MEDLEX Corpus
By the end of 2005, the MEDLEX Corpus was comprised of approximately 6 millions tokens, over 10,000 different articles. The corpus size is based on tokenized and free of formatting information text material. The MEDLEX is encoded using basic annotation featuress from the XML Corpus Encoding Standard (XCES); Ide et al. (2000). The corpus consists of text samples from a number of different text genres 1 and

Structural & Linguistic Annotation
The MEDLEX corpus was linguistically processed in the lines of other similar text collections (cf. Teufel & Elhadad, 2002).

Tokenization and Part-of-Speech Tagging
MEDLEX was firstly tokenized and segmented into sentences, using regular expressions. Tokenization and sentence segmentation of the texts into individual words and sentences is an important step since all other processing steps that follow depend on it. Spelling mistakes, "spurious" quoting or other erroneous linguistic phenomena where not corrected. Since some of the texts were of very special nature, we had to make certain modifications to our generic tokenizer in order to handle special cases that were erroneously tokenised. Some additional tokenization steps included adaptations for handling particular constructions, such as " After tokenization, the corpus was annotated with part-of-speech labels using a version of Brills tagger (Brill, 1994). The morphosyntactic annotation uses the Swedish MULTEXT tagset. Medical language as any other technical language exhibits some characteristics that differentiate it from general language e.g. sentence construction rules slightly differ from the normal constructions rules, use of idiosyncratic expressions, medical jargon of many terms etc. Therefore, the tagger's lexicon was completed with several hundreds of new entries of medical jargon, such as adjectives, particularly of Greek and Latin origin, and nouns, particularly: • names of diseases, (e.g. ending in "-it"/"lymfadenit") where the tagger annotated them as verbs (a common verb suffix in Swedish) • common drug names and chemical substances (e.g. ending in "-das"/"aminoxidas") where the tagger once again annotated them erroneously as verbs (also a verb suffix in Swedish) • (patho)anatomical terms (e.g. ending in "-is"/"pulmonalis") • frequent in the corpus words (compounds) ending in "es" or "ns": "insufficiens", "inkontinens", "pares" and "fluorescens" which were erroneously annotated as genitive nouns.
After the part-of-speech tagging, the annotated instances, nouns, verbs and adjectives, were lemmatized using finite-state machines. The input to lemmatization is tokens with their respective part-of-speech. Lemmatization provides direct access to all instances of a lexical entity via its base form.

Generic Named Entities
There is a whole range of named entities (NE) that can be encountered in various types of texts, and not only the "classical", in the NE recognition bibliography, types of named entities, i.e. person, location and organization. Along the line proposed by Sekine (2004), we also apply a rather fine-grained NE system for Swedish capable of recognizing eight main categories (person, location, organisation, event, object, work & art, time and measure) and nearly sixty subtype named entities, including a large set of different types of measure subgroups, such as: pressure, frequency, weight, dosage, speed, volume and temperature. The system is described in Kokkinakis (2004) and is based on a modular and scalable architecture consisting of five major components, making a separation between lexical, grammatical and algorithmic resources. The five components are: • lists of multiword names taken from various Internet sites; • a shallow parsing component that uses finite-state grammars, one grammar for each type of NE recognized • a module that uses the annotations produced by the previous two components (which have a high rate in precision) in order to make decisions regarding possibly un-annotated entities. This module is inspired by the Document Centred Approach by Mikheev et al. (1999). This is a form of on-line learning from documents under processing which looks at unambiguous usages for assigning annotations in ambiguous words 2 • lists of single names (approx. 100,000) • a theory revision and refinement module makes a final control on an annotated document with named-entities in order to detect and resolve possible errors and assign new annotations based on existing ones, for instance by combining various annotation fragments The generic NER system's performance has been evaluated on Swedish electronic patient records (Kokkinakis, 2005). The evaluation figures for each entity group ranged between 69,9%-100% precision and 66%-98% recall. During 2005 (in parallel with the implementation of a Swedish MeSH tagger, see next section), the system was extended with capabilities of recognizing and annotating medical terminology (a ninth NE category). This terminologically oriented module comprises seven subcategories, namely: "Anatomy (MDA)", "Living Organisms (MDO)" mainly names of viruses and bacteria, "Diseases (MDD)", "Chemicals, Vitamins, Enzymes and Drugs (MDC)", "Symptoms (MDS)", "Pharmaceutical Equipment/Devices (MDP)" and "Analytical, Diagnostic and Therapeutic Techniques (MDI)". All of the resources 3 used by the system have been either obtained from various medically-related internet sites or extracted from the MEDLEX Corpus. Some of the lexical resources we currently use, include parts of the Swedish ICD-10, the Swedish MeSH and a list of several thousands of drugs from the Swedish union of pharmaceutical industry FASS, (www.fass.se). Apart from the previous, we use morpheme lists consisting of a couple of hundreds of elements, mostly suffixes, specific to the various categories of the medical domain. For instance, in the "Diseases" module there are suffixes such as: "+opati", "+iasis", "+emi", "+itis", "+ism" and "+uri".

MeSH Terminology
During the second half of 2005, we started the implementation of a tagger based on the Swedish translation of the MeSH thesaurus (hierarchical terminology), after obtaining a license by the University Library of the Karolinska Institutet. The MeSH® (Medical Subject Headings) is the controlled vocabulary thesaurus of the NLM, U.S. National Library of Medicine. The original data from NLM have been supplemented with Swedish translations made by staff at the Karolinska Inst. Lib. based on the year 2006 MeSH. MeSH is used for subject analysis of biomedical literature; for more information visit: mesh.kib.ki.se/swemesh/swemesh.cfm. The implementation of the tagger uses the most important subtree hierarchies from MeSH, namely: A (Anatomy, 3277 terms), B (Organisms, 5407), C (Diseases, 16334 terms), D (Chemicals and Drugs, 18369 terms), E (Analytical, Diagnostic and Therapeutic Techniques and Equipment, 5265 terms) and F (Psychiatry and Psychology, 1528 terms). Moreover, in order to reduce the ambiguity space of the investigated problem, since we attempt to disambiguate the MeSH annotation (see later this Section), we decided to only use the upper level (level 0) of the lexical hierarchy for the classification of each term 4 . For instance, the term beta-Lactamases (betalaktamaser) has the label D08.811.277.087.180 which was reduced to D08 [Enzymes and Coenzymes]. By this reduction, we assume that terms having multiple labels under the same level are very similar.
However, prior to the implementation of the MeSH tagger we applied a number of conversion and normalization steps to the original material. For instance, changing the order of the head and modifier complements for several hundreds of entries in the database (e.g. "vacc-3 Symptoms are usually realized in Swedish texts either as periphrastic expressions or compounds and it is rather difficult to find suitable lexical resources on the Internet in order to simply apply some sort of dictionary lookup. Therefore, we have investigated the way these expressions are constructed in MEDLEX, by initially selecting a few characteristic symptom key-words and short phrase fragments and then create and apply regular expressions on an analysed version of the corpus. This way we could identify new symptoms in the near vicinity of the already matched ones and iteratively implement a new set of hand constructed rules using regular expressions with the data gathered by this process. 4 Reduced MeSH hierarchies are used among others by Rosario et al. (2002) in an experiment for assigning (English) noun compound relations. iner, orala" changed to "orala vacciner"); normalizing all inflected/non-inflected entries into a neutral non-inflected variant with optional inflected variants (e.g. "orala vacciner" changed to "oral(a)? vaccin(et|er|erna)?"); making adjustments to the implemented recognizer in order to capture some frequent phenomena of misspellings, agreement and grammatical errors that we could observe in annotated sample texts, probably caused due to the high variability in the expression of similar concepts by different authors (e.g. "escherichia coli" added "e. coli") or by the influence or "contamination" from the English language, particularly orthographic variation (e.g. use of 'ph' instead of 'f'). Finally, case folding was applied to all terms, except those consisting of uppercase letters, particularly acronyms. This was necessary in order not to introduce new forms of ambiguity, since the complete elimination of case information could introduce new ambiguities between homographs, i.e. uppercase/low case words. For instance, "kol/D01.268.150" (i.e. "carbon") and "KOL/C08.381. 495.389" (i.e. "Chronic Obstructive Pulmonary Disease"). An example sentence illustrates the annotation provided by both the generic NER (see previous section) and the MeSH annotator, for the sentence: "He and his colleague Riitta Ylitalo and rheumatologist Mikael Heimbürger have studied the vocal cords of 14 patients with different types of auto-immune diseases -reumatoid arthritis , systemic lupus erythematosus and Sjogrens syndrome ." (Figure 1).
Since the MeSH annotation is an ongoing activity and we have not any manually annotated material at our disposal at the time of writing the paper, we cannot provide any evaluation figures for its performance. However, we have experimented with the application of unsupervised disambiguation of the MeSH terms based on the principle of "one sense per discourse" statement by Gale et al. (1992). Thus, the current implementation applies a partial and simplistic disambiguation methodology in lack of suitable training material. We observed, therefore, that in many cases the unambiguous readings within the same document, can help disambiguating the meaning of an ambiguous term (relationship ambiguity). The results (Table 2) were obtained w.r.t. ambiguity reduction, for complete matches, on a sample of 10 articles, 8,490 tokens, from the Swedish Medical Association's magazine (www.lakartidningen.se).

Cascaded Parsing of MEDLEX
The results from the NER and terminology recognition are merged into a single representation format and fed into a syntactic analysis module, which is based on the Cass-parser, Cascaded analysis of syntactic structure. Cass applies a finite-state cascade mechanism and internal transducers for inserting actions and roles into patterns, and originates from the work by Abney, (1997). The parser we use has been developed by Kokkinakis & Johansson Kokkinakis (1999), and has been modified in such a way that can utilize the features provided by the pre-processors, which results into the effect of slightly decreased complexity of the grammar rules. Moreover, we also apply a number of pre-processing steps in order to capture a number of difficult linguistic problems at an early stage of parsing, and thus reducing ambiguity at the various levels of the linguistic processing 5 . Thus in all, but the final step, an input text passes a pipeline of finite-state grammars that may add or modify features to the part-of-speech annotated input; including the recognition and annotation of multi-word expressions, conjoined compounds, phrasal verbs, various types of appositions and pre-modifying measure/quantity words. The phrases in Cass consist of finite-state rules; in turn bundles of rules are divided into different levels depending on their internal complexity, simpler follow complex ones. The parsing involves a cascade of two major automata, the "phrasal" and the "clausal". The "phrasal" includes: phrases which include a named-entity annotation; phrases which do not include a named-entity annotation; adjectival phrases; prepositional phrases and verbal groups/chains. The "clausal" automaton includes: embedded questions with interrogative pronouns; relative clauses; adverbial and infinitive clauses; complement clauses, wh-questions with interrogative adverb/pronoun; yes/no questions; copula passive constructions; various types of main clauses; combinations of different types of main and subordinated clauses and constructions without a verbal predicate. All types of clauses are divided into different levels. The division depends partly on the type of the verbal group and the word order and partly on any available lexicalized complementizer or part-of-speech tags that can provide strong evidence for a particular type of clause. For an evaluation of the parser's performance on non medical texts see Kokkinakis & Johansson Kokkinakis (1999).
Finally, for the annotation scheme of the parsed output we have chosen the TIGER-XML encoding format (König & Lezius, 2003), since Cass doesn't provide any visualization capabilities or means of correcting the syntactic/semantic analysis (see Figure 2). TIGER XML is a flexible graph-based architecture for storage, indexing and querying. This way the syntactically analyzed results can be easily used for querying the partially parsed corpus by combing lexical features, semantic annotations and phrase labels. Moreover, TIGER-aware tools such as @nnotate (Plaehn & Brants, 2000) and SALSA (Erk & Pado, 2004) provide the means for modifying, correcting and enhancing both the syntactic and semantic analysis returned by Cass. At the moment the corpus is only available in the form of searchable concordances using the IMS Corpus Workbench, Figure 3; (Christ, 1994).

Conclusions
In this paper we have described the structure and content of a Swedish medical corpus, the MEDLEX Corpus. A resource that can be useful in many areas of Swedish me-dical informatics and Natural Language Processing. MEDLEX comprises over 10,000 articles, over 6 million tokens, annotated in many dimensions. We currently investigate different ways to make parts or the whole material copyright-free and hence make it available for further research. The lack of resources for linguistic processing of biomedical corpora can be a major obstacle in healthcare informatics, because it prohibits access to the information treasure in biomedical corpora for lesser spospoken languages, slowing down the promotion of healthcare-related technologies.
As stated earlier, the majority of corpora in the medical domain are in English, a phenomenon easily motivated by the fact that biomedicine is a field with global interest and researchers prefer to write in a language understandable by a wide audience; Teufel and Elhadad (2002) mention that in cardiology alone there are at least 700 journals. The situation in Sweden is not that bad, although it cannot reach the volumes written in Engish. For instance, there are a handful of scientific journals that do not cover a particular subfield but rather incorporate various new findings from diverse medical subfields.
Computer understanding of the underlying language is a far from trivial task that involves several layers of knowledge intensive processing and we have described some of the possible means for corpus enhancement. There are several issues that need to be investigated in more depth and possibly in a larger scale. For instance, the use of a human in the process loop, in order to inspect intermediate results; and the use of tools such as @nnotate for correcting or enhancing the syntactic analysis. The need to conduct an evaluation on a larger material, and possibly using the full MeSH levels, and/or doing things in another order. Maybe the MeSH results can benefit from applying parsing before annotation, and thus let the MeSH tagger only look inside noun phrases. For the coverage of MeSH, a trained physician would have been the right person to mark unlabelled terminology; however, we could not consult such a person at the time of this study. Some revisions and extensions of the disambiguation part are also worth further exploration. It is well-known that the polysemous words' meaning depend on the context of use, at least on non-technical corpora, a fact that might even be stronger in technical corpora, i.e. a term probably shares the same sense throughout a single document. The Swedish MeSH contains over 50,000 terms (including synonyms), but it still does not cover all clinically useful terminology and empirical studies can be of benefit for its content's growth.

Acknowledgements
This work has been partially supported by the Semantic Interoperability and Data Mining in Biomedicine -NoE 507505.