{'fraction_non_alphanumeric': 0.030553586896606033, 'fraction_numerical': 0.013071895424836602, 'mean_word_length': 5.0999039385206535, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Corpus-based Approach to Creating a Semantic Lexicon for Clinical Research Eligibility Criteria from UMLS.

We describe a corpus-based approach to creating a semantic lexicon using UMLS knowledge sources. We extracted 10,000 sentences from the eligibility criteria sections of clinical trial summaries contained in ClinicalTrials.gov. The UMLS Metathesaurus and SPECIALIST Lexical Tools were used to extract and normalize UMLS recognizable terms. When annotated with Semantic Network types, the corpus had a lexical ambiguity of 1.57 (=total types for unique lexemes / total unique lexemes) and a word occurrence ambiguity of 1.96 (=total type occurrences / total word occurrences). A set of semantic preference rules was developed and applied to completely eliminate ambiguity in semantic type assignment. The lexicon covered 95.95% UMLS-recognizable terms in our corpus. A total of 20 UMLS semantic types, representing about 17% of all the distinct semantic types assigned to corpus lexemes, covered about 80% of the vocabulary of our corpus.


INTRODUCTION
Clinical research eligibility criteria specify who is eligible for a clinical research study and, later, to whom clinical study results can be applied. There is an increasing need to efficiently transform free-text clinical research eligibility criteria into computable formats to provide decision support for clinical phenotype extraction, clinical research participants screening, and evidence-based medicine. Sim et. al have developed an annotation tool [1] to encode eligibility criteria with standard terminologies via The Unified Medical Language Systems (UMLS) [2]. However, this method did not resolve the inherent ambiguities in the UMLS semantic network, where a term can be mapped to multiple concepts and semantic types. A lexicon is central to all forms of medical language processing. At present, there is no semantic lexicon for standardizing the encoding of clinical research eligibility criteria. Many approaches to developing medical lexicons have benefited from the UMLS knowledge sources [3,4,5]. Our goal was to extend Johnson's approach [3] to reduce the ambiguity in UMLS semantic type assignment during the development of a semantic lexicon for clinical research eligibility criteria automatically from UMLS resources. In the rest of this paper, we describe a pipeline architecture and corpus-based approach to creating a semantic lexicon for clinical research eligibility criteria using the UMLS knowledge sources. Figure 1 illustrates the steps and knowledge sources used at each step.   [7] and the Penn Treebank tag set [8] for part-of-speech (POS) tagging. All words tagged as nouns, verbs, adjectives or adverbs were considered content words, which potentially had semantic types in the UMLS Semantic Network. In contrast, function words such as "the", "or", etc., numerals and operators (e.g., > or < ) do not have UMLS semantic types.

METHODOLOGY
MRCONSO contains a small range of lexical variants for lexemes. Matching against these variants is managed through the use of Specialist Lexicon Tools [9], which can be used to reduce lexical variation for general English text, providing normalized strings for a wider range of variants, including tense, number, and part-of-speech variants.
To take advantage of potentially expanded coverage through use of the Specialist Lexicon, we built a version of MRCONSO in which UMLS CUIs were mapped to these normalized strings. Input lexemes were normalized before checking against this version of MRCONSO. Strings in the base version of MRCONSO that did not have normalized terms in the Specialist Lexicon were used in their original form. Although the word vocabularies contained in the UMLS Metathesaurus and Specialist Lexicon were not identical, we identified and assigned semantic types for over 90% of all lexemes in our corpus. The MRSTY table of the UMLS contains semantic types defined for each CUI in MRCONSO. We mapped input lexemes to these CUI's as above and looked up their semantic types in MRSTY. We call lexemes successfully annotated with semantic types as semantic terms, or simply terms. The majority of terms were associated with a single semantic type.
However, many had multiple types, resulting in ambiguity (See table 1). When using the UMLS, one general source of ambiguity stems from the fact that the Semantic Network [10] is an ontology intended to cover medicine as a whole, including both medical science and clinical medicine. For example, the word "prednisone" has at least two senses, one describing a steroid chemical with a certain structure, and one describing a pharmaceutical medicine. The Semantic Network provides two types associated with these two senses, namely Chemical Viewed Structurally and Chemical Viewed Functionally. The sense of pharmaceutical medicine might be expected to be more appropriate in clinical text. Using a corpus of hospital discharge summaries, [3], this was verified. Many such cases were examined and, through a manual process of textual analysis, a set of hand-crafted preference rules was developed. Preference rules have the form: if TYPE-A (or any of its descendants in the UMLS Semantic Network) and TYPE-B (or any of its descendants) are specified for a given lexeme, then retain TYPE-B (or any descendant) and discard TYPE-A (or any descendant). For example, the lexeme "beta Hydroxyphenethylamine" is assigned the types Pharmacologic Substance and Organic Chemical. Pharmacologic Substance is a descendant of Chemical Viewed Functionally. Organic Chemical is a descendant of Chemical Viewed Structurally. Given preference for Chemical Viewed Functionally (the clinical sense) over Chemical Viewed Structurally (the biological sense), the type Pharmacologic Substance would be retained, and the type Organic Chemical would be discarded, for this lexeme. Preference rules can be formulated at any desired level of generality allowed by the Semantic Network. Table 2 shows 5 examples of frequently applied semantic preference rules. Stage 3: "Lexical Analysis". We investigated the coverage of the sample corpus provided by our annotation procedure, using the Metathesaurus, Semantic Network, and preference rules. Results are described in the next section.

Coverage
The corpus contained a total of 74,188 text tokens, including all content words and other text tokens. The average sentence length was 7.41 (text tokens). There were 47,129 content words (See Table 3). Of these, 15.56% were multiple-word lexemes and 84.43% were single-word lexemes. 95.95% of content words were assigned at least one semantic type. 4.05% were not assigned any type; all of these were single-word lexemes. In the corpus, there were 6,921 unique content words, 90.02% of which were assigned at least one semantic type and 9.08% of which were not assigned any type. We also examined coverage of the corpus vocabulary by individual semantic types. Table 4 lists the top 20 types in terms of percent of occurrences of corpus lexemes assigned the individual type. This set of the top 20 types represents 17.9% of the 117 unique types applied to the corpus, but covers 80.6% of the corpus vocabulary. These types can therefore be considered as the primary semantic classification of our randomly selected sample of eligibility criteria text, when analyzed through use of the UMLS ontology.

Disambiguation
Using the preference rules, 117 out of 134 semantic types were applicable to the corpus. 22,878 input content words had multiple semantic types and were processed by the preference rules. 24,251 content words were not processed by any preference rule, these being either singly-typed or non-typed content words in the text. The semantic types that were most frequently excluded by our preference rule were listed in Table 5. The table reflects the fact that the more concrete types are preferred to UMLS conceptual types wherever possible. Before applying preference rules, 88,594 semantic types were assigned to content words. After preference rules were applied, only 45,221 semantic types were assigned. Before applying preference rules, 2324 (33.57%) of unique content words had two or more types. After applying the preference rules, all these terms only had one UMLS semantic type each.

Non-content tokens in the corpus
All input tokens having POS tags classifying them as nouns, verbs, adjectives and adverbs were considered content words potentially having semantic types. The rest of the input text consists of what are traditionally referred to as function words, such as articles, prepositions, and others, as well as numbers, symbols, abbreviations, or units (See Table 6). Such input tokens still contain valuable semantics for interpreting eligibility criteria text. We will demonstrate the usage of both content words and noncontent words below.

EXAMPLES OF ANNOTATION
With the semantic lexicon, we can automatically annotate eligibility criteria sentences with unambiguous semantic types. We compared the performance of our annotation tool to MetaMap Transfer (MMTx 2.4C version) [11] as illustrated by the following two examples. The examples showed that our method produced finer-grained results than MMTx 2.4C. MMTx returned "such as serious cardia, renal" as a single constituent, which was questionable. In contrast, our annotation tool effectively decomposed the phrase into more granular semantic units: "such", "as", "serious", "cardiac", and "renal".

DISCUSSION
Previously developed lexicons had a coverage of 79% [3] for discharge summaries, and 77% [4] for non-clinical biological text. Our lexicon has 95% coverage of the vocabulary of our corpus of eligibility criteria. Approximately 80% of the corpus vocabulary was covered by only a small set of 20 types (17% of distinct occurring types). By contrast, Verspoor found a much smaller set of distinct types (3% of distinct occurring types) providing 77% coverage of his particular corpus. It can be seen that medical text varies considerably in the breadth of its vocabulary (affecting coverage by resources like the UMLS Metathesaurus), and in the specificity of its semantics (after the vocabulary is reduced to its conceptual content). We compared our annotations with MMTx mainly because MMTx is a widely used, general-purpose tool for UMLS-based semantic annotation, providing many conveniences and options. It would be possible to obtain more precise results from MMTx by passing it more detailed word strings (i.e. lexemes, as we have defined them) rather that full sentences. But this would require prior implementation of the lexemeidentification methods employed in this paper. Use of MMTx would then be redundant, because lexemes and their types can be directly looked up in MRCONSO and MRSTY.

CONCLUSION AND FUTURE WORK
We developed an annotation procedure which provides a UMLS-based, unambiguous semantic lexicon with 95% coverage for a random sample of eligibility criteria text (10,000 sentences). We also identified 20 semantic types defined by UMLS that can serve as a preliminary classification of terms in eligibility criteria text. These observed restrictions on type occurrence suggest that there are specific semantic constraints operating in the language of eligibility criteria text that can be studied further.
As part of our future work, a sublanguage of clinical research eligibility criteria will be explored, wherein only certain restricted sentence types and predica-tions can be expected to occur. This would further aid the development of procedures for extraction and standardization of eligibility criteria. We will also study whether the UMLS type classification that we have observed is optimal for developing a standard ontology for eligibility criteria. The ultimate goal is construction of an automated extraction procedure mapping raw text to a standards-based formal structure for eligibility criteria. Other research has been done in the area of eligibility criteria modeling [12]. Semantic classes highly specific to eligibility criteria have been defined, such as Assessments (of a patient), Interventions (performed on a patient) and Behavior (of a patient). These are entered into templates summarizing the criteria for a research study [13]. It may therefore be necessary to better align UMLS classes to those of optimal models. For example, the UMLS types Laboratory Procedure, Organism Attribute, Health Care Activity and Organism Function may all map to the class Assessments. This may provide a more concise representation. Mapping of UMLS types to models is also important because the UMLS Metathesaurus remains a crucial resource for text-based extraction procedures.