{'fraction_non_alphanumeric': 0.023604861997179503, 'fraction_numerical': 0.007454166946477738, 'mean_word_length': 5.04976640260004, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
CLINIQA: A Machine Intelligence Based Clinical Question Answering System

The recent developments in the field of biomedicine have made large volumes of biomedical literature available to the medical practitioners. Due to the large size and lack of efficient searching strategies, medical practitioners struggle to obtain necessary information available in the biomedical literature. Moreover, the most sophisticated search engines of age are not intelligent enough to interpret the clinicians' questions. These facts reflect the urgent need of an information retrieval system that accepts the queries from medical practitioners' in natural language and returns the answers quickly and efficiently. In this paper, we present an implementation of a machine intelligence based CLINIcal Question Answering system (CLINIQA) to answer medical practitioner's questions. The system was rigorously evaluated on different text mining algorithms and the best components for the system were selected. The system makes use of Unified Medical Language System for semantic analysis of both questions and medical documents. In addition, the system employs supervised machine learning algorithms for classification of the documents, identifying the focus of the question and answer selection. Effective domain-specific heuristics are designed for answer ranking. The performance evaluation on hundred clinical questions shows the effectiveness of our approach.

This reflects the need of a system that can intelligently interpret the clinicians' questions and retrieve the appropriate snippets from the document collection.
Evidence-based medicine has emerged as a widely accepted paradigm for medical practice, which emphasizes the importance of evidence from patient-centered clinical research in the health care process. It refers to the use of evidence from scientific and medical research in making decisions related to the care of an individual patient. Therefore, medical practitioners are urged to practice evidence based medicine when faced with the questions about patient care [11,12]. Due to the pressurized atmosphere of busy practice, clinicians more likely seek answers from readily available sources [13]. Although there is a large amount of information available to the clinicians through different sources such as digital libraries, electronic journal, and WWW, they often face difficulties in finding answers to their questions about a specific clinical problem [14].
A study revealed that the average time required in obtaining an adequate answer to the physician's question ranged from 2.4 to 6.5 minutes [15]. In another study conducted with a test set of 100 medical questions collected from medical students in a specialized domain, a thorough search in Google was unable to obtain relevant documents within top five hits for 40% of the questions [16]. Moreover, due to busy practice schedules physicians spend less than 2 minutes R. C. Joshi (e-mail: joshifcc@iitr.ernet.in) G. Atluri (e-mail: atlurpec@iitr.ernet.in).
on average seeking an answer to a question. Thus, most of the clinical questions remain unanswered [17]. The aforementioned studies clearly indicate the urgent need of a system which answers clinicians' questions effectively and quickly related to patient care.
The existing information retrieval systems usually fail to provide satisfactory results due to domain specific nature of the biomedical text, which comprises of complex technical terms and their inconsistent use in the domain [8]. For example, there are a large number of long multiword expressions such as "Melanogaster 5`-phosphoribosylaminoimidazole carboxylase-5'-phosphoribosy1-4-(N-succinocarboxamide)-5-aminoimidazole synthetase (Ade5)mRNA, complete cds". Moreover, the term order variations and abbreviations are common in this field [9]. Beside these domain-specific characteristics, there are many challenges with respect to question answering, which form a clear demarcation between biomedical and open domain question answering systems (OQAS). The initial feasibility study on medical question answering systems, conducted by Nui et. al. [10], gives a good analysis of the limitations of OQAS techniques when applied to Medical Question Answering Systems (MQAS).
In this paper, we present an implementation of the intelligent Clinical Question Answering system developed for medical practitioners and for the users acquainted with medical terminology. Currently, the system works on the pancreatic cancer abstracts collected from PUBMED, but the architecture is robust and applicable to any disease and knowledge source.
The collected documents are classified and indexed based on the domain requirements. Similarly, the system classifies the clinical questions to obtain question focus and expected answer types. Simple yet effective strategies are developed for answer extraction and ranking.
Finally, the answer based on the rank is returned to the user.
The paper is organized as follows, in section II, we give a detailed taxonomy of CLINIQA, 4 where each component is explained and analyzed in detail. In section III, the system is evaluated on user effort and Mean reciprocal Rank. A survey of related work is presented in section IV.
II. CLINIQA: SYSTEM ARCHITECTURE There are four major components in CLINIQA, which are question classification, query formulation, answer extraction, and answer ranking. The architecture is shown in Fig. 1.
Classification and indexing of the documents are performed as a data preprocessing step, which significantly improve the retrieval performance, and are explained in section A. CLINIQA accepts natural language questions and parses them using MMTx [6], a programming implementation of MetaMap [25], which maps free text to their Unified Medical Language System (UMLS) [2] concepts and related semantic types. The obtained medical phrases and its similarity. The answer extraction module selects the documents that have relevance score or cosine value above a certain threshold, as discussed in section D.
Finally, the answer ranking module ranks the document and assigns scores to the sentences based on the co-occurrence of question terms and the required answer type. The answer ranking module is discussed in section E.
A. Data Preprocessing

1) Biomedical document indexing
To answer the user's questions quickly and effectively using a knowledge-base that scales to gigabytes of text, the data must be preprocessed and each document is to be represented by a set of keywords that describe the document. The unique property of the biomedical corpus is that, it contains a large set of multiword expressions which are out of the scope of general English language resources. Many methods have been developed for automatic indexing of biomedical documents [9,25]. Keeping in view the unique property of the biomedical corpus mentioned above, we index the documents based on both medical phrases and semantics.
The semantic indexing is useful in interpreting the content of the document. We used the MMTx program, which first parses text and then separates it into noun phrases. Each noun phrase is then mapped into UMLS concepts and each concept is assigned some weight where the highest weight represents the most likely concept the document represents. MMTx also assigns a semantic tag to each of the concepts. An example of the indexing used in the system is shown in Table I.

Abstract:
Pancreatic ductal adenocarcinoma is the most common pancreatic neoplasm. There are approximately 33,000 new cases of pancreatic ductal adenocarcinoma annually in the United States with approximately the same number of deaths. Surgery represents the only opportunity for cure, but this is restricted to early stage pancreatic cancer. Pancreatic ductal adenocarcinoma evolves from a progressive cascade of cellular, morphological and architectural changes from normal ductal epithelium through preneoplastic lesions termed pancreatic intraepithelial neoplasia (PanIN). These PanIN lesions are in turn associated with somatic alterations in canonical oncogenes and tumor suppressor genes. Most notably, early PanIN lesions and almost all pancreatic ductal adenocarcinomas involve mutations in the K-ras oncogene. Thus, it is believed that activating K-ras mutations are critical for initiation of pancreatic ductal carcinogenesis. This has been proven through elegant genetically engineered mouse models in which a Cre-activated K-Ras(G12D) allele is knocked into the endogenous K-Ras locus and crossed with mice expressing Cre recombinase in pancreatic tissue. As a result, mechanistic insights are now possible into how K-Ras contributes to pancreatic ductal carcinogenesis, what cooperating events are required, and armed with this knowledge, new therapeutic approaches can be pursued and tested.   Fig. 2(a) and Fig. 2(b) show the frequency of medical phrases and semantic tag in a sample of documents collected from the corpus.

Index
In this phase documents are indexed using unique medical phrases and semantic tags. The efficiency of the indexing is inversely proportional to the document length. As the length of document increases the size of the index will decrease. In a sample of 1700 abstracts, 9260 unique phases and 345 unique medical concepts are identified.

2) Term weighing and scoring
The system is assessed after assigning weights to each term in the document based on different weighing schemes. The weights are assigned to the terms based on the statistical properties of the terms in the documents. For example, Term Frequency (TF) is the measure of how many times a term has appeared in the document. The TF weighing scheme considers all the terms equally important, but this is always not true. Certain terms have no discriminating power but have more occurrences in the documents. The proper strategy to achieve this is to assign less weights to frequently appearing terms and high weights to rarely appearing terms. Inverse Document Frequency (IDF) follows this weighing scheme and can be measured as follows: gives RSV more than the other relevant documents as shown in Fig. 3(a) and Fig. 3(b) respectively. On the other hand, when normalization is used, the relevant documents have higher RSV as shown in Fig. 3 figure 2.

3) Document classification
The questions involving intervention and non intervention are the potential candidates for answering based on evidence from biomedical literature and other medical resources [6]. As only intervention and non-intervention question can be answered, we constrain CLINIQA's We compared the accuracy of five different classifies: SVM, Naïve Bayes, K-nearest neighbor, Decision Tree based J48, and Fuzzy classifier. A good discussion on these classifiers is given in [28]. A precise discussion of these classifiers is given below: be used for classification. On the other hand, if the data cannot be separated by a hyperpalne then the data can be projected into a higher dimensional Hilbert space and can be separated linearly in the higher dimensional projection space.
Let the training pattern be represented as ) , , where x is an element of feature space and y is an element of class space. Let Φ denote a nonlinear map H R l → Φ : , where H is a higher dimensional Hilbert space and l is the dimensionality of x. SVM construct the optimal separating plane in H. Therefore the decision function is of the form x a learning vector of size n, x is is a test vector, K is a kernel function, and i λ are nonnegative lagrange multipliers associated with the quadratic optimization problem: where w and θ are the parameters of the optimal separating plane in H. Whereas, D is a parameter which penalizes the error and i ξ are positive slack variables.
In this classification paradigm, k-nearest neighbors of a training data is computed first. Then the similarities of one sample from testing data to the k-nearest neighbors are aggregated according to the class of the neighbors, and the testing sample is assigned to the most similar class. One of advantages of KNN is that it is well suited for multi-modal classes as its classification decision is based on a small neighborhood of similar objects. So, even if the target class is multi-modal, it can still lead to good accuracy. We used 3 = k for our experiments, this value of k gives optimal results on the medical document and questions datasets.

c) Naive Bayes
Naïve Bayes (NB) is a probabilistic classifier. The basic idea in NB approaches is to use the joint probabilities of words and categories to estimate the probabilities of categories given a document. The naïve part of NB methods is the assumption of word independence, i.e., the conditional probability of a word given a category is assumed to be independent from the conditional probabilities of other words given that category. This assumption makes the computation of the NB classifiers far more than the exponential complexity of non-naïve Bayes approaches because it does not use word combinations as predictors.
Suppose there are m classes, Given an unknown data sample X , the classifier will predict that X belongs to the class having highest posterior probability, conditioned on X .
That is, the naïve Bayesian classifier assigns an unknown sample X to the class i C if and only if is maximized is called maximum posteriori hypothesis. By Bayes Theorem, 13 ) ( Given a data set with many attributes, it would be computationally expensive to . In order to reduce computation in evaluating ) / ( i C X P , the naïve assumption of class conditional independence is made. This presumes that the values of the attributes are conditionally independent of one another. Thus A decision tree is a simple structure where non-terminal nodes represent tests on one or more attributes and terminal nodes reflect decision outcomes. The basic algorithm for decision tree is greedy and constructs the tree in top-down recursive divide and conquer manner. The basic strategy is as follows [27]: Decision tree algorithms are unstable. Slight variations in the training data can result it different attribute selections at each node within the tree. The effect can be significant since Basic Strategy for Decision tree: • The tree starts as a single node representing the training samples.
• If the samples are all of the same class, then the node becomes a leaf and is labeled with that class.
• Otherwise, the algorithm uses an entropy-based measure known as information gain as a heuristic for selecting the attribute that will best separate the samples into individual classes. • A branch is created for each known value of the test attribute, and the samples are partitioned accordingly. • The algorithm uses the same process recursively to form a decision tree for the samples at each partition. Once an attribute has occurred at a node, it need not be considered in any of the node's descendents. • The recursive partitioning stops only when any one of the following conditions is true: 1. All samples for a given node belong to the same class.  Table III.

B. Question Classification
The question processing is the vital component of any question answering system, which is The hand crafted rules created for a specific domain cannot be used for other domains. The rules created for specific question ontology must be redesigned before being applied to different ontologies. To overcome these problems machine learning methods have been researched and applied successfully [26].
Crafting rules for question classification in biomedical domain is difficult due to the complex structure of the questions, which may include patient description, test finding and so on. Rule  words and structure is difficult. For example, the question [10] given below, contains patients description which increased the complexity of interpreting the answer type.
A medical question analysis by clinicians in [4][5][6]17] can be used for classifying the medical questions and identifying the question focus. These studies are based on the questions asked by medical students and clinicians during practice. As experts have analyzed them, they are sound enough to be used in an automated medical question answering system.
The questions whose answers incorporate clinical studies are considered as good questions, however, all other questions are considered as patient specific questions and cannot be answered [6,7]. We select a subset of questions as good questions from a set of 10 generic questions given in [5]. We identify the focus of each question with the help of experts. The set of questions and their focus is shown in Table IV.
Recently, the researchers have focused on machine learning techniques for classifying medical questions. Yu et al. [7] used different classifiers to classify a set of 200 questions to the medical question taxonomy defined in [6]. Their result shows that SVM's performance is better than the other methods with 59.5% accuracy when used in ladder approach, and less than 54% when Q: Do patients surviving an AMI and experiencing transient or ongoing congestive heart failure (CHF) have reduced mortality ad morbidity when treated with an ACE inhibitor (ex. Ramipril)?  Once the question classification is performed the medical phrases and semantics are directed to the query formulation module.

C. Query formulation
Query formulation module receives the medical phrases and semantic tags from the question classification module. We use vector space model due to its simplicity and high performance in TREC conference evaluation [29]. At this stage, the query formulation module assigns weights  The similarity between document and query is often measured using Cosine metric and is As the term weights in Term Document × matrix are assigned using Normalized term frequency scheme, and binary weights are assigned to the query terms (normalizing then will not affect the result), the cosine measure for similarity can be modified and computed as follows: We use same similarity measure for semantic tags. Finally, the measured similarities on both medical phrases and semantics are given as input to the answer extraction module.

D. Answer extraction
The answer extraction module accepts two sets of similarity vectors, corresponding to medical phrases and semantics as input, and identifies a set of abstracts that fall above certain threshold. A consensus between two similarity vectors is drawn by a simple heuristic. As a first step, the common documents to both the vectors are selected. In second step, the similarity scores of both medical phrases and semantics are added to get a new similarity score. Here, the bias can be incorporated while adding two similarity measures. For example, the semantic tags represent the generic classes; therefore they can have higher bias. Currently, we use simple addition of two similarity scores for simplicity without any bias to the vectors. The top ten abstracts are selected as the potential candidates for answering the user's question. The selected candidates are given as input to the answer ranking module, which ranks the candidates and returns the most relevant abstract to user.

E. Answer ranking
The answer ranking module is responsible for the ranking of the candidate abstracts based on sentence scores. The scores are assigned to sentences using question focus obtained from question classification module. There are two advantages of using question focus: (1) ranking of the answers to assign the top rank to the most relevant abstract, and (2) elimination of the abstracts which are selected by the answer extraction module due to high frequency of the general question words.
We rank the answer based on the simple heuristics as discussed below. The method first sets the flag associated with each sentence of the abstract if it contains the question focus, otherwise, the flag is assigned a zero value. In the second step, the sentences with flags having values 1 are investigated for the percentage of matching question words in the sentence. The percentage of matching question words is then assigned as the score of the sentence. Finally, each sentence score in the abstract is added to obtain the abstract score.
In case of a tie between the abstract scores, an abstract which contains a sentence with maximum sentence score is given the priority. When there is a tie between the abstract scores and sentence score then one of the abstracts is arbitrarily selected for the better rank, and the other abstract for the subsequent rank.

III. PERFORMANCE EVALUATION
In this section we evaluated CLINIQA based on mean reciprocal ranking and user effort measures and also compared the performance of the CLINIQA using different term weighting schemes. The evaluation results of three components, indexing and term weighting, document classification and question classification have been already presented in the previous section.
Very little work [30,31] has been published focusing the evaluation methods for restricted domain question answering systems. Diekema et al. [30] described the factors other than TREC evaluation to assess the restricted domain question answering.
In the following section A, we discussed the evaluation criteria used to assess CLINIQA and using different weighting schemes. Section B describes the data set used for the experiments and in section C we present a comparative study of different information retrieval systems with respect to medical domain.

A. Evaluation metrics
We used two metrics to evaluate the system, user effort and mean reciprocal ranking suggested in TREC [32] for the assessment of question answering systems.
Traditional information retrieval systems use the precision and recall to assess the retrieval performance. Precision is the percentage of retrieved documents that is relevant and recall is the percentage of relevant documents retrieved. The precision can be viewed as the measure of the effort user has to extend to find the relevant documents and recall measures how efficient the system is in locating the relevant documents. In question answering the recall can be defined as the percentage of questions answered correctly from the test set [33].
As we do not use web for extracting candidate answers the user effort function is slightly different from the one used in [33]. Let Mean reciprocal rank is used to compute the overall performance of the question answering system and is defined above as MRR.
We evaluated CLINIQA on both user effort and MRR measures and the results are given in section B.

B. Experimental evaluation
To evaluate the system we used 1700 abstracts related to pancreatic cancer collected from PUBMED and 100 questions collected from clinicians and cancer discussion groups. The questions were distributed over the four classes, each of which identifies a unique question focus, as shown in section 2B. Although the questions were selected from different sources we made sure that they were answerable using our document collection.
The document collection was then classified manually by the human experts into three categories, non evidence based, intervention and non-intervention based documents. The class distributions for document collection over the three classes are distributed as, 420 intervention documents, 250 non-intervention documents and the remaining documents were classified as non-evidence based documents.
As a first step, we evaluate the CLINIQA based on user effort with different weighting strategies such as TF, IDF, normalized term weights and ranking strategy. The experiment shows the superiority of the CLINIQA (with normalized weighting and abstract ranking strategy) over the other term weighting strategies without ranking. CLINIQA answered approximately 85%. Fig. 6 shows that the 36% of the questions were answered with user effort of 100 words using CLINIQA, whereas the 30% of the questions were answered by CLINIQA without ranking at the same user effort. In general, reading nearly 100 words take less than a minute by an average reader without any special reading skills. Approximately 40% of the questions are answered in the first abstract. Answers to the 50% of the questions is obtained in the first two abstracts, the remaining 35% of the questions answered at user effort ranging from 600 to 1200 words. An interesting study on reading capabilities reveal that an average person reads 280 words per minute [34] and another study reveal that physicians spend less than 2 minutes on average seeking an answer to a question [17]. The results discussed above indicate that CLINIQA offers a better solution for clinicians.
The evaluation of CLINIQA based on mean reciprocal ranking (MRR) is shown in Fig. 7. The results indicate that the ranking strategy helps in promoting the most relevant answer to the best rank, which in turn reduces the user effort, discussed above. CLINIQA ranks approximately 70% Fig. 6. Recall vs. User effort: when phrase frequency, inverse document frequency, normalized weights are used and normalized weights with abstract ranking strategy. Fig. 7 The MRR results for CLINIQA with phrase frequency, inverse document frequency, normalized term weights, and normalized term weights with ranking. of the total questions answered at the top most rank whereas CLINIQA without ranking candidate answers approximately 60%.
IV. RELATED WORK In recent years, the research in the area of medical question answering has received a great deal of interest. Most of the publications till date present an overview of the biomedical question answering system. One of early papers on biomedical question answering by Zweigenbaum [18] gives an overview of the system and reveals the different linguistic and knowledge sources available for the domain. Different strategies have been proposed for answering clinical questions such as, answering by role identification [10,19,20], and answering based on document structure [21].
The systems based on role identification accept queries in a PICO format, renowned in evidence based medicine [22]. In this format, a clinical question is represented by a set of four fields that correspond to the basic elements of the questions: P represents the description of patient, I represents intervention, C represents a comparison or control intervention and O represents the clinical outcome. Demner-Fushman et al. [20] worked on the extraction of PICO frame elements from MEDLINE abstracts. This cannot be considered as fully automated question answering system as format for questions is predefined, similar to any other query language. Another difficulty with the systems is that all the questions cannot be formatted as PICO [6].
Sang et al. [21] proposed an offline strategy for answering clinical questions. This was developed and evaluated for non-expert users. They did not consider diagnostic questions. Hence most of the questions were answerable from medical encyclopedias and FAQs. The system is simple and the strategies are not suitable for evidence-based medicine.
Significant contribution to individual components of the biomedical question answering has been done in recent years. A very effective and useful analysis of biomedical questions is given in [4][5][6]17] by the physicians. In [4,5,17], Ely et al., presented the significance of the medical question answering system, and proposed a list of generalized medical questions based on a survey conducted on medical students and practitioners. They also presented good list of obstacles in answering medical questions, from a physician's point of view. Interestingly, to the best of our knowledge, no biomedical question answering system has used the analysis presented in [4][5][6]17]. Recently, an effort is made by Yu et al. [5,6]  The MEDLINE database is one of the most reliable sources for knowledgebase to answer the medical questions and is often used by the clinicians [23]. Studies have shown that the existing systems for searching MEDLINE are often inadequate and unable to supply clinically-relevant answers in a timely manner [24].
To the best of our knowledge CLINIQA is the first machine intelligence based clinical question answering system, whose components are thoroughly evaluated to arrive at a highly accurate system. It classifies the questions based on generic classification of medical questions proposed in [4][5][6]17]. It makes use of UMLS for interpreting the user question and indexing medical documents. Finally, the PUBMED abstracts are used to extract and locate the most relevant answers. Further more, the architecture is robust and can used with any other medical literature knowledgebase related any other disease. We used UMLS for interpreting the medical documents and UMLS is designed for all kind of medical literature made CLINIQA robust for any medical knowledge source and disease.

V. CONCLUSION
In this paper, we presented a novel implementation of machine learning based clinical question answering system. In this effort we tried to assist clinicians in evidence based information retrieval, by using authenticated and proved medical literature resources like PUBMED. The best machine learning tools were employed for the classification of document and questions. The system relies on the clinicians' study for question analysis. Effective answer ranking strategies are employed to reduce the user effort. CLINIQA is rigorously evaluated from the user's point of view and the performance point of view of supervised machine learning techniques. The results indicate that CLINIQA helps clinicians to obtain answers to approximately 50% of questions with a user effort of approximately 1 minute. These results support CLINIQA for clinicians' use in daily practice.