{'fraction_non_alphanumeric': 0.0315658346260518, 'fraction_numerical': 0.006745640013162225, 'mean_word_length': 4.784772263766145, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Interpretable Multi-Step Reasoning with Knowledge Extraction on Complex Healthcare Question Answering

Healthcare question answering assistance aims to provide customer healthcare information, which widely appears in both Web and mobile Internet. The questions usually require the assistance to have proficient healthcare background knowledge as well as the reasoning ability on the knowledge. Recently a challenge involving complex healthcare reasoning, HeadQA dataset, has been proposed, which contains multiple-choice questions authorized for the public healthcare specialization exam. Unlike most other QA tasks that focus on linguistic understanding, HeadQA requires deeper reasoning involving not only knowledge extraction, but also complex reasoning with healthcare knowledge. These questions are the most challenging for current QA systems, and the current performance of the state-of-the-art method is slightly better than a random guess. In order to solve this challenging task, we present a Multi-step reasoning with Knowledge extraction framework (MurKe). The proposed framework first extracts the healthcare knowledge as supporting documents from the large corpus. In order to find the reasoning chain and choose the correct answer, MurKe iterates between selecting the supporting documents, reformulating the query representation using the supporting documents and getting entailment score for each choice using the entailment model. The reformulation module leverages selected documents for missing evidence, which maintains interpretability. Moreover, we are striving to make full use of off-the-shelf pre-trained models. With less trainable weight, the pre-trained model can easily adapt to healthcare tasks with limited training samples. From the experimental results and ablation study, our system is able to outperform several strong baselines on the HeadQA dataset.


INTRODUCTION
Neural network models have achieved great success on the recent progress of question answering (QA). In some of the popular datasets, such as SQuAD [32] and bAbI [43], the machine can achieve near human-level performance. However, these datsets are easy for machine since the context contains the answer and often surface-level knowledge is sufficient to answer [44]. The recently released HeadQA [39] is an ambitious test for AI systems. This dataset consists of 6,765 multiple choice questions authored for college students in the healthcare area to have the specialization license. The dataset contains question (containing context) with four or five candidate option choices from 6 categories including Medicine, Pharmacology, Psychology, Nursing, Biology and Chemistry. A small percentage (∼ 14%) of the Medicine questions refer to images, that provide additional information to answer correctly. These questions require sophisticated reasoning and language understanding abilities to be answered correctly, and even for humans (i.e., medical college students), these questions take over a period of one year or more for them to pass the exam.
Compared to basic reading comprehension based QA setup where the answers to a question are usually found in the given small context, the HeadQA setup needs to extract relevant knowledge according to the context and question. Another characteristic of this dataset is that unlike current datasets like TRIVIAQA-open [18], SQUAD-open [32] and ARC [8], the gold document and relevant search document set are not provided for each question. This makes HeadQA represent a unique obstacle in the QA as the system now needs to search for relevant documents from the whole Wikipedia corpus. The performance of current state-of-the-art methods is only slightly better than random guess [39]. The performance degradation is mainly from failing to retrieve the relevant documents for the question answering model [16].
In previous works, single-step retrieve-and-read question answering (QA) systems [5] failed to perform well on complex questions dataset [8,47] as the question does not contain sufficient retrievable clues and, thus, all the relevant context cannot be obtained in a single retrieval step [31]. The recently popular multi-hop based QA systems, like the HotpotQA [47] and WikiHop [42], are designed such that they require to reason with information taken from more than one document in order to arrive at the correct answer. The reasoning chains in HotpotQA are well-designed by human; specifically, these datasets assume supporting documents are already obtained and the reasoning chain is generated by human along with the dataset. However, the HeadQA is a more natural dataset as it is collected from the healthcare specialization exam, so the reasoning chains are unknown and the model needs to extract the relevant healthcare knowledge by itself.
The above mentioned differences lead to multiple challenges for HeadQA. First, finding the relevant supporting documents from the large corpus, like Wikipedia is a challenge, especially since standard IR approaches can be misled by distractions. Second, finding the multi-hop reasoning chain among [47] the plentiful documents is another challenge, since their reasoning path is unclear and thus would need to have a well understanding between the natural language texts. To solve these challenges, our proposed model, Muti-step reasoning with Knowledge extraction framework (MurKe), solves this problem in two steps: extract relevant knowledge and reason with the background knowledge. The relevant knowledge extraction aims to narrow the document search space from the whole Wikipedia to the relevant document set by using a combination of token-level and semantic-level retrieval. In the reasoning step, it is possible that the answer might not be present in the initially selected documents or that the model would need to combine information across multiple documents [25]. Thus, we extend the single-step retrieval to multi-step iterative selection, reformulation and entailment module. Given an input question, the selection module finds the most relevant document with the current question. The selected relevant document is sent to the reformulation module which aims to find the guided clue from the selected document and reformulate the current question. For this purpose, we use a reformulation module that is equipped with extractive reading-based attention to reformulate the question. The important pieces of the selected document are highlighted by what we call a reading-answer attention and integrated into a representation of the question via our reformulation module. This new question vector is then used by the selection model to re-rank the context. In this way, it allows the model to select new documents and combine evidence across multiple documents, which could provide the interpretability of the reasoning path. Moreover, the input of the reformulation and entailment module is the same and they can be processed in parallel. Therefore, our method is still efficient, even if the method needs to do iterative several steps.
The main contributions of this paper are: (a) a combination of token-level retrieval and semantic-level retrieval to settle down the search space as a small but sufficient document set, (b) an efficient and effective iterative retrieval-reformulation-entailment framework capable of complex healthcare reasoning, (c) a natural language question reformulation approach that guarantees interpretability in the multi-step evidence gathering process, and d) we illustrate the advantages of our model on the HeadQA dataset.

RELATED WORKS 2.1 Question Answering with Knowledge Extraction
Performing question-answering need knowledge extraction setting is far more challenging than its counterpart closed-domain setting as in the latter case the answer can be extracted from a preselected passage [41]. For example, although the recently released AI2 Reasoning Challenge (ARC) dataset [8] contains science-related questions, which also require powerful knowledge and reasoning, it has accompanying ARC corpus with relevant science sentences. As a result, in comparison to HeadQA, it is easier to find answers to ARC questions as the former requires large-scale search to find supporting documents, alongside a reading comprehension module to generate the answer. QA with knowledge extraction originally found answers in the large corpus of unstructured texts [5], but over the years many works have explored QA from knowledge bases (KB) such as Freebase [3] or DBPedia [1]. However, the main drawbacks of KBs are that they are incomplete as well as expensive to construct and maintain [16]. This has rendered free-corpus such as Wikipedia as the preferred choice for knowledge source to provide additional evidence in answering questions, not to mention that it also provides up-to-date information [5]. Most pipelines base their information retrieval module returning the relevant documents on standard IR mechanisms (e.g., TF-IDF), which can fail to contain the correct answer in the ranked documents [24].

Multi-Step Datasets and Reasoning
Instead of getting the answer from a single context, the questions in the multi-step datasets need to locate multiple contexts to get the answer. [18] developed TriviaQA containing question-answer pairs with several associated evidence documents, which requires inference over multiple sentences to answer correctly. Answering questions in the bAbI dataset [43] requires combining multiple disjoint evidence in the context, however, as the text is synthetic, it fails to completely resemble the complexity of passage structures in human-generated texts [2]. The WikiHop dataset [42] requires more than one Wikipedia document to answer. More recently, the HotpotQA dataset [47] has gained traction in this direction. It contains crowdsourced questions with more diverse syntactic and semantic features [17]. A sequential approach is followed by Memory Network-based models to iteratively store the information gathered from passages in a memory cell [22,34,38]. Works by [4,11,36] use graph convolutional network [21] to do multi-hop reasoning.
Reasoning chains fed into a BERT-based QA model is proposed by [6]. Although much progress has been made with large-scale reasoning datasets [40,46], these datasets contain gold document contexts corresponding to each question. When it comes to performing multi-step reasoning they still lag behind in performance [47]. Compare to those datasets, HeadQA is more difficult due to it does not have any relevant gold documents for each question and the reasoning chain is unknown.

Query Reformulation
One direction of query reformulation works on reformulating queries by rewriting the query or only retaining their most salient terms. By selecting important terms from the retrieved document, Nogueira et al. [28] uses reinforcement learning to reformulate the query to maximize the number of relevant documents retrieved. Beyond just selecting important terms, work by [26] refine the query in a well-formed way. In the multi-choices question answering domain, [27] use a sequence to sequence model to retain the most salient term and use an entailment model to set scores to each choice. Instead of reformulating the explicit query, another direction works on reformulating the latent query vector. Work by [30] showed that query refinement is effective for IR in the bio-medical domain. [10,13] turns the query reformulation to the multi-step setting, that retrieval and reader model iterative work. The reader model sets the document latent representation and uses that to reformulate query latent representation. Our work is in the latent vector reformulation direction which is more flexible and also contains the interpretation.

TASK DEFINITION
In complex healthcare reasoning, we are given a question Q containing m tokens Q = [q 1 , q 2 , ..., q M ] with a context description C = [c 1 , c 2 , ..., c L ] in the question where L < M. In some cases, an image I is provided which contains information related to the question Q. The option set has h candidate option choices O = The goal is to select the correct answer A from the candidate option set. For simplicity, we denote X = {Q, O, I} as one data sample and denote y = [y 1 , y 2 , . . . , y h ] as a one-hot label, where each y i = 1(O i = A) is an indicator function. In the training, N sets of (X, y) N are given and the goal is to learn a model f : X → y. In the testing, we need to predict y test given test samples X t est .
We observe that the context itself is unable to provide enough clues to the correct answer. Hence, we seek to bring supporting knowledge for each data sample from the open knowledge base, like Wikipedia. In this work, our proposed MurKe model first extracts the supporting document as the background knowledge, and then finds the reasoning path among them. MurKe extracts the question-related supporting documents set D N = {D 1 , ..., D K } from the Wikipedia corpus D 1 . Note that the extracted relevent document set D N comes from a large corpus of documents D, where |D | ≫ |D N |. Then the document set D N is used as external background knowledge to predict the answer option O i ∈ O. By using the supporting document from the first part, we design the iterative selection, reformulation and entailment models to find the latent multi-step reasoning chain.

THE PROPOSED FRAMEWORK (MURKE)
Since there are no correct search documents for each question in the HeadQA dataset, to get the background knowledge the model needs to search the supporting documents from the whole Wikipedia. However, this is computationally expensive and time-costing. To solve this problem, we need to settle down the space of the supporting documents such that they can cover the information of the question as much as possible, while keeping the size of the supporting document set acceptable enough for downstream processing.
After getting the supporting documents, MurKe mimics how humans answer the complex question using background knowledge. Namely, based on the question and extracted supporting knowledge, humans first search one relevant background knowledge and decide whether the current background knowledge could answer the question. If it could, they get the answer.

Knowledge Extraction
To provide supporting documents for each healthcare question, we first introduce an effective and efficient preprocessing method to narrow the supporting document space from the whole Wikipedia to the relevant document, as shown in Fig. 1. Since the document scale is few hundred million, only using the semantic-level retrieval is both computation-costing and time-costing. Therefore, we use the token-level retrieval first to settle down the document number, and then use the semantic-level retrieval to further select the relevant documents. A question-based document ranking approach is employed to retrieve the most relevant supporting documents to a given question from a knowledge source Wikipedia D. Since all the questions in our dataset are healthcare science-related, we filter the documents as the categories of depth 4 under "Health" topic 2 . We refer to this corpus of extracted Wikipedia documents as the "WikiHealth" corpus D H .

Token-level Retrieval
To begin with, we use a combination of the neural keyword matching method and TF-IDF method to narrow down the search document scope from "WikiHealth" corpus D H down to a set of question related documents D I . The focus of this step aims to efficiently select a candidate set that can cover the information as much as possible while keeping the size of the set acceptable enough for downstream processing.
Specifically, following Musa et al. [27], we treat the key-term selector as an encoder-decoder model, which inputs the question The BM25 scoring mechanism 3 is then used to retrieve the top-100 (tested in the experiment) possibly relevant documents to each question from the WikiHealth corpus D H . The search document space of question Q is the unit document of the retrieved documents from the h queries, represented as D I .

Semantic-level Retrieval
After obtaining the related documents D I using token-level retrieval, we seek to use the semanticlevel retrieval to further narrow down the documents. The outputs of the neural model are treated as the relatedness score between the input question and the documents. The scores will be used to sort and limit all the upstream documents. This step, as shown in Fig. 1, aims to screen D I to the semantic-level related documents D N , which can be helpful for the downstream modeling. From these retrieved documents, to find the document that is the most semantically related to each question, we use the language model BERT [12] which is pretrained on biomedical corpora, namely BioBERT [23]. We use the special token to concatenate the question and document together as the input to the BERT model: We applied an affine layer and sigmoid activation on the last layer output of the [CLS] to get the scalar value. Subsequently, the documents of each question whose value is above the document relevance threshold th r is considered as the search documents input to our model. In our experiment, we set the threshold th r as 0.9. In the end, we can get the semantic-level related documents D N for each of the search question Q.

Iterative Multi-Step Reasoning
After getting the supporting documents D N , in this section, MurKe proposes three modules -selection module, reformulation module and entailment module -which work iteratively to find the latent multi-step reasoning path. The reasoning diagram of the MurKe framework is shown in Fig. 2. Specifically, MurKe contains three sub-networks: Selection module first computes a relevance score for each document with regard to a given question and ranks them according to the score. The top one document is sent to both reformulation and entailment module. Reformulation module uses reading-answer attention to extract relevant information from top one selected document in order to update the latent representation of the question. At the same time, entailment module computes the entailment score of each candidate option and the selected document to get the final answer. Since the update of the reformulation conditions on the result of the selection module and the reformulated question can help get the confidence of the candidate choices in the following entailment model, this provides a way for the multi-step interaction between search engine (selection) and the matching model (entailment) to communicate with each other. Moreover, the model is processing very fast as the entailment model and reformulation model can be processed in parallel.

Selection Module.
The selection module computes a relevance score between each related document and the given search question, which is represented in Fig. 3 a). The related document representations are computed independently of the question and once computed, they are not updated. The relevance score of a related document is computed as an inner product between the related document and the question vectors. The related document and question representations are computed as follows.
in the relevant document set D N of question P consisting of N tokens, a bidirectional multilayer GRU (BiGRU) [7] encodes each token in the document [d 1 , d 2 , is the concatenation of the forward and backward GRU last layer hidden units. The question Q = [q 1 , q 2 , ..., q M ] with M tokens is encoded by another network with the same architecture to obtain the question embedding [q 1 , q 2 , ..., q M ] = BiGRU([q 1 , q 2 , ..., q M ]). To solve the long-term dependencies in the document, we compute the probability distribution α j depending on the degree of relevance between word and the other words (in its context). The self-attention document vector E D ∈ R N ×2d is computed as a weighted combination of all contextual embeddings: where w ∈ R 2d and W s ∈ R 2d ×2d , used in the bilinear term, is a learned weight matrix. In the same way, we calculate the Figure 3: The different building blocks of the proposed end-to-end trainable model. R M×2d as the question embedding. As some queries may contain image I, so we fuse the hidden states of question embedding and the corresponding image embedding to the initial question vector as U (0) = Fuse(E Q , E I ). We tried different fusion methods, such as con(), avg() and use bilinear module bil(), which are discussed in the experiment section.
The relevance score of a document with regard to the question (score(U (t) , E D )) is computed by a simple inner product ⟨U (t ) , E D ⟩, where t means t-th iteration. The document retriever ranks all the documents in D N and sends the embedding of top-1 scoring document to the following modules, reformulation and entailment. For notation simplicity, we use V (t ) instead to refer to the top-1 scoring document E

Latent Question Reformulation
Module. The latent question reformulation aims to find some evidence from the selected document so that combining the evidence with the current question representation formulates a new representation that can help answer more correctly, as shown in Fig. 3 b). The reformulated question is sent back to the entailment and retriever, which uses it to calculate the entailment score between hypothesis and premise and re-rank the documents in the corpus, respectively. More formally, a reformulation module takes the encoding of the top one selected document from the previous selection module, V (t ) , and the previous representation of the question, U (t ) , as input, and produces an updated reformulation of the question U (t +1) . Moreover, in order to provide the interpretability of the model, we want to extract sub-phrase of the document which can bring the guided clue for the current question. Therefore, we formulate it as a reading comprehension task [33] which aims to find the answer span in the document and use the found answer span to update the question.
Reading-answer Attention: The matching of stop words is presumably less important than the matching of content words. In this step, the goal is to compare the question embedding and the contextual document embeddings and select the pieces of information that are relevant to the question. We plan to learn the reading-based attention of a token as the probability that the predicted span has started before this token and will end after. Therefore, we calculate the question-aware document representation as j represents the j-th token in V (t ) and the W c is the training weight. Following [13], we use the idea of reader module to compute the reading-based attention vector. Given the question-aware document representation S(t), we compute the starting and ending index position probability p (t ) s and p (t ) e using two BiGRUs followed by a linear layer and a softmax operator. They are computed from: where w e and w s are trainable vectors of R 2d . The two probability vectors p (t ) s ∈ R N and p (t ) e ∈ R N are not used to predict an answer, but to compute a reading-based attention vector γ (t ) over the document. Intuitively, these probabilities represent at step t how likely each word is to be the beginning and the end of the answer span respectively. We define the reading-based attention of a token as the probability that the predicted span has started before this token and will end after, which can be computed as follows: Further, we use these attention values to re-weight each token of the document representation. We compute V i (t ) ∈ R N ×2d with: Dynamic Knowledge Extraction Max-Pooling: This layer aims at collecting the relevant evidences of V i (t ) with length N to add to the current question representation with length M. We partition the row of the initial sequence into M approximately equal parts. It produces a grid of M × 2d in which we apply a max-pooling operator in each window. As a result, a matrix of fixed dimension adequately represents the input, preserving the global structure of the document, and focusing on the important elements of each region. This can be seen as an adaptation of the dynamic pooling layer proposed by Socher et al. [35]. Formally, let V i (t ) be the input matrix representation, we dynamically compute the kernal size, w, of the max-pooling according to the length of the input sequence and the required output shape: w = ⌈ N M ⌉, ⌈·⌉ being the ceiling U (t +1) ← M.re f ormulation(V (t ) , U (t ) ) # Use a sequence of token-level attention to extract relevant information and update query. 7: # Text entailment to measure the similarity between hypothesis (the retrieved document) and premises (the query with the different choices). 8: t = t + 1 9: end while 10: i ) # update module based on loss function 11: return Predicted answer optionŷ i . 1 function. Then the output representation of this pooling layer is the extracted knowledge from the document represented as Finally, the updated representation of the question U (t +1) ∈ R M ×2d is the sum of U (t ) and G (t ) .

Entailment Module.
Given the selected top-1 document, the module needs to select a particular answer from the candidate In the approach pioneered by [20], a multiple reading comprehension problem is converted into an entailment problem wherein each top-1 selected document is a premise and the question combined with each candidate answer is used as a hypothesis, and the model's probability that the premise entails this hypothesis becomes the candidate answer's score. In our method, the embedding of selected top-1 document V (t ) is treated as the embedding vector of the premise P (t ) , while the hypothesis is the combination of question and candidate choices. Since the question representation is vectorized as U (t ) but the candidate choices is still in the token-level, the question embedding U (t ) is treated as the initial hidden states and the choice token O = {O 1 , O 2 , ...O h } are passed through a BiGRU separately in order to capture the dependency between the words, which returns a new hypothesis representation H (t ) 1,2...,h ∈ R 2d ×(M +l en(O 1, 2. . .,h )) . Then, an attention mechanism is used to determine the attention weighted representation of the j-th word in the premise as follows: The matching layer is a BiGRU(·) with the input M is the concatenation operator). Finally, the max-pooling result over the hidden states of the matching model is used for softmax classification to get the entailment score for each choice h }. The diagram of entailment module is Fig. 3 c). It is worth to notice that the input of the entailment module and question reformulation module are the selected top-1 document V (t ) and the latent question representation U (t ) , so those two models can be processed in parallel.

Multi-
Step Iterative Reasoning with knowledge extraction. Our multi-step reasoning architecture with knowledge extraction is summarized above in Algorithm 1. Given a large-scale text corpus (as Wikipedia), the question text and candidate option set, our model returns the predicted answer choice. To narrow down the search document from the whole Wikipedia to the question relevant document set, we use the token-level retrieval followed by the semantic-level retrieval (line 3). The multi-step interaction between the selection, reformulation and entailment model can be best understood by the while loop from line 4 to line 9. The initial question U 0 is first used to rank all the documents in the relevant documents (line 5), followed by which the top-1 document is sent to both the reformulation model and the entailment model. The reformulation uses the selected document to calculate the important span in the document and update the question representation U t +1 (line 6). The entailment model treats the top-1 selected document as the premise and the combination of question and candidate choices as the hypothesis to compute the entailment score for each choice (line 7). The updated question is sent back to the selection module (line 5). The selection module uses this updated question to re-rank the documents and the entire process is repeated for T steps. At the end of T steps, the model returns the choice with the score returned by T steps Score During inference, based on the multi-step score of each choice aggregation, the choice with the highest score is the predicted answer.

EXPERIMENT
We now present experiments to show the effectiveness of each component of our framework. We test the HeadQA dataset both on supervised and unsupervised settings, and further perform ablation study to evaluate the contribution from each part of the model. In the end, we present a case study to demonstrate the interpretability of our model.

Evaluation on Reasoning Ability
In this section, we evaluate the performance of reasoning ability of MurKe on the HeadQA data. Since the HeadQA data is very small, we use the other related datasets to pre-train the different modules of MurKe. Recent studies [12,15] have shown the benefit of fine-tuning on similar tasks or datasets for knowledge transfer. Considering the unique challenge of HeadQA, we explore the related retrieval, reading comprehension and entailment task-specific datasets for knowledge transfer. We directly adapt the pre-train weight without further training on the HeadQA dataset, which is defined as the unsupervised setting. In the supervised setting, we initialize MurKe with the pre-trained weight and then finetune on HeadQA.
Metrics We use Accuracy (Acc) and POINTS metric (used in the official exams): a right answer counts 3 points and a wrong one subtracts 1 point. 4 5.2.1 Training Details. All the bi-directional GRU are with a single hidden layer (d = 200). The input of BiGRU at each token is the pre-trained BioWordVec embedding (200-dimensional) 5 , which trains the word embedding using PubMed 6 and the clinical notes, and this BioWordVec covers 98% words in our dataset. Additionally, to capture the structural representation of the words, we incorporate the background knowledge in the form of graph embedding using the ConceptNet [37] knowledge base. In the end, both embeddings are concatenated to form the final word embeddings (300-dimensional).
In the unsupervised setting, the BiGRU in the retrieval model is pre-trained with the document and question encoder-decoder model. The reformulator is pre-trained using supervised learning (using the correct spans as supervision), where we use SQuAD data [32] to train this model as the pre-trained weight. The entailment model is trained using entailment task-specific dataset SciTail [20]. In the supervised setting, we first train the model by setting the number of multi-step iterative method steps (T = 1), then we train the model with different step numbers. We train the models for 50 iterations using SGD with a learning rate of 0.015 and learning rate decay of 0.05.

Baselines.
DrQA [5] consists of a Document Retriever module based on bigram hashing and TF-IDF matching to return the five most relevant Wikipedia articles to a given question and a machine comprehension module, that is implemented using a multilayer recurrent neural network trained on SQuAD [32] to find the exact span containing the correct answer. Similar to [39], the answer containing the most overlapping tokens with the selected span is considered as the correct answer for the multi-choice setting. To apply to our dataset, we calculate the similarity between selected spans and options. The option with the highest score is treated as the correct answer. BiDAF [33] the document retriever is the same as DrQA [5], but the Document Reader uses bi-directional attention flow mechanism and hierarchical embedding process to obtain a question-aware context representation that is used to predict the correct answer span. DecompAttn [20] is a textual entailment system that first forms hypothesis by appending each candidate answer option to the question. The hypothesis is then used in turn as a question to retrieve relevant sentences to be considered as the premises. The degree of a premise entailing a hypothesis is then computed as the entailment score and the answer in the hypothesis leading to the highest score is the correct answer. DGEM [29] is a neural attention-based entailment system that decomposes the task into sub-problems to be solved in a parallelizable manner, where the results are merged to produce the final classification output. TFIDF Retrieval, which is similar to the IR baselines by [8,9], uses the DrQA [5] 's Document Retriever, which scores the relation between the queries and the articles as TF-IDF weighted bag-of-word vectors, and also takes into account word order and bi-gram counting. The predicted answer is defined as the one having the maximum score in the question for which we obtained the highest document relevance. Retrieval + BioBERT∥BERT entailment model uses the top one document from the retrieval module and pre-trained BioBERT∥BERT to get the entailment score between premise (search document) and hypothesis (question with the choice), where the top one hypothesis is the answer. Multi-step TF-IDF retrieval (using keywords) uses the keywords obtained from the previous document to reformulate the question and then uses TF-IDF to retrieve the new document. Multi-step-reasoner [10] the multi-step framework using reinforcement learning (RL) where retriever and reader (DrQA) iteratively interact with each other to get the final answer. Table 2 shows the accuracy and POINTS scores for HeadQA. Our model MurKe performs best among the baselines. As can be seen, even a powerful model like BERT performs unsatisfactorily on the HeadQA dataset. The main reason is that the initial question does not contain sufficient retrievable clues to find the document containing the answer. Whereas, the multiple steps of iterative reasoning of our proposed    model help to reformulate the question with the missing information, which in turn facilitates in retrieving the document related to the answer and uniformly increases performance over the base model. Moreover, using different task-related datasets to pre-train each module separately is promising to achieve an acceptable performance.

Supervised Setting
We show the performance of the top models on the test split corresponding to the supervised setting in Table 3. Our proposed model MurKe performs substantially better than the other baselines, which shows that by using the multi-step iteration it is possible to have the model better match the gold document and get a better entailment score. The other multi-step methods, like multi-step TFIDF-IR and Multi-step Reasoner perform worse than our method. This is primarily because the multi-step TFIDF-IR methods rely on statistical features like frequency of terms in the document, and fail to explicitly use information about entities that may not be frequently occurring in the document. We also find that RL approaches, Multi-step Reasoner, are slow to converge as rewards from a down-stream task are sparse and action space in information retrieval is very large. Table 4 shows humans performance. The first row is the average of the top 10 scores gotten by humans and the second row is the passing score, meaning that the examinee can pass the exam if it receives above this score. Compared to the score of the pass mark, MurKe passes three categories (EIR, PIR, and QIR) and the avg point is higher than the pass mark. Nevertheless, there is still a long way to beat the best performance of humans.

Influence of the number of Reasoning
Step. As we can see from Fig. 4, without the multi-step (using 1 step), the performance is very poor 41.  Table 5: Different fusion methods on both unsupervised and supervised settings on MID data. Here avg stands for average and con for concatenate.
quality in terms of answer accuracy becomes better, which indicates that even though the correct document (containing the answer string) was not retrieved in the first step, the retriever is able to fetch relevant documents later. The performance keeps increasing with the increase of the iterative step. But when the number of steps is too high (6 steps), the performance declines, which may indicate that more noises are added with more steps. Therefore, in most cases the optimal value of T lies in a small range of values as demonstrated in [10] and it is not time-consuming to find it using the grid search strategy in practical applications. It is also unsurprising to see that when correct documents are retrieved, the performance of the entailment model also increases and it is easy to find the correct final answer.

Performance using multi-modality fusion.
We also test the performance of the proposed model using different multi-modality fusion methods on the questions in the Medicine category (MIR) Question: "In the layer closest to the pigment epithelium of the retina are which cells" Choices: A: Ganglionares, B: Cones and canes, C: Amacrinas, D: Bipolar, E: Horizontal] Step 1 Step 2

Document retrieved
Retinal pigment epithelium is firmly attached to the underlying choroid and overlying retinal visual cells.
Cone cells, or cones, are photoreceptor cells in the retinas of vertebrate eyes Answer: B: Cones and canes Question: "The main vessels that regulate blood flow are" Choices: A: Arterioles, B: Arteries, C: Capillaries, D: Venules, E: Veins Step 1 Step 2 Arteries take blood away from the heart. The main artery is the aorta that branches into major arteries that take blood to different limbs and organs The major arteries diverge into minor arteries, and then smaller vessels called arterioles, to reach more deeply into the muscles and organs of the body Answer: A: Arterioles Step 3 Document retrieved the smallest arteries, vessels called arterioles, further branch into tiny capillaries, where nutrients and wastes are exchanged. In the unsupervised setting, we compare without fusion method (w/o) and average the question embedding and image embedding (avg). In the supervised setting, we evaluate without fusion method (w/o), use the bi-linear model (bil) and concatenate the question embedding and image embedding and project to the pre dimension (con).
As seen from Table 5, fusing additional image information with the question representation can help improve the performance. It is not surprising that the question is related to the image, so by adding image information, it helps the model to better retrieve the document. In the supervised setting, the pre-trained weight can help the image embedding learn a projection from the image space to the text space, so it has more improvement.

Evaluation on Knowledge Extraction
In this section, we want to demonstrate the performance of the knowledge extraction which combines token-level retrieval and semantic retrieval, and indicates the necessity of using multi-step reasoning. We use the NCRF++ [45] to get the selected question key-terms and is trained on the Essential Terms dataset 8 introduced by Khashabi et al. [19]. Similarly, we use the same processing on the answer. We calculate the amount that both the question and answer appear in the same document. We find that only 21 questions (6765 in total) have one document that contains both question and answer key term, which shows the complexity of the question and the importance of using multi-step iterative method to solve this problem.

Influence of different supporting document scale.
We want to narrow the supporting document space that not only contains the information needed by question and answer, but also will 7 https://github.com/qubvel/classification_models 8 https://github.com/allenai/essential-terms not be redundant. To know the influence of supporting document scale, instead of using threshold after the semantic-level retrieval, we rank the supporting documents and select the scale from the top 10 to top 50 to assess the performance. From Fig. 6, we can see that with the growth of supporting documents of question, the performance gets better. When the number of documents is around top 30, all methods reach the best performance. However, when the number of supporting documents goes beyond that, the performance shows a little bit drop. This may be because with the increase of documents, it leads to more deceptive documents.

Interpretable Ability of MurKe
Fig . 5 shows two instances where iterative interaction is helpful. In the left figure, the retriever is initially unable to find a document that can directly answer the question. However, it finds a document which has a different description of "visual cells", "photoreceptor cells", allowing it to find a more relevant document that directly answers the question. In the figure at the right side, the retrieved documents indicate that both "Arterioles" and "Arteries" could be the answer. Based on the fact that the smallest "arteries" is "arterioles", which reach into the muscles and organs of the body, so "arterioles" is the main vessels that regulate the blood flow.
Since we aggregate (sum) the scores of entailment of each retrieved documents with choices, this leads to an increase in the score of the choice ("Arterioles") to be the predicted answer. Therefore, by using reading-answer attention in the reformulation module, our module MurKe can clearly show us which part in the document is highlighted as the clue regarding current question and provides the interpretability of the reasoning path.

CONCLUSIONS
In this paper, we present a system MurKe that answers healthcare exam questions by using knowledge extraction and multi-step reasoning. To get a relevant document for each question, MurKe retrieves supporting documents from a large, noisy corpus on the basis of keywords extracted from the original question and semantic retrieval. MurKe proposes the multi-step iterative method to solve complex healthcare QA, which uses information selected by combining iterative question reformulation and textual entailment. Our neural architecture uses a sequence of token-level attention mechanisms to extract relevant evidence from the selected documents in order to update the latent representation of the question, which shows the interpretability of the reasoning path. Through empirical results and case study, we demonstrate that our proposed system is able to outperform several strong baselines on the HeadQA dataset.