{'fraction_non_alphanumeric': 0.030140110785272074, 'fraction_numerical': 0.010182469859889215, 'mean_word_length': 4.675913083680074, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Reading Comprehension as Natural Language Inference: A Semantic Analysis

In the recent past, Natural language Inference (NLI) has gained significant attention, particularly given its promise for downstream NLP tasks. However, its true impact is limited and has not been well studied. Therefore, in this paper, we explore the utility of NLI for one of the most prominent downstream tasks, viz. Question Answering (QA). We transform the one of the largest available MRC dataset (RACE) to an NLI form, and compare the performances of a state-of-the-art model (RoBERTa) on both these forms. We propose new characterizations of questions, and evaluate the performance of QA and NLI models on these categories. We highlight clear categories for which the model is able to perform better when the data is presented in a coherent entailment form, and a structured question-answer concatenation form, respectively.


Introduction
Given two sentences, a premise and a hypothesis, the task of Natural Language Inference (NLI) is to determine whether the premise entails the hypothesis or not. † The concept of semantic entailment is central to natural language understanding (Van Benthem et al., 2008;MacCartney and Manning, 2009) and therefore, NLI models have been used to help with various downstream tasks like reading comprehension (Trivedi et al., 2019), summarization (Falke et al., 2019;Kryściński et al., 2019), and dialog systems (Welleck et al., 2019). However, the performance of an NLI system on these downstream tasks has not been studied with respect to semantic or reasoning categories.
In this work, we use NLI to perform the task of multiple choice reading comprehension (MRC, or RC). We analyse the performance of an NLI model on this task through the lens of semantics by identifying the reasoning categories (type of questions) where it is beneficial to use an NLI model.
Drawing inspiration from the prior work in the area (Clark et al., 2018;Demszky et al., 2018;Trivedi et al., 2019), we use rule-based conversion to create an NLI version of the largest available RC dataset -RACE (Lai et al., 2017). We train a RoBERTa based RC model on the original dataset, and a similar RoBERTa based NLI model on the NLI version of the dataset. We evaluate and analyse the performance of both these models by characterizing the question types that are better suited for an NLI model and a QA model.

Related Work
Reading comprehension (RC) is one of many potential downstream tasks that can benefit from NLI (MacCartney and Manning, 2009). It is easy to see that RC naturally reduces to a two-class NLI problem; specifically, it can be cast as the task of identifying if a given piece of text entails the statement formed by converting a question and a potential answer to an assertive statement (hypothesis).
Given the intuitive conversion between RC and NLI, Demszky et al. (2018) designed both a rulebased conversion system as well as a trained neural model to convert question-answering datasets such as SQuAD (Rajpurkar et al., 2016) and RACE (Lai et al., 2017) to an NLI form. However, it is unclear what these converted NLI datasets offer compared to the original question-answering datasets w.r.t semantics. We show that converting a RC task to an NLI task helps in answering certain types of questions. This establishes the usefulness of the converted datasets. Jin et al. (2019) show that despite the different form of NLI and QA tasks, performing coarse pretraining of models on NLI datasets like SNLI (Bowman et al., 2015) and MultiNLI (Williams et al., 2018) not only improves the performance of these models on downstream reading comprehension tasks, but also helps with faster convergence. We show that -for certain types of questions in reading comprehension datasets -simply transforming the task to NLI can show improvement in performance, even without pretraining on any NLI dataset. Trivedi et al. (2019) introduced a learnt weightand-combine architecture to effectively re-purpose pretrained entailment models (trained on SNLI and MultiNLI) to solve the task of multi-hop reading comprehension. They show that for certain datasets, this strategy can produce good results. However, their study focuses mainly on improving model performance using a pre-trained NLI model, and lacks an analysis of the reasoning differences arising due to the different form of the NLI and QA tasks. We focus our analysis on this aspect.

NLI for Reading Comprehension
This section describes our experimental setup for comparing a QA based approach and an NLI based approach for the task of reading comprehension. We first obtain a parallel NLI and QA dataset by converting existing RC dataset into an NLI dataset. We then train two models, one on each form of the data, and analyse their performance.

Converting RC to NLI
We use the RACE dataset (Lai et al., 2017) for our experiments. It is a large-scale reading comprehension dataset comprising of questions collected from the English exams for junior Chinese students. Each question contains four answer options, out of which only one is correct. However, about 44% of the RACE dataset consists of cloze style (fillin-the-blank) questions which are already in NLI form. Hence, in order to have a fair comparison, we only use the subset of RACE dataset which does not contain cloze style questions. This subset consists of 48890 train, 2496 validation and 2571 test examples.
We convert a RC example into an NLI example by reusing the passage as premise and paraphrasing the question along with each answer option as individual hypotheses as shown in Figure 1. Specifically, we generate the dependency parse of both the question and the answer option by using Stanford NLP package (Qi et al., 2018), then we follow the conversion rules proposed in Demszky et al. (2018) to generate a hypothesis sentence * . We make a few additions to these rules to handle a some peculiar question categories in the RACE dataset. The most prominent of the added rules is the one for questions containing "which of the following are (not) true". Such questions are very frequent (about 6% of all questions) and are not handled correctly by the rules in Demszky et al. (2018). * Appendix C presents example conversions generated using these rules.

Model
In order to perform apple-to-apple comparison we use the same model architecture for both QA and NLI. Specifically, we use the state-of-the-art reading comprehension model -consisting of a RoBERTa model (pretrained on the masked language modeling objective) as the encoder and a two layer feed-forward network on its [CLS] token as the classification head -as described in Liu et al. (2019). The input sentence is the combination of the passage and its hypothesis. The hypothesis are created using the rule-based conversion method mentioned in Section 3.1 (NLI setup) or by concatenation of question and answer option (QA setup). Table 1 shows the accuracy achieved by the RoBERTa model on the RACE dataset and its subset when presented in different forms. As we can see, the NLI model performs much better than the QA model on RACE subset. We think that the reason for this is the more natural form of the hypothesis statement used by the NLI model compared to the Q+A concatenation form used by the QA model. Moreover, while the RACE-subset consists of only those questions which have question-words such as {who, what, when...}, about 95% of the rest of the dataset, i.e. RACE \ RACE-subset, consists of only fill-in-the-blank (FITB) type questions. These FITB questions are largely with the blank at the end of the question and a naive questionanswer concatenation is very similar to a NLI form hypothesis. We believe that helps the QA model to perform better on the full RACE dataset compared to the RACE-subset, where NLI model is able to outperform the QA model showing the clear benefits of coherent conversion on complex question formulations (such as W word questions).

Analysis
In order to analyse the performance difference from a semantic perspective, we characterize question into 7 semantic categories by identifying the kind of reasoning required to answer the question.

Categories based on manual analysis
In order to perform manual analysis, we construct a delta subset consisting the 328 dev set examples on which the predictions of the QA and NLI models differ. We further divide the delta subset into gain and loss subsets. The gain subset consists of questions which the NLI model gets right, but the QA model gets wrong, and the loss subset is its complement in the delta.  Figures 2 and 3 show the distribution of labels over the Gain and Loss regions respectively. The distribution reflects that the QA models clearly outperforms the NLI model in negation questions whereas the NLI model outperforms the QA model in dialogue and deductive reasoning categories. Matching words between the question and a sentence in the passage

Passage : Food cooks quickly in parabolic cookers
Question: If you want to cook food quickly, which kind of sun-cooker is your best choice?
Main Idea Require topicality judgements What's the best title for this passage?

Negation
Picking the incorrect statement Which of the following statements is NOT true?

Dialogue
Can be inferred from a dialogue or direct speech in the passage By saying "her pen dared travel where her eyes would not", the writer means

Math
Mathematically combining facts How many functions of snow are discussed in the text?
Deductive None of the above but can be answered precisely from the text Which of the following statements is TRUE?
Inductive None of the above and cannot be answered precisely from the text How old is most likely the writer's father?

Type Heuristics
Main Idea Questions containing the words 'mainly', 'title', 'purpose' or 'topic' Negation Questions containing the 'not', 'except' or 'which of the following is wrong' Dialogue Passages containing more than 10 quotation marks (") Math Questions containing the words 'how many', 'how old' or 'how much' Deductive Questions containing the word 'true'

Categories based on heuristics
We also define another set of non-exclusive categories using heuristics, as described in Table 3. As shown in Table 4, the NLI model outperforms the QA model significantly in the dialogue, math and deductive reasoning categories. This overall trend further emphasizes the benefits of proper hypothesis generation as opposed to question and answer concatenation for the reading comprehension task.

Conclusion
There is limited work providing a comprehensive analysis of how NLI can be used for QA. In our work, we show that NLI can be used for the task of reading comprehension simply by converting the data into NLI form. We convert a large RC dataset into NLI form and perform a comparative study of the performance of the RoBERTa model trained on QA and NLI settings. We propose a categorization of questions that allows for effective comparison of models trained on NLI and QA forms of data. Our analysis clearly shows that using the NLI-based approach is at par with a QA-based approach for most reasoning categories, and it is even better for some. Specifically, we find that questions involving deductive reasoning, dialogue interpretation and math are better handled by a model trained on the NLI form of data than the QA form. However, questions involving negation favor the QA form. Our work allows for careful selection of modeling strategy based on the type of data at hand.   Table 5 lists the hyperparameter settings for both versions of the dataset.  Suburban commuters travel to about one third of the suburban commuters travel by bike and from the city in Copenhagen at present. Q: What's the best title of the passage? A: Blame! Blame! Blame! The best title of the passage's blame.

C Conversion examples
Q: What influence did the experiment have on Alexander ? A: He realized that slowing down his life speed could bring him more content.
The experiment had he realized that slowing down his life speed could bring him more content on Alexander.
Q: Which of the following is TRUE about the report findings? A: The reading scores among older children have improved.
The reading scores among older children have improved is TRUE.