{'fraction_non_alphanumeric': 0.029599150517653303, 'fraction_numerical': 0.010910538890363685, 'mean_word_length': 4.686188679245283, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 2, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Multi-Instance Training for Question Answering Across Table and Linked Text

Answering natural language questions using information from tables (TableQA) is of considerable recent interest. In many applications, tables occur not in isolation, but embedded in, or linked to unstructured text. Often, a question is best answered by matching its parts to either table cell contents or unstructured text spans, and extracting answers from either source. This leads to a new space of TextTableQA problems that was introduced by the HybridQA dataset. Existing adaptations of table representation to transformer-based reading comprehension (RC) architectures fail to tackle the diverse modalities of the two representations through a single system. Training such systems is further challenged by the need for distant supervision. To reduce cognitive burden, training instances usually include just the question and answer, the latter matching multiple table rows and text passages. This leads to a noisy multi-instance training regime involving not only rows of the table, but also spans of linked text. We respond to these challenges by proposing MITQA, a new TextTableQA system that explicitly models the different but closely-related probability spaces of table row selection and text span selection. Our experiments indicate the superiority of our approach compared to recent baselines. The proposed method is currently at the top of the HybridQA leaderboard with a held out test set, achieving 21 % absolute improvement on both EM and F1 scores over previous published results.


Introduction
Question answering (QA) methods for various applications, aided by transformer models, have evolved rapidly in recent years. Documents with densely-packed information often contain embedded structured elements such as tables in addition to free form text. Furthermore, answering questions over such documents often require cross-referencing between tables and free text, possibly involving multiple hops of reasoning. Though existing QA datasets and benchmarks measure performance over homogeneous data sources, such as with text (Rajpurkar et al. 2016;Chen et al. 2017;Joshi et al. 2017;Dua et al. 2019) and more recently on tables (Pasupat and Liang 2015;Zhong, Xiong, and Socher 2017;Liang et al. 2017;Herzig et al. 2020;Yin et al. 2020), there is less emphasis on heterogeneous inputs. Question answering over hybrid context, with intermingling table and text, has remained relatively less studied. Here, the answer to a *Equal Contribution particular question could be a span of text or a cell from the table. Even a relatively simple table from Wikipedia often co-references several entities, definitions or descriptions from the available text (see Figure 1).
State-of-the-art BERT-based systems are not equipped to tackle the heterogeneity of these input data sources. Most such systems linearize the table to suitably tagged token sequences (Zhong, Xiong, and Socher 2017;Liang et al. 2017;Herzig et al. 2020). Moreover, training BERT on such datasets, either via self-supervision or fine-tuning, is also limited by a maximum input sequence length of 512 tokens. A few solutions instead build graph representations of tables (Zayats, Toutanova, and Ostendorf 2021;Wang et al. 2021), but these then employ graph neural networks, and are not closely integrated with effective transformer-based solutions for linear text. Therefore, it is pragmatic to perform a selection-retrieval step to arrive at the answer serially.
We raise and answer key design questions for such a serial approach: (1) What is the unit of retrieval? (2) How to prune question-relevant contextual information for a cell? and (3) How do we train a model with the possibility of multiple correct answer spans occurring in several different table rows and linked passages?
Given the weakly supervised nature of the problem and the available datasets, current approaches suffer from two major problems (1) Multiple instance problem: There could be multiple rows containing the answer either as a table cell value or as a span of text; and (2) Multiple answer problem: candidate answers may appear as multiple token spans or as values in table cells.
In this paper, we propose a simple, yet effective two-step approach for TextTableQA. We pose the problem of QA over these hybrid input sources as a row selection and answer generation problem, with a table row as the fundamental unit of retrieval. The first step involves selecting the most relevant row from the table, based not only on the table row contents but also the contents of passages connected to the cells in that row. In the second step, we use an answer predictor module to generate answer(s) based on a joint table row and passage representation. With these pragmatic reformulations of the problem inspired by the nature of the data, our system, MITQA, achieves a new state-of-the-art performance on the HybridQA dataset and is currently the top of Figure 1: Question answering over hybrid context of table and text. Correct answer in correct context is highlighted in blue and correct answer text appearing in irrelevant context is highlighted in red. The context used to arrive at the answer in the correct passage is underlined. The relevant row to be retrieved is in green and irrelevant rows are in red. the leaderboard 1 .

Related Work
TableQA has gained much popularity in recent years, resulting in multiple different approaches. Early TableQA systems (Pasupat and Liang 2015;Zhong, Xiong, and Socher 2017;Liang et al. 2017) focused on semantic parsing to capture the query intent in an intermediate logical form. This general approach included fully supervised systems (Zhong, Xiong, and Socher 2017) having questions and logical form pairs as training data and also weakly supervised systems (Pasupat and Liang 2015;Krishnamurthy, Dasigi, and Gardner 2017;Dasigi et al. 2019) which take only a question and its final answer as training instances.
Further advances in TableQA systems (Herzig et al. 2020;Yin et al. 2020) have extended BERT to encode the entire table including headers, rows and columns. They aim to learn a table-embedding representation that can capture correlations between question keywords and target cells of the table. TAPAS (Herzig et al. 2020) is designed to answer a question by predicting the correct cells in the table in a truly end-to-end manner. TaBERT (Yin et al. 2020) is a powerful encoder developed specifically for the TableQA task. However, it does not include linked passages in its scope for finding answers.
Very recently, TableQA systems have been studied for more practical use cases, where tables are embedded in documents that provide surrounding text. A natural language query may require information from both a table and textual context to obtain an answer. HybridQA ) pioneered a research benchmark on table+text QA.
The benchmark included open-domain tables linked with free text passages related to the entities present in the tables (e.g., Wikipedia entity definition pages). They explored questions which needed information from both tables and text to arrive at the correct answer. The authors suggested HYBRIDER, a modular approach that sequentially worked on table cells and linked passages to obtain an answer for 1 https://competitions.codalab.org/competitions/24420 -Our team name is "iidrv" a natural language. However, HYBRIDER, being the first work on table+text, achieved an F1 score of 50% which is far from the possible skyline accuracy by human experts and therefore, has many scopes of improvement. We have used HYBRIDER as the baseline in our experiments.
OTT-QA Chen et al. (2021) followed up with a revised benchmark over Hybrid-QA, where the links between tables and passages were hidden. Therefore, the OTT-QA benchmark included a table and text retrieval challenge. TAT-QA (Zhu et al. 2021) is another benchmark that involves table and text for question answering. Unlike HybridQA and OTT-QA, which were open-domain, TAT-QA is specifically focused on finance and needs numerical reasoning for question answering over tabular numbers and associated text.
MultiModalQA (Talmor et al. 2021) is another question answering benchmark which introduces images as a new modality in addition to table and text for complex question answering in the open domain. ManyModalQA (Talmor et al. 2021) also proposes a new benchmark that combines table, text and image. They specifically focus on questions which are ambiguous in terms of the source modality and therefore, propose a baseline that has a dedicated network to disambiguate the target modality first, before proceeding with the remaining steps of question answering.
In summary, with the recent surge in multimodal datasets being proposed around TableQA, improved architectures for solving Text+Table QA are strongly motivated. Our significant performance lead in the popular HybridQA benchmark proves that a rethink of a unified model of text+table modalities, and a training regimen tailored to their unified representation, can reap rich benefits.

Our Approach
We propose a row-grounded question answering system capable of multi-hop reasoning over hybrid context of table and text. We follow a two step procedure with a row retriever to identify the correct row and an answer extractor which predicts the answer span from the retrieved row. We also propose to use a passage filter to pick the passages needed to answer the question. This is necessary as row retrieval and answer extractor modules can't take big passage as input. Through table splitting, we have a single row as a smallest unit of operation. Given the row and the query, we filter the passages before concatenating them into a context. The row retriever outputs high confidence rows pertaining to the query, which are then fed to an answer extractor module to get an answer per question-row-context fed as input. Finally, a reranker outputs a final answer taking row retriever and answer extractor scores into account. Figure 2 presents the overall architecture of our proposed method MITQA. Given a question, the row retriever module retrieves the top-k relevant rows from the table using a multi-instance training paradigm. The answer extractor module trained using multi-answer paradigm extracts answers (which could be table cell or a span of text in the passage connected to a cell) from top-k rows retrieved by row retriever. The final re-ranker module re-ranks the answers based on a learned weighted combination of row retriever and answer extractor confidence scores. We describe the components module in the rest of this section.

Row Retriever (RR)
Given a question and table with cells optionally connected to passages, a row retriever's task is to identify the correct row from which the answer can be obtained. A BERT based sequence classification model (Devlin et al. 2018) trained on a binary classification task with correct rows labelled as 1s and the rest as 0s is used as the row retriever. The question, table row, and the passages linked to the cells in the row are passed to a BERT-Large encoder in a specific format to get a the latent representation. Let us denote the question as q, the column headers of table as {c 1 , . . . , c n }, the values for these column for a particular row r as {v 1 , . . . , v n } and let the passages linked to the cells in the row be {p 1 , . . . , p m }. Then the input to the BERT encoder becomes (1) where 'is' is literally the word 'is'. The [CLS] embedding output by BERT is sent to a feed-forward neural network to make the prediction. At the inference stage, all the {question, row} pairs are passed through this sequence classification and the logits/scores for class 1 is used to identify the best row. A row retrieval system that expects supervision in the form of gold retrieval unit (i.e., rows) exacts a high cognitive burden from annotator in preparing training instances. In the case of HybridQA, we only have final answer-text as supervision, not the relevant row/s or the span/s. Given a table Figure 3: Distribution of number of rows containing the answer-text in the training set of HybridQA. "2: 12.8%" in the chart means that 12.8% instances of training set has exactly 2 rows with answer-text appearing in them.
with connected passages and a question, we identify potential gold rows by exact string matching answer-text on rows (cells and linked texts). We observe that there are multiple rows containing the correct answer-text. Figure 3 shows a pie chart for the number of such potentially "positive" rows that are observed in HybridQA's training set. One can observe that more than 40% instances in the training set have the problem of multiple rows containing the correct answer text. For some instances, the answer-text appears in as many as 19 rows! Summarizing the discussion thus far, the basic distant supervision approach has two main limitations: a) the important relevant context from passages relevant to the question may be lost during training as BERT has a limitation that it will truncate all tokens after 512 tokens limit; and b) if all potential correct rows are labelled as 1 and used for training, it will reduce the performance of row retriever as most of them should be labelled 0.
In response to these limitations of the basic row retriever model, we propose the following novel enhancements. a) Passage Filter (PF) (Sec.3.4) to rank passages most relevant to the context higher than the irrelevant passages, so that even if the BERT model truncates its input, we do not lose out on passages most relevant to the questions. b) Multiple-Instance loss to address the problem of multiple potentially correct rows. We map this problem into a multiple-instance learning setup (Dietterich, Lathrop, and Lozano-Pérez 1997;Andrews, Tsochantaridis, and Hofmann 2003), with question-row pairs as instances and potential correct rows for a question forming a bag. Formally, with question q, and table with rows r i ∀i ∈ {1, . . . , T }, we have a subset of rows B ⊆ {1, . . . , T } labelled 1 (relevant) and the rest labelled 0 (irrelevant). Let f (.) denote the prediction model which inputs row r i , encoded in the form x i (1), and let be the binary classification loss computed as (y i , f (x i )), where y i is the gold label of instance x i . For a given table and a question, we define row retriever loss as (2) For our experiments we keep to be the standard crossentropy loss. Apart from this new loss function we also deployed a curriculum learning (Bengio et al. 2009) type of training procedure. In the initial epochs we only use instances whose labels we are most confident about: negative rows, and questions with only one positive row. In the later epochs we increase the fraction of instances with multiple relevant rows.

Answer Extractor (AE)
Given a question and a row, the answer extractor aims to extract the answer in the form of a table cell or a span in the passages connected to the table cells of that row. Generally, a machine reading comprehension (RC) system requires ground truth start and end index as supervision for training, but in HybridQA, neither ground truth start and end index is available (for the case where answer is available in the passage connected to cells in the row) nor is the table cell coordinates (for the case where the answer is a table cell). Furthermore, high level supervision of whether the correct answer is a cell in the table or span in the text (passages connected to cells) is also not available which makes this a challenging task.
Multi Answer Training (MAT) Extractive RC models require exact answer spans along with answer text as training data. This information, however, is absent from most datasets, including HybridQA. Former methods have avoided the issue by assuming the first span as the correct span and passing it to the model. But this is often not the case. Consider the correct passage from Figure 1. The answer occurs multiple times in the context and moreover, the first answer span is not at all related to the query context. It is as good as selecting a random span within the context. Inspired by data programming methods, we propose a multi answer training (MAT) paradigm which aims to suppress incorrect spans matching the gold answers, and aims to find correct labels for the given answer text. Essentially, we treat contexts with multiple answer-matching spans as noisy. So all instances with only a single occurrence of the correct answer are the (initially) 'denoised' labels D s . Given that there is a significant fraction of these denoised labels, we Algorithm 1: Multi Answer Training (MAT) Dm now has the denoised labels 9. Train model M on Ds ∪ Dm train a model M on these labels. We then use this model M to score spans from the noisy instances in D m . This is different from pure inference because we are in a highly constrained output space i.e. we know the answer can only be among the few choices. The best scoring span out of all should give us denoised label d si . This combined with the earlier pure labels give us a much better training set on which we can train our next version of the answer extractor.
Multi Instance Reranking (MIR) The row selector is bound to occasionally return an incorrect row as the top scored row. To recover in such situations, we take the top-k (typically, 5) rows from the row retriever along with their relevance scores. The motivation for widening the beam comes from the fact that row selector recall jumps 8-9% from top-1 to top-5. These rows are all passed through the answer extractor which outputs its own set of scores corresponding to start logit and the end logits. Through a reranker, we aim to simulate a joint selection across the row retriever and answer extractor modules to select the best overall answer. We achieve this through weighted scoring of these scores and selecting the best set of weights according to the best performance on the development set. These weights can either be optimized for using grid search or even trained for using model outputs as features and evaluation scores as labels. We do a grid search over an estimated search space and optimize over the dev set scores. From table-1, we can see that the reranking leads to quite significant improvements over both the test and development sets. Algorithm 2 outlines this approach in detail.

Table Passage Ranker and Filter (PF)
Given a question, table, and a set of passages connected to cells in the table, table passage ranker and filter aims to rank the passages based on its relevance with the question. We use Sentence-BERT (Reimers and Gurevych 2019) to get question and passage embeddings and we perform asymmetric semantic search to rank the passages. This passage ranking plays a vital role in row retrieval as well as answer extraction. BERT models has a limitation that they cannot process sequence of length more than 512 tokens and passage ranking ensures that even if we truncate the context to fit BERT, we are unlikely to lose passages relevant to the question.
Moreover, in case the context contains multiple spans, passage ranking helps to bring the correct answer span in the Algorithm 2: Multi Instance Reranking Input: Dev fold D, trained row retriever R, trained answer extractor A, selectivity k, search space of combining weights W 1. for W ∈ W: 2.
for di in D:
for dij in Di: 6.
end for 11.
end for 14.
scoreW ← evaluate model A on D 15. end for 16. Select/train W to maximise scoreW front, thus avoiding possible noisy labels. This is particularly important, because the basic model of answer extractor without multi answer training (MAT), backpropagates through the first span in the passage matching with the gold answer.

Generating training data for row retriever
We use weak supervision signals from the final answer text. We label rows containing the answer directly as a table cell value or indirectly as a span of text in passage connected to the table cell as a positive row. This results into an additional challenge of handling multiple correct instances as the answer could be present in a spurious row as well.
We use row-retrieval accuracy to evaluate the performance of row retriever. Row retrieval accuracy is defined as the ratio of the number of instances containing the answer in the retrieved row divided by the total number of instances. We use exact match and F1 score to evaluate the predictions using final answer extractor.

Generating training data for answer extractor
For training, we concatenate the row and all the passages into a single context for the RC module. Here we introduce two novel training paradigms. Firstly, to tackle the multimodality between the row and plain text, we concatenate row contents as <header> is <cell content>. This bypasses the need for very heavy pretraining on additional special tokens as column and row delimiters. Secondly, all linked concatenated passages exceeds the input length limit for most models. This may lead to missed answer if the 'true' passage gets truncated away. Therefore, rather than following the order of hyperlinks, we rank the passages in order of relevance before concatenation to minimise the chances of recall loss. For passage ranking, we compare cosine similarities between question embeddings and passage embeddings output by a msmarco-distilbert-base-tas-b (Reimers and Gurevych 2019) model.

Retriever Feedback
We employ two different strategies for answer extractor training. A vanilla approach involves training on all rows whose content contain the gold answers. This can be multiple rows for a single question as outlined in the multiple instance issue. Alternatively, we can make use of retriever feedback during its training. We use training scores of train data to choose only a single row per question. This constitutes our data for training the answer extractor.

Data
We evaluate our proposed approach on HybridQA  dataset. HybridQA is a large scale open-domain question answering dataset that require reasoning over hybrid context of table and text. This dataset contains 62,682 instances in train set, 3466 instances in dev set and 3463 instances in test set. For the test set ground truth answers are not available. The authors employ Amazon Mechanical Turk crowd-workers to generate questions based on carefully crawled wikipedia tables with cells linked to wikipedia pages. We split tables into rows with column headers attached. This enables us to pose the QA problem as row retrieval and answer extraction from the retrieved row. Multiple Rows containing the answer text pose a major challenge for question answering on HybridQA dataset. As depicted in Figure 3, ≈40% instances have more than one row in the table with exact answer text. This makes retrieving the most relevant row really difficult. Multiple Answer Spans pose additional challenges. Further analysis revealed that ≈ 34.5% instances in the training set have the multiple answer spans problem.

Baselines
HYBRIDER We compare our model's performance with the standard HYBRIDER (Chen et al. 2020) baseline. HY-BRIDER uses a two phase process of linking and reasoning to answer questions over heterogeneous context of table and text. This approach attempts to use cell as a unit for linking, hopping and answer prediction. Other baselines can be found on the HybridQA challenge leaderboard. 2 There are no linked papers to the submissions as yet. We compare our model's performance on test split with all of them.

Our Models
We describe our models and elementary components in detail below. RATQA: Row retrieval Augmented Table-text Question Answering (RATQA) is the basic model which includes a BERT-Large (Devlin et al. 2018) based row retriever trained on standard cross-entropy loss and BERT-Large based answer extractor which utilizes row retrieval scores to select the most relevant row during training. During inference, we get the top scored row from the retriever and get the final answer with the help of the answer extractor. Retriever Feedback: We use a pre-trained row retriever to score rows in the train set. This score is used to select the most relevant row while constructing the training data for answer extractor (as described in Section 3.7). In settings without retriever feedback, we create separate instances for all the rows where gold answer text appears.
MAT: Multi Answer Training (MAT) follows Algorithm 1 to deal with multiple gold answer spans. In settings where MAT is not used, we choose the first answer span in the text.

MIR:
Multi Instance Re-ranking (MIR) uses Algorithm 2. We select top-5 rows based on row relevance score given by the row retriever and extract answers for a particular question from all five rows. Finally, we rank the answers based on row relevance score and answer extractor confidence. In the setting without MIR, we only select the top scoring row and use it to get the final answer.
PF: Passage Filtering (PF) uses passages ranked based on semantic similarity between the question and the passage. Passage filtering module is described in detail in Section 3.4. In the setting without PF, we just concatenate passages connected to cells in cell order (from left to right) while constructing the context.

MITQA: Multi-Instance Training for table-text Question
Answering (MITQA) includes a BERT-Large based row retriever trained on a novel multi-instance loss function (explained in Section 3.2) and BERT-Large based answer extractor which utilizes row retrieval scores to select the most relevant row during training. We preprocess the context using passage filtering (PF) mechanism described in the previous paragraph. During inference, we get the top scored row from the retriever and get the final answer with the help of the answer extractor.  We study performance on various combination of models and components. We performed ablation experiments using different combinations of our proposed components: Retriever Feedback, MAT, MIR, PF on both RATQA and MITQA.

Results and Analysis
In this section, we look at the model performance on various settings and we aim to seek answers to the following set of research questions. RQ1: Does identifying correct row first help in accurate question answering over hybrid context of table and text? RQ2: How to use weak supervision on rows where multiple rows or their linked passages can contain the target answer? Does multi-instance based training strategy work? RQ3: How to use weak supervision on passages when a passage can have multiple spans containing the target answer? Does multi-answer based training strategy work? RQ4: Does the multi-instance re-ranker use row retrieval score and answer extractor score efficiently to improve the rank of correct answer? RQ5: Does passage filtering help in improving the row retrieval performance? RQ6: Apart from the reduced size of training data, are there any performance gains derived from retriever feedback?
Row Selector Performance In Table 2, we present row retrieval accuracy of our models on the dev split of HybridQA dataset. We observe that passage filtering improves the row  (September 18, 1888-April 13, 1938, commonly known as Grey Owl, was a British-born conservationist, fur trapper, and writer who pretended to be a First Nations person. While he achieved fame as a conservationist during his life, after his death the revelation that he was not indigenous(0.63), along with other autobiographical fabrications, negatively affected his reputation. ... Figure 4: Benefits from MAT. The model loss is shown in brackets along with the instances. It is clear that the correct instance (in blue) rightly gets the lowest loss while the ones which are irrelevant (in red) have higher losses. Relevant context to answer the question is underlined. retrieval accuracy by ≈3% (RQ5). Changing standard cross entropy loss to multi instance loss (explained in Section 3.2) further boosts the row retriever accuracy by ≈2%. (RQ2) Overall QA Performance In Table 1, we compare the performance of the proposed models on the dev and test sets of HybridQA dataset. We evaluate the performance in terms of exact match and F1 scores between predicted answer and ground truth answer. Table 1 shows some key takeaways as follows: • Our strategy to retrieve correct rows first works better than HYBRIDER, producing ≈12% F1 score improvement even without any other enhancements and without retriever feedback. This shows that identifying the correct/best rows is of utmost importance and brings huge benefits. (RQ1) • Passage filtering indeed helps prune irrelevant passages before using BERT based models for the row retriever and answer extractor. The effect of passage filtering can be seen in the higher F1 scores of RATQA + PF models as compared to RATQA alone. (RQ5) • The results in Table 1 also give evidence that multi instance training helps the RR increase overall F1 score beyond passage ranking alone. (RQ2) • The results also confirm that, in all the variations of our models including passage ranking and multi instance training, retriever feedback acts as a positive influence for answer extractor, always yielding better F1 scores than models without retrieval feedback. This translates to better answer extractor performance on much less data. With retriever feedback, the model is only trained on the best row, while without retriever feedback, thrice as much training data is available, but it is more noisy. This demonstrates the superiority of our row retriever in enhancing answer extractor performance. (RQ6) • Multi Answer Training (Algorithm 1) usually boosts performance by 0.5-1%. This demonstrates the effectiveness of training on denoised data. Figure 4 demonstrated one such instance of this denoising. (RQ3) • Our strategy for multi answer training (MAT) and multi instance re-ranker (MIR) improves the F1 score as compared to model variations not applying these strategies. In fact, these strategies can be all combined together and as seen in Table 1, model variations with MIR+MAT produce the best results. (RQ3, RQ4) •We observe that MITQA+MIR+MAT, which incorporates passage re-ranking, multi answer training and multi-instance re-ranking achieves the best performance on dev as well as test set in terms of both EM and F1. The final best model achieves ≈21% absolute improvement over HYBRIDER in both EM and F1 on the test splits. At the time of writing, our system also has a ≈7% lead in both EM and F1 over the next best submission on the public leaderboard. Passage Filtering Performance We find that average number of tokens in the context for the dev set is 585, with 49% examples exceeding BERT's maximum token count of 512 (thus needing truncation). We see that, if we follow our passage ranking and filtering strategy before truncation, the answer is retained in the truncated context in around ≈1-2% more dev set examples. Interestingly, the observed performance gain for our answer extractor is slightly larger than this. This can be attributed to the fact that with passage ranking, the correct span more often appears as the first one and gets correctly chosen during back-propagation training for answer extraction.

Conclusion and Future Work
Question answering over tables and text requires reasoning over table cells and linked passage contents. Weak supervision in table+text setup poses a unique research challenge because the target answer might be mentioned in multiple row cells and/or as multiple spans in linked passages. In this work, we propose a novel training strategy that works with multiple instance (i.e., rows) and multiple answers (i.e., spans in passages) based weak supervision. We design a novel QA pipeline that uses multiple instance and multiple answer based training to identify correct rows first and then use the row cells for relevant passage lookup. We also propose efficient strategies for filtering linked passages to retain the most relevant ones for the question, and a novel re-ranker to rank the answers obtained from different rows and their respective linked passages. We have empirically validated the end-to-end QA model on the recent HybridQA benchmark, with 21% improvement in F1 score. We have also tried different combinations of our proposed strategies to substantiate the benefit from each of them separately. In future, we would like to explore two directions: (1) handling the retrieval challenge where table cell-to-passage links are not given, and (2) address more complex table structures with structural hierarchies and/or numeric reasoning.

Multi-Instance Training for Question Answering
Across Table and Linked Text (Appendix)

A Benefits of Passage Filtering
In Figure 5, we show an example that demonstrates the effectiveness of passage filtering and ranking. We can see the passage having maximum overlap with question have been ranked highest.  Table Cell We have already shown examples of gains when answers are extracted from linked text passages. We present in Figure  8 an example where MITQA is able to predict the answer correctly even when the correct answer is in a table cell and not a span in the passages. Figure 9 depicts an example instance where MITQA is able to predict the correct answer to the question, whereas HYBRIDER fails to predict the answer correctly. MITQA predicts 'Veor', which is the correct answer whereas, HY-BRIDER predicts 'Lanner' which is incorrect. One possible reason could be that HYBRIDER got confused by the presence of the word 'Cornwall' in the first row.

C.2 HYBRIDER vs MITQA
C.3 Multi Instance Re-ranking (MIR) Figure 11 depicts an instance where MIR is able to rectify the error made by MITQA. The incorrect answer also ap-  Relevant context to answer the question is underlined. The first 'Marianna' in entirely irrelevant as it refers to a different context. The second one is the best answer which is made very clear by its sentence itself.
peared in a context very similar to the context of correct answer but Multi-Instance re-ranker is able to rank the correct answer higher than the incorrect answer. Also, in Figure 10, MIR ranks the correct answer higher even though both the Figure 8: MITQA is able to extract answer even if the answer is only present in the table as a cell value. The correct answer is highlighted in blue. Despite having other numbers in the table and phrases mentioning ranks like '7th largest bank', 'sixthlargest oil and gas company', 'sixth-largest energy company' etc. in the passage MITQA was able to predict the correct answer from the table. Figure 9: The benefit of MITQA over HYBRIDER. The correct answer is highlighted in blue and the incorrect one is in red. MITQA was able to predict the correct answer whereas, HYBRIDER failed to predict. Figure 10: The benefits of MITQA and MITQA+MIR. The correct answer is highlighted in blue and the incorrect ones are highlighted in red. 'British' is the answer predicted by MITQA (and MITQA+MIR). 'German' is the second ranked answer by MITQA+MIR. Notice that the row containing 'German' also has value '4' for the column 'No'. HYBRIDER predicts 'Brazilian' as the answer, which is incorrect.
answers seems plausible answers. Figure 11: The benefit of Multi-Instance Reranking. The correct answer is highlighted in blue and the incorrect answer is highlighted in red. Both 'Piero Alva' and 'Renzo Revoredo' played for 11 years in the same stadium ('Universitario de Deportes'). But only 'Piero Alva' (answer after reranking) has retired while 'Renzo Revoredo' (answer before reranking) has not. Thus, the MIR is helping in ranking the correct answers higher than the incorrect ones in the similar context and confusing scenarios.

D Implementation details
We have implemented all our models using Pytorch 3 framework based on Huggingface's transformers 4 library. We train our models using two NVIDIA A100 GPUs. We train the row retriever and answer extractor for 5 epochs and select the best model based on model's performance on the dev set. We optimize the model parameters using AdamW algorithm with a learning rate of 5 × 10 −5 . We set the train batch size to 24 while training row retriever on two A100 gpus. We set per-GPU train batch size to 16 while training the answer extractor.

E More Statistics About HybridQA Data
Out of total 62682 instances in the train set, 61684 (98.4%) instances have answer in either table or text. 998 questions does not have answer directly available in table or text. Out of 991801 question-answer pairs in the train set, 180333 were labelled 1 (relevant) as answer-text was present in the row. In principle, only one row should be the correct row per question (that is 62K+ positives), which implies that training the row-retriever without multi-instance consideration uses over 117K wrongly labelled instances.

G Approximating an Oracle
We try to gauge a skyline performance for MITQA. We perform best answer selection (from top-5) using the gold answer instead of re-ranking by using the string match between 3 https://pytorch.org/ 4 https://huggingface.co/ predicted answer and gold answer. The results are shown in Table 4. We can see that there is still a considerable gap which needs to be covered, especially when the answer is in the table.

H Evaluation Metrics
EM: This is the exact match score. This metric measures the percentage of predictions that match any one of the ground truth answers exactly as a string. F1: The F1 score metric is a more relaxed metric that measures the overlap (in terms of word/token overlap) between the prediction and ground truth answer.