{'fraction_non_alphanumeric': 0.03329661683713611, 'fraction_numerical': 0.016963021243115658, 'mean_word_length': 4.6813874485964595, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 6, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Improving Low-Resource Question Answering using Active Learning in Multiple Stages

Neural approaches have become very popular in the domain of Question Answering, however they require a large amount of annotated data. Furthermore, they often yield very good performance but only in the domain they were trained on. In this work we propose a novel approach that combines data augmentation via question-answer generation with Active Learning to improve performance in low resource settings, where the target domains are diverse in terms of difficulty and similarity to the source domain. We also investigate Active Learning for question answering in different stages, overall reducing the annotation effort of humans. For this purpose, we consider target domains in realistic settings, with an extremely low amount of annotated samples but with many unlabeled documents, which we assume can be obtained with little effort. Additionally, we assume sufficient amount of labeled data from the source domain is available. We perform extensive experiments to find the best setup for incorporating domain experts. Our findings show that our novel approach, where humans are incorporated as early as possible in the process, boosts performance in the low-resource, domain-specific setting, allowing for low-labeling-effort question answering systems in new, specialized domains. They further demonstrate how human annotation affects the performance of QA depending on the stage it is performed.


Introduction
Machine Reading Question Answering (MRQA) is a challenging and important problem. Facilitating targeted information extraction from documents, it allows users to get fast, easy access to a vast amount of documents available. MRQA models generally need plenty of annotations, therefore several methods have been devised for augmenting data by generating new annotated samples, with the ultimate goal of improving quality of predictions. Some of these approaches show a real benefit in the downstream MRQA task; however, there is no work focusing on employing Language Models (LM) fine-tuned for generating question-answer pairs in low-resource, domain-specific settings, which are often observed in practice. In these cases, most of the times only few annotated samples are available, due to the specialized domain being usually vastly different from the publicly available labeled data of the source domain. Moreover, labeling in general is an expensive procedure, as it requires a significant amount of time from domain experts. However, we consider collecting and annotating a small set of samples (e.g. 200), accurately selected to boost performance of the MRQA model, feasible and not too expensive. In this work we introduce a novel approach that follows this idea.
Although Shakeri et al. (2020) show that a question-answer generation approach trained on SQuAD (Rajpurkar et al., 2016) transfers fairly well to other (similar) domains, this does not hold in general with datasets from specialized domains, which are very relevant in practice. Additionally, so far it stays unclear how few samples from the target domain, selected based on various criteria, affect the performance of the MRQA model when they are used in the training of the sample generation model as well. In this work, we explore a novel approach that enables this, together with investigating how employing Active Learning (AL), either at the data generation model or at the MRQA model, affects the overall performance of the MRQA task. We consider only a small set of documents from the target domain annotated in terms of questions and answers, in addition to a potentially large set of samples from a different, generic source domain. We also assume to have access to a large amount of unlabeled data (documents) available from the target domain, which is common in real scenarios arXiv:2211.14880v1 [cs.CL] 27 Nov 2022 (e.g. publications from PubMed 1 when considering the medical domain). We then employ a state-ofthe-art model for generating question-answer pairs on these documents from the target domain, to enrich our training sets with synthetic data. We study the application of AL in order to label those samples which are most relevant for increasing performance, with the aim of reducing the amount of labeled samples needed. We consider these labeled samples in both the data generation stage as well as for the MRQA model itself. Our experimental results show great improvements for the MRQA task considering two domain-specific datasets, namely TechQA (Castelli et al., 2019) and BioASQ (Tsatsaronis et al., 2015), with few samples specifically annotated for the data generation model.
In this work, our main contributions are the following: 1) We introduce a novel approach that combines data augmentation for MRQA via questionanswer generation with Active Learning; 2) we identify the most relevant samples for AL by adapting to our setting scoring functions recently used for unsupervised quality assessment of machine translation; 3) we introduce a new sample relevance score specific to MRQA by coupling the generated samples with the downstream task, so that the MRQA model influences the data augmentation process as well; and 4) we perform extensive experiments 2 to demonstrate how to utilize AL in MRQA for improving performance in lowresource, domain-specific scenarios.
2 Related Work 2.1 Low-Resource MRQA In the literature, there are several approaches for tackling the low-resource problem, including scenarios where few labeled data are available and others where no labels are available at all.
One approach is to use pre-trained Language Models Radford et al., 2019;Lewis et al., 2019) backed by transformers (Vaswani et al., 2017), which can be especially useful in cases where little or no labeled data exists and it is costly to generate more. In the best case, the LM can be used without further fine-tuning (when the target task and domain is similar to those used in the pre-training objective). Otherwise, if unlabeled data is available, it may be used for adaptation of the LM in a self-supervised fashion.
If the low-resource domain is accompanied by some annotated samples, weak supervision -where external sources of information are used -becomes relevant (Hedderich et al., 2021;. Moreover, data augmentation (Zhang et al., 2020;Van et al., 2021) and LM domain adaptation (Nishida et al., 2020;Zhang et al., 2020) have been shown to improve performance for MRQA.
A different approach is to use domain transfer, where a model is trained on a source domain and then adapted to a different target domain, e.g. by employing adversarial training .

Data Generation
Recent work has shifted from domain adaptation towards data generation (Shakeri et al., 2020;Puri et al., 2020;Luo et al., 2021;Lee et al., 2020). Given a passage of text, the generator model learns to output question-answer pairs. The generation model is trained on a large amount of data from the source domain and is applied to any document in any target domain, commonly employing pre-trained LMs as decoder.
Common generation methods focus only on generating the questions and assess their performance exclusively on the generated questions using automatic metrics (Sun et al., 2018;Liu et al., 2020;Tuan et al., 2019;Mitkov and Ha, 2003;Ma et al., 2020;Song et al., 2018;Yin et al., 2021;Sachan and Xing, 2018;Zhao et al., 2018;Du et al., 2017). Only a few approaches are based on generating full question-answer pairs, where the generated data is evaluated by means of the MRQA model. While Klein and Nabi (2019) and Luo et al. (2021) only perform in-domain experiments -where the training data and data used for generating new samples come from the same domain - Shakeri et al. (2020) and Lee et al. (2020) show the generalizability of their data generation models by also performing out of domain experiments. Furthermore, automatically generated question-answer pairs can be filtered Puri et al., 2020;Shakeri et al., 2020) to get rid of noisy samples.

Active Learning
Active Learning (Settles, 2012;Cohn et al., 1996) is a framework aimed to reduce the amount of annotated data required for training machine learning 1.

2.
AL on data augmentation AL on MRQA  (2017)), and it has been shown to be helpful for MRQA as well (Hong et al., 2019). While Kratzwald et al. (2020) learn whether samples are annotated automatically or manually, others make use of pool based sampling strategies, based on heuristics, to score samples for annotation (Lin and Parikh, 2017).

Method
Our novel approach to tackle MRQA in lowresource settings with minimal human input, is based on combining data generation with AL. In this section we describe the models involved, i.e. the data generation and the MRQA model, as well as our iterative sample selection, which adapts and combines various existing scores, with the addition of a novel one. A high-level overview of our approach is provided in figure 1.

Data Generation for MRQA
Model Hereafter, we denote the question as q, the answer as a and the corresponding context as c.
Regarding data generation, we consider the QA2S model proposed by Shakeri et al. (2020), since this model shows best overall performance in their and our experiments. The considered model makes use of a pre-trained encoder-decoder LM and it is fine-tuned to generate the question given the context, followed by generating the answer given the context and the previously decoded question in a subsequent decoding step. Hence the same model is trained to approximate both probability distributions p(a|q, c) and p(q|c). Therefore, the question is generated in an auto-regressive manner and conditioned on the context: Similarly, the answer is conditioned on the context and the question: The pairs of special tokens <a>, </a> and <q>, </q> mark the answer and the question in the sequence, respectively. During inference, a two-step decoding process is used and <q> and <a> are given as begin-of-sequence tokens to start generating the question and the answer, respectively.
Decoding Generating questions is performed by nucleus sampling (Holtzman et al., 2020) from the output distribution over the vocabulary, considering 95% of the probability mass and top 20 tokens in each step. The answer is then decoded greedily with a beam search of size 10. These hyperparameters were also used in prior work (Shakeri et al., 2020). Samples for which the answer does not occur in the context are discarded since we focus on extractive MRQA. Furthermore, only samples for which the corresponding end tokens are predicted correctly are considered as valid.
Filtering Finally, only a subset of the generated samples is kept. Therefore, they are filtered using two sample filtering approaches, namely the LM score filtering, introduced by Shakeri et al. (2020), and the round-trip consistency (RTcons) . In LM score filtering, the generated samples are sorted according to the probability p(y|y <t , x) as given by the generation model and the top n samples are kept (we use n = 5). Regarding the RTcons filtering, an MRQA model is used to assess the generated question-answer pair: The generated question along with the context is fed into an MRQA model to predict the answer. The sample is then discarded if the predicted answer does not match the generated one. Since RTcons might discard all samples generated from a context, we fine-tune the MRQA model (previously fine-tuned on the source domain) used for RTcons filtering also on the available samples from the target domain.

MRQA Model
We employ pre-trained BERT  for encoding the question concatenated with the context. On top, a span extraction head models the probability for each context token to be the start and the end of the answer span. We refer to  for more details.

Algorithm 1 Model Training with AL
Require: data D pool , D eval scoring function f score iterations r, number of samples n 1: Initialize models M Gen , M MRQA with pre-trained weights 2: D train = ∅ 3: for iteration = 1, 2, . . . , r do 4: D selected ← select n top scoring samples using f score and M Gen , M MRQA for labeling 5: D train ← D train ∪ D selected 6: D pool ← D pool \ D selected 7: Re-initialize models M Gen , M MRQA with pretrained weights 8: Fine-tune M Gen , M MRQA on D train and load best models w.r.t. D eval (using e.g. loss, F1) 9: end for

Active Learning
We consider different context scoring functions to be used in an AL scenario with pool based sampling for our specific task. We borrow Sentence Probability (SP), Dropout-based Sentence Probability (D-SP) and Dropout-based Lexical Similarity (LS) from unsupervised quality estimates (Fomicheva et al., 2020;Xiao et al., 2020) and adapt them as scoring functions in our setting. We also introduce round-trip scoring (RT), which incorporates the MRQA model in the sample selection process: We believe that, during data generation, it is important to link the sample selection process with the eventual downstream task, in order to generate high quality samples. Otherwise, the data generation model is not constrained in terms of questions being generated, potentially ending up generating questions which do not take the given context or evidence into account. Finally, we also compare scoring functions based on the data generation model with an approach that only uses the MRQA model. For this purpose, we consider Bayesian Active Learning by Disagreement (BALD) (Houlsby et al., 2011;Gal et al., 2017) as a baseline. In all cases, we initially fine-tune the models on the generic source domain in order to have a good starting point. Our novel approach, combining data generation together with AL, is briefly described in Algorithm 1.

Scoring functions
Sentence Probability (SP) This scoring function is based on the probability distribution of the data generation model. The contexts are scored according to the sentence probability of the answer being generated: We generate the question and answer by greedy decoding with beam search of size 10, but only use the produced answer for scoring a sample's context.

Dropout-based Sentence Probability (D-SP)
In contrast to SP, D-SP makes use of multiple data generation models to compute the sentence probability: We employ dropout at inference time (in addition to during fine-tuning) to realize different subsets of the model, as described by Gal and Ghahramani (2016), running N forward passes. The output is decoded similarly to SP and the same prediction is used in all forward passes.
Dropout-based Lexical Similarity (LS) Regarding LS, multiple answers are decoded using multiple models (again realized via dropout), and all of them are compared pairwise at the lexical level using the Meteor (Banerjee and Lavie, 2005) metric (with i = j): Meteor(a i , a j ). (5) Decoding is again implemented greedily using beam search of size 10.
Since the ultimate goal is to improve the performance of the MRQA model, we propose the following novel methods integrating the MRQA model in the computation of the ranking scores: Round-trip (RT) We generate question-answer pairs from the context similarly to SP, and then we apply an MRQA model in order to rank the samples according to the F1 score between the predicted answer and the answer generated by the data generation model. We therefore prefer documents for which the data generation model will generate question-answer pairs which the MRQA model cannot predict correctly.
D-SP+RT We combine the D-SP and the RT scores sample-wise to obtain the final ranking score. To this end, the D-SP score is rescaled so that its range matches the RT score (thus the best prediction corresponds to the value of 1.0) and the distribution is similar to RT and has been found using manual inspection of samples' scores:

Scoring function for MRQA model
When only the MRQA model is considered, we use the BALD score (Houlsby et al., 2011) to measure the uncertainty of the model's predictions, using entropy H given training data D and model weights θ: In order to make computation feasible, we calculate the BALD score (using dropout) independently for start and end probabilities. The sum of both scores is then used to rate and order the unlabeled samples.

Experimental Setup
We focus on the low-resource scenario, where we assume to be able to use an extremely small amount of annotated samples (200) from the target domain.
In addition to a large annotated source domain, we assume to have many unlabeled documents available from the target domain as well, which can be used for generating new samples. We consider this assumption to be valid in general, as one of the main goals of MRQA is to provide better access to the vast amount of existing documents.

Data
We consider SQuAD as source domain and Natu-ralQuestions (NQ), TechQA and BioASQ as tar-

Question & Answer Generation
Training Similarly to Shakeri et al. (2020), in our approach we use bart-large (Lewis et al., 2019) with 406M parameters for the data generation model as this showed better results in our preliminary experiments compared to gpt2-large (Radford et al., 2019). It is fine-tuned using cross entropy loss and the model with the best loss on the evaluation data is selected. Finetuning of the data generation model is performed for 5 epochs on source domain data first, followed by fine-tuning on the target domain. As we found that 5 epochs is too few for fine-tuning in the lowresource setting, on the target domain we fine-tune it for 10 epochs instead. In addition, we perform early stopping based on the dev set loss.
To fit the context concatenated with the question into the model we split the context into several chunks. We only consider those chunks where the answer does occur in the context. Additionally, since TechQA has long questions we truncate the questions to the first 200 tokens, to allow for sufficient space in the input for the context as well.
Synthetic Data Generation Regarding question generation (i.e. the first decoding step), we allow contexts to have a size of up to 724 tokens; this allows the generated questions to be composed of up to 300 tokens, as both will be fed into the model in the second decoding step and their sum cannot exceed the total of 1024 tokens (including special tokens). Chunking this input would be difficult, since aggregating the generated text is not simple (especially if outputs do not overlap). We generate 10 questions per input, followed by one answer for each question-context pair, yielding up to 10 samples per context. In total we choose 100000 documents from the target domain, ignoring contexts with less than 100 tokens. Regarding the NQ dataset, this is the reason why only 50535 documents are considered.

MRQA Model
We use bert-base-uncased  in the MRQA model for comparability with prior work, with a maximum input length of 512 tokens and a stride of 128. Similarly as for the data generation model, when considering the TechQA dataset we truncate the questions (whether from the train/dev corpus or synthetically generated). Since chunking is applied to the inputs and predictions are aggregated afterwards (by choosing the best span over all chunks' predictions), it may happen that the answer of a sample is not completely included in a chunk's context. We especially observe this for TechQA, where the performance on the evaluation data is greatly degraded as the model never has the chance to predict the answer correctly for some samples.

Active Learning
In the scenarios with AL, we score available samples using each of the scoring functions described in Section 3.3, running 10 forward passes for D-SP, LS, D-SP+RT and BALD. Available samples include only samples with labels since we simulate the domain expert. We run 4 iterations, starting initially with all available samples from the target domain in the pool, and then at each step selecting the 50 samples the model is least confident about. In each iteration we remove the selected samples from the query pool and fine-tune the models on all samples selected so far, always initializing the model with weights gained when fine-tuned on the source domain (or on the synthetic data for MRQA) in order to not introduce any biases. Since D-SP, LS and RT are computationally expensive, with LS actually decoding generated text after each forward pass, performing AL with LS, RT and D-SP+RT on NQ was not feasible for us. Therefore, we selected 10000 random samples from this domain for use with AL, denoted as NQ#10000 in our experiments.

Baselines
In order to assess the performance of our approach we compare it against several baselines. Firstly, we consider an MRQA model fine-tuned on the target domain data only. Then, we fine-tune a second MRQA model on SQuAD first, followed by further fine-tuning on samples from the target domain. In order to determine which scenario benefits from AL the most, we perform AL on the MRQA model with BALD. Prior to applying AL, one MRQA model is fine-tuned on SQuAD while a separate MRQA model is fine-tuned on synthetic data generated using the data generation model fine-tuned on SQuAD.

Experimental Results
We assess the performance of the MRQA models using exact match (EM) and F1 score of the answers at the token level.

AL in Multiple Stages
We report results of our experiments in table 2. In our scenario, AL performs better than random sampling, both when it is applied on the MRQA model and when it is applied on the data generation model as well, as the results obtained with our various scoring functions demonstrate. Without AL, we get the best results for our low-resource setting on all datasets when the data generation model fine-tuned on both SQuAD and the target domain data is employed for data augmentation. In terms of F1 metric, we get 67.13% on NQ, 57.04% on TechQA and 80.15% on BioASQ.
We also observe that AL improves performance on all domains, with an absolute increase of F1 of 0.55% (NQ), 1.15% (NQ#10000), 1.85% (TechQA) and 2.42% (BioASQ). Performance on NQ is in general not improved much when including additional data from the target domain, probably because of its similarity with the source domain (SQuAD) w.r.t. the others. Although fine-tuning the data generation model on target domain data gives already superior results w.r.t. BALD, applying AL at the MRQA model still improves performance on all datasets, if compared to simply fine-tuning the MRQA model on SQuAD and random target domain data. In general, we find that target domain data is necessary when tackling more specialized domains, as in the case of TechQA and BioASQ, as this improves F1 by up to 39.13% and 19.58%, respectively, with only 200 samples annotated.

Scoring functions for data generation
Regarding the scoring functions, in our experiments we observe that RT, D-SP and its combination D-SP+RT work best w.r.t. the others. While RT yields high F1 scores for NQ#10000 (68.28%) and TechQA (58.89%), BioASQ profits the most from D-SP (82.57%) with the latter two domains actually outperforming the case where all target domain samples are annotated.
In addition, we do not observe any degradation of performance when subsampling NQ. Therefore, subsampling large pools of (unlabeled) samples for better efficiency seems to be a valid choice when applying AL.

Insights about drawn samples
To better understand the performance of the considered AL methods, we analyze different aspects of the samples which are selected for labeling.   Overlap of drawn samples We analyze the overlap of the chosen samples between the different AL strategies. Table 4 reflects this for the BioASQ domain. We observe rather small sample overlap among the approaches, a trend that holds for the other domains as well. The overlap depends on the dataset with larger datasets having less overlap.
Distribution of scores Figure 2 shows the distribution of scores over all available samples in each  Figure 2: Sample score distribution for TechQA: RT scores many samples low, but surprisingly also rates some samples as best, although the task of predicting the generated answer for a generated question is complex. Scores have been rescaled to [0, 1].
iteration of AL for the TechQA dataset. Regarding RT scoring, we observe a steep curve where, depending on the dataset, many samples are rated as 0 and many as 1. This indicates that both models, data generation and MRQA, work very well in combination since well working samples are rated high. The RT scoring seems to be also a good measure to quantify the "difference" between the target domain and the source domain. For example, for both NQ and BioASQ, RT scores are much better than those for TechQA. This happens probably because the RT scoring method considers the downstream MRQA task when computing the score thus reflecting the larger distribution shift when more specialized domains are considered. Therefore, for the TechQA dataset, the visualization suggests that one might as well select all 200 samples in one iteration, instead of annotating 50 samples in four consecutive iterations. Another interesting behavior we observe is that the RT method scores samples best in the first iteration. This might occur due to potential overfitting of the data generation model, being fine-tuned with 50 samples from the target domain, resulting in a decrease of score in the following iteration.
Distribution of samples In order for the MRQA model to perform well, it is important that a set of diverse samples is selected by the AL strategy. Figure 3 shows a visualization, based on t-SNE, of the diversity of samples drawn from the BioASQ dataset according to RT scoring in the last iteration. More examples can be found in the appendix.

Lessons Learned
For all domains, applying AL improves the MRQA task, especially when it is applied at the data generation stage. Furthermore, our proposed method -RT scoring -provides a competitive scoring function. F1 scores are consistently increased by 1.15% (NQ#10000), 1.85% (TechQA) and 2.42% (BioASQ). We expect that this improvement could be further increased when more diverse (unlabeled) samples are available for querying. This can thus be used as an approach to minimize the costly annotation effort when targeting specialized domains. When a small amount of labeled samples from the target domain exists, generating synthetic data to augment the training set is a robust solution. However, if possible, AL should be used at the stage of data generation, thus incorporating humans early in the process, as our findings from AL on the data generation model in contrast to AL on the MRQA model demonstrate.

Conclusion
Although data scarcity is ubiquitous in practice, there is a lack of approaches that address this issue for the MRQA task in specialized domains. Therefore, appropriate methods applicable to realistic low-resource domains are of great importance since deep learning models usually work well when plenty of annotated, in-domain samples are available. In our work, we showed how to utilize Active Learning to boost performance when addressing the challenging MRQA problem in low-resource, domain-specific scenarios relevant in practice. To this end, we introduced a novel approach that combines data generation with AL to tackle the MRQA problem, resulting in a boost in performance with low labeling effort. We further analyzed perfor-mance of AL when applied to the MRQA model directly in comparison to when applied to the data generation process, showing evidenced improvements of predictive performance in the latter scenario, especially when using our newly introduced scoring function tailored to the MRQA task.

Limitations
Due to a lack of resources, we were not able to conduct experiments varying the amount of iterations in case of AL or the number of annotated samples.
Furthermore, the results shown in this paper may not be effortlessly transferable to all domains, like e.g. in different languages, although the described setup could be adapted. Also, there is a chance that a domain we did not consider behaves completely different due to its characteristics, which might not be obvious before testing it in our setup.
Lastly, we should mention that fine-tuning the data generation model followed by generating plenty of synthetic data and MRQA model training requires significant amount of computing time and resources.
SQuAD as source domain for fine-tuning the data generation model and for fine-tuning the MRQA model in addition to target domain data without data generation.
NaturalQuestions We use the NaturalQuestions dataset (Kwiatkowski et al., 2019), preprocessed for the MRQA Shared Task 2019 4 (Fisch et al., 2019). The train split contains 104071 samples, while the dev split has 12836 samples. In our work, this dataset was chosen due to its domain being similar to SQuAD. The documents from the training set are used to generate question-answer pairs, while documents from the dev set are excluded.
TechQA TechQA 5 is a dataset released by IBM (Castelli et al., 2019). Questions are asked in forum post-style and the answer is given as spans within technotes (i.e. the documents). The dataset contains 600 training samples, including unanswerable questions. In our work we only consider the answerable subset of 450 questions. The official dev data with 160 answerable samples was used for evaluation, since the test set is kept secret to ensure integrity of the benchmark. Passages from the corpus of technotes containing 800K+ documents were used to generate data.
BioASQ Regarding the BioASQ dataset (Tsatsaronis et al., 2015), we rely on the version preprocessed for the MRQA Shared Task 2019 6 (Fisch et al., 2019). It includes a dev set with 1504 samples. We create random splits for training, development and testing with 70%, 20% and 10% of samples, respectively. For the data generation process, we crawl PubMed 7 abstracts as unlabeled documents.

B Implementation Details
In our implementation we rely on PyTorch (Paszke et al., 2017) and the transformers library (Wolf et al., 2020) for the models, as well as for training. The datasets library (Lhoest et al., 2021) is used for loading and preprocessing data.

B.1 Hyperparameters
The MRQA model, as well as the data generation model, have been both trained with Adam Optimizer (Kingma and Ba, 2017), a learning rate of 3e−5 and a batch size of 24. Warm-up was set to 10% for fine-tuning the data generation model, while it was disabled for fine-tuning the MRQA model. Similarly to the models used for the downstream MRQA task, bert-base-uncased was used for RTcons filtering.
MRQA model fine-tuning turned out to be quite fluctuating in terms of the evaluation score. Therefore, we performed a hyperparameter search including learning rate, warm-up steps (using linear scheduler), L2 regularization, pre-trained weight decay for the encoder and for the output layer separately, freezing the encoder or its embedding, training of only top-n layers and re-initializing top-n layers. As a result of manual tuning, only pre-trained weight decay on the encoder with λ = 1e−7 was employed, while all layers were fine-tuned without re-initialization.

C Computational Resources
All our models were run on RTX A6000 48GB, RTX 3090 24GB or GTX Titan X 12GB graphics cards. This includes training, evaluation, inference (for RTcons filtering and RT scoring) of models and generation of synthetic samples. Tables 5 and 6 show the statistics and sample overlap for all domains.

D Further Analysis
Figures 4, 5, 6 and 7 show the distribution of scores over all queryable samples in each iteration of AL for the different datasets. Figure 8 shows the sample distribution for BioASQ using RT scoring for AL.