{'fraction_non_alphanumeric': 0.03302980410604482, 'fraction_numerical': 0.01629649541346057, 'mean_word_length': 4.779029126213592, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Questions Are All You Need to Train a Dense Passage Retriever

We introduce ART, a new corpus-level autoencoding approach for training dense retrieval models that does not require any labeled training data. Dense retrieval is a central challenge for open-domain tasks, such as Open QA, where state-of-the-art methods typically require large supervised datasets with custom hard-negative mining and denoising of positive examples. ART, in contrast, only requires access to unpaired inputs and outputs (e.g. questions and potential answer documents). It uses a new document-retrieval autoencoding scheme, where (1) an input question is used to retrieve a set of evidence documents, and (2) the documents are then used to compute the probability of reconstructing the original question. Training for retrieval based on question reconstruction enables effective unsupervised learning of both document and question encoders, which can be later incorporated into complete Open QA systems without any further finetuning. Extensive experiments demonstrate that ART obtains state-of-the-art results on multiple QA retrieval benchmarks with only generic initialization from a pre-trained language model, removing the need for labeled data and task-specific losses.


Introduction
Dense document retrieval methods (Karpukhin et al., 2020;Xiong et al., 2021), initialized with encoders such as BERT  and trained using supervised contrastive losses (Oord et al., 2018), have surpassed the performance achieved by previously popular keyword-based approaches like BM25 (Robertson and Zaragoza, 2009). Such retrievers are core components in models for open-domain tasks, such as Open QA, where state-of-the-art methods typically require large supervised datasets with custom hard-negative mining and denoising of positive examples. In this paper, we introduce the first unsupervised method, based on a new corpus-level autoencoding approach, that can match or surpass strong supervised performance levels with no labeled training data or task-specific losses.
We propose ART: Autoencoding-based Retriever Training which only assumes access to sets of unpaired questions and documents. Given an input question, ART first retrieves a small set of possible evidences documents. It then reconstructs the original question by attending to these documents (see Figure 1 for an overview). The key idea in ART is to consider the retrieved documents as a noisy representation of the original question and question reconstruction probability as a way of denoising that provides soft-labels for how likely each document is to have been the correct result.
To bootstrap the training of a strong model, it is important to both have a strong initial retrieval model and to be able to compute reliable initial estimates of question reconstruction probability when conditioned on a (retrieved) document. Although document representations from BERT-style models are known to be reasonable retrieval baselines, it is less clear how to do zeroshot question generation. We use a generative pretrained language model (PLM) and prompt it with the document as input to generate the question tokens using teacher-forcing. As finetuning of the question-generation PLM is not needed, only the retrieval model, ART can use large PLMs and obtain accurate soft-label estimates of which documents are likely to be the highest quality.
The retriever is trained to penalize the divergence of a document likelihood from its soft-label score.
For example, if the question is "Where is the bowling hall of fame located?" as shown in Figure 1, then the training process will boost the retrieval like-lihood of the document "Bowling Hall of Fame is located in Arlington," as it is relevant and would lead to a higher question reconstruction likelihood, while the likelihood of the document "Hall of Fame is a song by ..." would be penalized as it is irrelevant. In this manner, the training process encourages correct retrieval results and vice-versa, leading to an iterative improvement in document retrieval.
Comprehensive experiments on five benchmark QA datasets demonstrate the usefulness of our proposed training approach. By simply using questions from the training set, ART outperforms models like DPR by an average of 5 points absolute in top-20 and 4 points absolute in top-100 accuracy. We also train using all the questions contained in the NaturalQuestions (NQ) dataset (Kwiatkowski et al., 2019) and find that even with a mix of answerable and unanswerable questions, ART achieves strong zero-shot generalization on out-of-distribution datasets. Our analysis further reveals that ART is highly sample efficient, outperforming BM25 and DPR with just 100 and 1000 questions, respectively, on the NQ-Open dataset, and that scaling up to larger retriever models consistently improves performance.

Problem Definition
We focus on open-domain retrieval, where given a question q, the task is to select a small set of matching documents (i.e. 20 or 100) from a large collection of evidence documents D = {d 1 , . . . , d m }. Our goal is to train a retriever in a zero-shot manner, i.e., without using questionpassage pairs, such that it retrieves relevant documents to answer the question. Our proposed approach consists of two core modeling components ( §2.2, §2.3) and a novel training method ( §2.4).

Dual Encoder Retriever
For the retriever, we use the dual-encoder model (Bromley et al., 1994) which consists of two encoders, where • one encoder computes the question embedding f q (q; Φ q ) : X → R d , and • the other encoder computes the document Here, X = V n denotes the universal set of text sequences, V denotes the vocabulary consisting of discrete tokens, and R d denotes the (latent) embedding space. We assume that both the question and document embeddings lie in the same latent space. The retrieval score for a questiondocument pair (q, d) is then defined as the inner product between their respective embeddings, denotes the retriever parameters. We select the top-K documents with maximum inner product scores and denote them as We use the transformer network (Vaswani et al., 2017) with BERT tokenization  to model both the encoders. To obtain the question or document embedding, we do a forward pass through the transformer and select the last layer hidden state corresponding to the [CLS] token. As the input document representation, we use both the document title and text separated by [SEP] token.

Zero-Shot Cross-Attention Scorer
We obtain an estimate of the relevance score for a question-(retrieved) document pair (q, z) by using a pre-trained language model (PLM). In order to do this in a zero-shot manner, we use a large generative PLM to compute the likelihood score of a document conditioned on the question p(z | q).
The quantity p(z | q) can be better approximated by the autoregressive generation of question tokens conditioned on the document and teacherforcing (Sachan et al., 2022). More formally, this can be written as where Θ denotes the parameters of the PLM, c is a constant independent of the document z, and |q| denotes the number of question tokens. Here, Eq. 2a follows from a simple application of Bayes'  rule to p(z | q) and assuming that the document prior p(z) in Eq. 2b is uniform for all z ∈ Z.
We hypothesize that calculating the relevance score using Eq. 2b would be accurate because it requires performing deep cross-attention involving all the question and document tokens. In a large PLM, the cross-attention step is highly expressive, and in combination with teacher-forcing, requires the model to explain every token in the question resulting in a better estimation.
As the input document representation, we concatenate the document title and its text. In order to prompt the PLM for question generation, we follow (Sachan et al., 2022) and append a simple natural language instruction "Please write a question based on this passage." to the document text.

Training Algorithm
For training the model, our only assumption is that a collection of questions (T ) and evidence documents (D) are provided as input. During training, the weights of the retriever are updated while the PLM is not finetuned, i.e., it is used in inference mode. Our training algorithm consists of five core steps. The first four steps are performed at every training iteration while the last step is performed every few hundred iterations. Figure 1 presents an illustration of our approach.
Step 1: Top-K Document Retrieval For fast retrieval, we pre-compute the evidence document embedding using the initial retriever parameters (Φ d ). Given a question q, we compute its embedding using the current question encoder parameters (Φ q ) and then retrieve the top-K documents (Z) according to Eq. 1. We then embed these top-K documents using the current document en-coder parameters (Φ d ) and compute fresh retriever scores as, Step 2: Retriever Likelihood Calculation Computing the exact likelihood of the document conditioned on the question requires normalizing over all the evidence documents where τ is a temperature hyperparameter. Computing this term is intractable, as this would require re-embedding all the evidence documents using Φ d . Hence, we define a new distribution to approximate the likelihood of z i as which we also refer to as the student distribution.
We assume that documents beyond the top-K contribute a very small probability mass, so we only sum over all the retrieved documents Z in the denominator. While this approximation leads to a biased estimate of retrieved document likelihood, it works well in practice. Computing Eq. 3 is tractable as it requires embedding and backpropagating through a much smaller set of documents.
Step 3: PLM Relevance Score Estimation We compute the relevance score log p(z i | q) of all the documents in Z using a large PLM (Θ). This requires scoring the question tokens using teacherforcing conditioned on a document as described in §2.3. We then define a teacher distribution by applying softmax to the relevance scoreŝ .
Step 4: Loss Calculation and Optimization We train the retriever (Φ) by minimizing the KL divergence loss between the teacher distribution (obtained by PLM) and the student distribution (computed by retriever).
Intuitively, optimizing the KL divergence pushes the document likelihood scores of the retriever to match the document relevance scores from PLM by considering the relevance scores as soft-labels.
Step 5: Updating Evidence Embeddings During training, we update the parameters of both the question encoder (Φ q ) and document encoder (Φ d ). Due to this, the pre-computed evidence embeddings that was computed using initial retriever parameters (Φ d ) becomes stale, which may affect top-K document retrieval. To prevent staleness, we re-compute the evidence document embeddings using current document encoder parameters (Φ d ) after every 500 training steps.

ART as an Autoencoder
Since our encoder takes as input question q and the PLM scores (or reconstructs) the same question when computing the relevance score, we can consider our training algorithm as an autoencoder with a retrieved document as the latent variable.
In the generative process, we start with an observed variable D (the collection of evidence documents), which is the support set for our latent variable. Given an input q, we generate an index i and retrieve the document z i . This index generation and retrieval process is modeled by our dual encoder architecture. Given z i , we decode it back into the question using our PLM.
Recall that our decoder (the PLM) is frozen and its parameters are not updated. However, the signal from the decoder output is used to train parameters of the dual encoder such that the loglikelihood of reconstructing the question q is maximized. In practice, this improves the dual encoder to select the best document for a given question, since the only way to maximize the objective is by choosing the most relevant z i given the input q.

Experimental Setup
In this section, we describe the datasets, evaluation protocol, implementation details, and baseline methods for our passage retrieval experiments. 3

Datasets and Evaluation
Evidence Passages The evidence corpus includes the preprocessed English Wikipedia dump from December 2018 (Karpukhin et al., 2020). Following convention, we split an article into nonoverlapping segments containing 100 words each resulting in over 21 million passages. The same evidence is used for both training and evaluation.  Evaluation To evaluate retriever performance, we report the conventional top-K accuracy metric. It is the fraction of questions for which at least one passage among the top-K retrieved passages contains a span of words that matches humanannotated answer(s) to the question.

Implementation Details
Model Sizes We use BERT base configuration  for the retriever, which consists of 12 layers, 12 attention heads, and 768 embedding dimensions, leading to around 220M trainable parameters. For the teacher PLM, we use T5-XL configuration (Raffel et al., 2020) consisting of 24 layers, 32 attention heads, and 2048 embedding dimensions, leading to 3B parameters.

Model Initialization
We initialize the retriever with unsupervised masked salient spans (MSS) pre-training (Sachan et al., 2021a) as it provides an improved zero-shot retrieval over BERT pretraining. We initialize the teacher PLM with the instruction-tuned T0-3B language model (Sanh et al., 2022), which has been shown to be an effective zero-shot re-ranker for information retrieval tasks (Sachan et al., 2022). 4 Compute Hardware We perform training on instances containing 8 or 16 A100 GPUs, each containing 40 GB RAM.
Passage Retrieval To perform fast top-K passage retrieval at every training step, we precompute the embeddings of all the evidence passages. Computing embeddings of 21M passages takes roughly 10 minutes on 16 GPUs. The total size of these embeddings is around 30 GB (768dimensional vectors in FP16 format). For scalable retrieval, we shard these embeddings across all the GPUs and perform exact maximum inner product search using distributed matrix multiplication.
Training Details For all the datasets except WebQ, we perform training for 10 epochs using Adam with a batch size of 64, 32 retrieved passages, dropout value of 0.1, peak learning rate of 2 × 10 −5 with warmup and linear scheduling. Due to the smaller size of WebQ, we train for 20 epochs with a batch size of 16. We save the retriever checkpoint every 500 steps and perform model selection by evaluating it on the development set. We use mixed precision training to train the retriever and performed inference over T0-3B using bfloat16 format.

Baselines
We compare ART to both unsupervised and supervised models. Unsupervised models train a single retriever using unlabeled text corpus from the Internet while supervised models train a separate retriever for each dataset.
Unsupervised Models These include the popular BM25 algorithm (Robertson and Zaragoza, 2009) that is based on the sparse bag-of-words representation of text. Dense models typically use Wikipedia paragraphs to create (pseudo-) query and context pairs to perform contrastive training of the retriever. These differ in how the negative examples are obtained during contrastive training: they can be from the same batch (ICT; Sachan et al., 2021a), or contexts passages from previous batches (Contriever ;, or by using other passages in the same article (Spider; Ram et al., 2021). Context passages can also be sampled from articles connected via hyperlinks (HLT; Zhou et al., 2022).
Supervised Models These consists of approaches that use questions and positive passages to perform contrastive training of the retriever. To obtain improved performance an additional set of hard-negative passages is often used (DPR; Karpukhin et al., 2020), iterative mining of negative contexts is done using model weights (ANCE; Xiong et al., 2021), or the retriever is first initialized with ICT or MSS pre-training followed by DPR-style finetuning (ICT-DPR / MSS-DPR; Sachan et al., 2021a). Supervision can also be infused from the annotated answers to a question by jointly training the PLM and the retriever (EMDR 2 ; Sachan et al., 2021b). TriviaQA  NQ-Open  WebQ  Average  Top-20 Top-100 Top-20 Top-100 Top-20 Top-100 Top-20 Top-100 Top-20 Top-

Zero-Shot Passage Retrieval
For the passage retrieval task, we report results on SQuAD-Open, TriviaQA, NQ-Open, and WebQ and train ART under two settings. In the first setting, we train a separate retriever for each dataset using questions from their training set. In the second setting, to examine the robustness of ART training to different question types, we train a single retriever by combining the questions from all the four datasets, which we refer to as ART-Multi. As our training process does not require annotated passages for a question, we refer to this as zeroshot passage retrieval. Table 2 presents the top-20 and top-100 retrieval accuracy in these settings alongside recent baselines that train a similar number of parameters. ART achieves substantially better performance than previous unsupervised approaches. It outperforms the recent Spider and Contriever models by an average of 9 points on top-20 and 6 points on top-100 accuracy. When comparing to supervised models, despite using just questions, ART outperforms strong baselines like DPR and ANCE and is at par or slightly better than pre-trained retrievers like MSS-DPR. In addition, ART-Multi obtains comparable performance to its single dataset ver-sion, a considerable advantage in practical applications as a single retriever can be deployed rather than training a custom retriever for each use case.
Our results suggest that an accurate and robust passage retrieval can be achieved by training with questions alone. This presents a considerably more favorable setting than the current approaches which require obtaining positive and hard-negative passages for such questions.

Sample Efficiency
To measure the sample efficiency of ART, we train the model by randomly selecting a varying number of questions from NQ-Open training questions and compute the top-K accuracy on its development set. These results are presented in Figure 2 and we also include the results of BM25 and DPR for comparison. We see that performance increases with the increase in questions until about 10k questions, after which the gains become less pronounced.
When trained with just 100 questions, ART significantly outperforms BM25 and when trained with 1k questions, it matches DPR performance levels for top-{50, . . . , 100} accuracy. This demonstrates that ART in addition to using just questions is also much more data efficient than DPR, as it requires almost ten times fewer ques- TriviaQA  WebQ  EQ  Top-20 Top-100 Top-20 Top-100 Top-20 Top-100 Top-20 Top-100 Training on answerable questions  Table 3: Top-20 and top-100 retrieval accuracy when evaluating zero-shot out-of-distribution (OOD) generalization of models on the test set of datasets. † denotes that these results are from (Ram et al., 2021). ART generalizes better than supervised models on OOD evaluation even when trained on all the questions of the Natural Questions dataset which contains a mix of answerable and unanswerable questions. When trained with 100 questions, ART outperforms BM25 and when trained with 1k questions, it matches DPR's performance for top-K > 50 passages, illustrating that ART is highly sample efficient.

SQuAD-Open
tions to reach a similar performance.

Zero-Shot Out-of-Distribution Transfer
In the previous experiments, both the training and test sets contained questions that were sampled from the same underlying distribution, a setting that we refer to as in-distribution training. However, obtaining in-domain questions for training is not always feasible in practice. Instead, a model trained on an existing collection of questions must be evaluated on new datasets, a setting that we refer to as out-of-distribution (OOD) transfer. We train ART using NQ-Open and NQ-Full questions and then evaluate its performance on SQuAD-Open, TriviaQA, WebQ, and EQ datasets. While it is desirable to train on answerable questions such as the ones included in NQ-Open but this is not always possible, as real user questions are often imprecisely worded or ambiguous. Due to this, training on NQ-Full can be considered as a practical testbed for evaluating true OOD generalization as a majority of the questions (51%) were marked as unanswerable from Wikipedia by human annotators. 5 Table 3 presents OOD generalization results on the four QA datasets including the results of DPR and Spider models trained on NQ-Open. 6 ART trained on NQ-Open always performs significantly better than both DPR and Spider, showing that it is better at generalization than supervised models. When trained using NQ-Full, ART performance further improves by 3 and 0.5-1 points on EQ and other datasets, respectively, over NQ-Open. This highlights that in addition to questions annotated as having short answers, questions annotated with long answers also provide meaningful supervisory signals and unanswerable questions do not necessarily degrade performance.
We also train ART using MS MARCO questions and perform OOD evaluation. Due to the larger size of MS MARCO and a smaller num-5 The reasons for question unanswerability can be partly attributed to imprecise Wikipedia article retrieval during the annotation process, ambiguity in information-seeking questions, information required to answer not being localized to a single paragraph, etc (Kwiatkowski et al., 2019). 6 We also include BM25 results for reference but do not directly compare with them because there is a high lexical overlap between question and passage tokens in the SQuAD-Open and EQ datasets which renders dense retrievers at a disadvantage over BM25, especially in the transfer setting. TriviaQA  Top-20 Top-100 Top-20 Top-  ber of evidence documents, we use a batch size of 512 and retrieve 8 passages for training. Quite surprisingly, it obtains much better performance than previous approaches including BM25 on EQ (more than 10 points gain on top-20 over training ART on NQ-Open). We suspect that this may be due to the similar nature of questions in MS MARCO and EQ. Further finetuning the pretrained MS MARCO model on NQ-Full significantly improves performance on WebQ.

Scaling Model Size
We examine if scaling up the retriever parameters can offer further performance improvements. To this end, we train a retriever of BERT-large configuration (24 layers, 16 attention heads, 1024 embedding dimensions) containing around 650M parameters on NQ-Open and TriviaQA. Results are presented in Table 4 for both the development and test sets. We also include the results of other relevant baselines containing a similar number of trainable parameters. By scaling up the retriever size, we see small but consistent improvements in retrieval accuracy across both the datasets. Especially on TriviaQA, ART matches or exceeds the performance of previous best models. On NQ-Open, it comes close to the performance of EMDR 2 (Sachan et al., 2021b), a supervised model trained using thousands of question-answer pairs.
We also attempted to use larger teacher PLMs such as T0-11B. However, our initial experiments MSS init ICT init BERT init Figure 3: Effect of retriever initialization on ART training. The plot reveals that the training process is not sensitive to initial retriever parameters.

Retrieved Passages
Top-  did not lead to any further improvements over the T0-3B PLM. We conjecture that this might be either specific to these QA datasets or that we need to increase the capacity of the teacher PLM even more to observe improvements. We leave an indepth analysis of using larger teacher PLMs as part of the future work.

Analysis
Sensitivity to Retriever Initialization To examine how the convergence of ART training is affected by the initial retriever parameters, we initialize the retriever with (1) BERT weights, (2) ICT weights (as trained in Sachan et al., 2021a), and (3) MSS weights, and train using NQ-Open questions. Figure 3 displays the top-20 performance on the NQ development set as the training progresses. It reveals that ART training is not sensitive to the initial retriever parameters as all three initialization schemes converge to similar results. However, the convergence properties might be different under low-resource settings, an exploration of which we leave for future work.  Table 6: Analysis reveals that ART can even match the performance of end-to-end trained models like EMDR 2 when retrieving a larger number of passages. However, DPR and EMDR 2 still outperform ART when retrieving a small number of passages such as top-K ∈ {1, . . . , 5} (highlighted in bold). A Closer Inspection of ART with Supervised Models In order to have a better understanding of the tradeoff between supervised models and ART, we examine their top-1 and top-5 accuracy in addition to the commonly reported top-20 and top-100 scores. Table 6 presents these results for ART (large) along with supervised models of DPR (large) and EMDR 2 . Supervised models achieve much better performance for top-K ∈ {1, . . . , 5} passages, i.e., these models are more precise. This is likely because DPR is trained with hard-negative passages and EMDR 2 finetunes PLM using answers resulting in an accurate relevance feedback to the retriever.  different settings by varying the passage types. Specifically, we train with a mix of positive, hardnegative, and uniformly sampled passages. We also perform in-batch training by defining Z to be the union of positive and hard-negative passages for all the questions in a batch. Results in Table 7 illustrate that when Z consists of uniformly sampled passages, it leads to poor performance. Including a (gold) positive passage in Z leads to good performance improvements. Results further improve with the inclusion of a hard-negative passage in Z. However, in-batch training leads to a slight drop in performance. As the gold passages are not always available, our method of selecting the top passages from evidence at every training step can be seen as an approximation to using the gold passages. With this, ART obtains even better results than the previous settings, an improvement by 4 points absolute in the top-20 accuracy.

Related Work
Our work is based on training a dense retriever using pre-trained language models (PLMs), which we have covered in previous sections. Here, we instead focus on other related approaches. A popular method to train the dual-encoder retriever is to optimize contrastive loss using inbatch negatives (Gillick et al., 2019) and hardnegatives (Karpukhin et al., 2020;Xiong et al., 2021). Alternatives to using hard-negatives such as sampling from cached evidence embeddings have also shown to work well in practice (Lindgren et al., 2021). Multi-vector encoders for questions and documents are more accurate than dualencoders, (Luan et al., 2021;Khattab and Zaharia, 2020;Humeau et al., 2020), although at the cost of an increased latency and storage requirements.
PLMs have been shown to improve docu-ment rankings as they can perform cross-attention between the question and the retrieved documents (Lin et al., 2021). Supervised approaches to re-rank finetune PLMs using question-document pairs (Nogueira et al., 2020) while unsupervised approaches are based on zero-shot question generation (Sachan et al., 2022). The re-ranking process is slow due to the cross-attention step and is bottlenecked by the accuracy of first-stage retrievers. To address these limitations, cross-attention distillation approaches from the PLM to retriever have been proposed (Qu et al., 2021). Such distillation can be performed either in a single end-toend training step (Guu et al., 2020;Sachan et al., 2021b) or in a multi-stage process (Khattab et al., 2021;). An alternative approach to using PLMs is to generate data that can aid retrieval. The data can be either the title or an answer that provides more information about the question (Mao et al., 2021). Generating new questions to augment the training data has also been shown to improve performance (Ma et al., 2021;Bonifacio et al., 2022;Dai et al., 2022). In comparison, we do not generate new questions but train the retriever using existing questions and PLM feedback. Data augmentation is likely complementary, and can further improve accuracy.

Conclusions and Future Work
We introduced ART, a novel approach to train a dense passage retriever using only questions. ART does not require question-document pairs or hard-negative examples for training and yet achieves state-of-the-art results. The key to making ART work is to optimize the retriever to select relevant documents such that conditioning on them, the question generation likelihood computed using a large pre-trained language model iteratively improves. Despite requiring much less supervision, ART substantially outperforms DPR when evaluated on multiple QA datasets and also generalizes better on out-of-distribution questions.
ART presents several directions for future work. It would be interesting to apply this approach in low-resource retrieval including multilingual (Clark et al., 2020) and cross-lingual question answering (Asai et al., 2021). Our training framework can also be extended to train crossmodality retrievers such as for image or code search (Li et al., 2022;Neelakantan et al., 2022) using textual queries. Finally, another direction worth exploring would be to train multi-vector retrievers with this approach.