{'fraction_non_alphanumeric': 0.037069246657245356, 'fraction_numerical': 0.016523535166865964, 'mean_word_length': 4.77858040201005, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Passage Retrieval for Outside-Knowledge Visual Question Answering

In this work, we address multi-modal information needs that contain text questions and images by focusing on passage retrieval for outside-knowledge visual question answering. This task requires access to outside knowledge, which in our case we define to be a large unstructured passage collection. We first conduct sparse retrieval with BM25 and study expanding the question with object names and image captions. We verify that visual clues play an important role and captions tend to be more informative than object names in sparse retrieval. We then construct a dual-encoder dense retriever, with the query encoder being LXMERT, a multi-modal pre-trained transformer. We further show that dense retrieval significantly outperforms sparse retrieval that uses object expansion. Moreover, dense retrieval matches the performance of sparse retrieval that leverages human-generated captions.


INTRODUCTION
: An example of passage retrieval for OK-VQA. Boldface denotes a potential answer. Image ©gsloan, https://www. flickr.com/photos/gsloan/8137199999/ Specifically, we focus on a Visual QA (VQA) task referred to as Outside-Knowledge VQA (OK-VQA). Classic VQA benchmarks [1,12,25,46,47] and models [3,10,17,23,26,41] mainly focus on questions about counting, visual attributes, or other visual detection tasks, whose answers can be found in the given image. In contrast, images in our task help to define the information need, instead of simply being the knowledge source by which the question is answered. OK-VQA resembles open-domain QA [37] in the sense that both tasks require access to an outside and open knowledge resource, e.g., a large collection of passages, to answer the questions. Open-domain QA systems typically follow a retrieveand-read paradigm [5,16,20,[30][31][32]43], where the system first retrieves a number of documents (passages) from a collection and then extracts answers from them. This paradigm is less studied for multi-modal information needs, which is the focus of this paper. In this work, we focus on the retrieval phase for OK-VQA as illustrated in Fig. 1.
Unlike previous knowledge-based VQA work that retrieves knowledge from a knowledge base [11, 21, 28, 29, 38-40, 45, 48] or using a Wikipedia Search API [27], we systematically study passage retrieval for OK-VQA with generic information retrieval approaches so that our methods can be applied to a wider range of unstructured knowledge resources. In particular, we seek answers to the following research questions: (RQ1) How helpful are the visual signals in OK-VQA? (RQ2) What is the most effective way to incorporate visual signals into sparse retrieval models that are based on term matching? (RQ3) How well does dense retrieval [13,16,20,24,42] work with multi-modal information needs?
To answer these important research questions, we study passage retrieval for OK-VQA queries with a large Wikipedia passage collection. We first conduct sparse retrieval with BM25. We investigate the performance of expanding the original question with different human-annotated object names and image captions. We further study the impact of using different rank fusion methods for different expansion types. We verify that visual clues play an important role in our task. In particular, captions tend to be more informative than object names in sparse retrieval. We further reveal that it is desirable to exploit the most salient matching signal (CombMAX [9,19]) when using object expansion while it is better to consider the matching signals for all captions with CombSUM [9,19] or Reciprocal Rank Fusion [14] when we expand with human-generated captions.
We then adopt a dual-encoder architecture to construct a learnable dense retriever following previous work [16,20,24,30,42]. We employ LXMERT [35], a pre-trained Transformer model [36], as our multi-modal query encoder to encode both the text question and the image as an information need. We observe that our dense retriever achieves a statistically significant performance improvement over sparse retrieval that leverages object expansion, demonstrating the effectiveness of dense retrieval with a multimodal query encoder. Furthermore, our dense retriever manages to match the performance of sparse retrieval with caption expansion, even though the latter leverages human-generated captions that are often highly informative. Our research is one of fundamental steps for future studies on retrieval-based OK-VQA. Our code is released for research purposes. 1

PASSAGE RETRIEVAL FOR OK-VQA 2.1 Task Definition
We are given an information need (query) denoted as = ⟨ , ⟩. It consists of a text question and an image . The task is to retrieve passages that can be used to fulfill , from a large passage collection. Following the work on open-domain QA [16,20,30], a passage is deemed as relevant if it contains the ground truth answer.

Sparse Retrieval
The backbone of our sparse retrieval approach is BM25, which works with text queries. Therefore, we expand with different textual descriptions of visual clues to construct the BM25 queries. Visual signals in an image are typically expressed in two forms. The first form is a set of object names { 1 , 2 , · · · } produced by an object detector. Each object is a Region of Interest (RoI) that reveals a meaningful component of the image. The second form is a set of captions { 1 , 2 , · · · } produced by an image descriptor to describe the image. We adopt human-annotated object names and captions for sparse retrieval. Although the human annotations do not necessarily give the performance upper bound, they would be strong baselines for dense retrieval and make sure that our analysis will not be affected by the quality of automatic annotations produced by object detectors and image descriptors. We study different expansion of visual signals as follows: • BM25-Orig: taking the original only, i.e., orig = { }.
Since obj/cap/all contains multiple BM25 queries for the same information need , we need rank fusion methods to consolidate the ranked lists generated by queries within each query set. This resembles an ensemble process to combine results obtained with  different visual signals. We consider CombMAX [9,19] (taking the maximum score of a passage in different ranked lists), Comb-SUM [9,19] (taking the sum of scores of a passage in different ranked lists), and RRF (Reciprocal Rank Fusion) [6] (the fusion score for a passage is defined as is the rank of ). CombMAX could help the model be more robust to distracting visual signals while the other two fusion approaches make sure that the impacts of lower-ranked passages do not vanish.

Dense Retrieval
Following previous work [16,20,30,42], we adopt a dual-encoder architecture to construct a learnable retriever. The retrieval process is "dense" in the sense that the queries and passages are encoded to low-dimensional dense vectors, as opposed to the high-dimensional sparse vectors used in sparse retrieval. As shown in Fig. 2, the retriever consists of a query encoder and a passage encoder.

Query Encoder.
We adopt the LXMERT model [35] as the query encoder since it can encode both the question and image components of . LXMERT is a pre-trained Transformer model [36] designed to learn vision and language connections. It consists of three encoders, an object relationship encoder, a language encoder, and a cross-modality encoder. The first two single-modality encoders function in a similar way to BERT [8] except that the object relationship encoder works with a set of object detections produced by a Faster R-CNN [34] model pre-trained on Visual Genome [2,18]. Each detection representation can be considered as an "image token embedding" that consists of its RoI features (fixed) and position features (trainable). The cross-modality encoder conducts bi-directional cross attention between vision and language representations. We refer our readers to the LXMERT paper [35] for further details. We project the cross-modality output of LXMERT to an -dimensional query representation. The dense retriever with a LXMERT query encoder is referred to as Dense-LXMERT. To adopt a deeper analytical view, we also consider BERT as the query encoder, which only works with the question component of the query, resulting in the Dense-BERT model.

Passage
Encoder. We use BERT as the passage encoder and project the [CLS] representation to an -dimensional passage representation. The retrieval score is defined as the dot product of the query and passage representations. After training, we encode all passages in the collection during an offline process. At inference time, we use Faiss [15] for maximum inner product search.

2.3.3
Training. We train the dual-encoder retriever with a set of training instances. Each instance is denoted as ⟨ , + , − ⟩, where + is a positive passage that contains the answer while − is a negative passage that does not contain the answer. In our case, we select − from the top passages retrieved by a sparse retrieval method. Thus, − can be referred to as a retrieved negative. We present more details on constructing the training data in Sec. 3.1.2.
In addition to the retrieved negatives, one might want to take advantage of the other passages in the batch as in-batch negatives. Although in-batch negatives resemble randomly sampled negatives that can be less effective [16], it is extremely efficient since passage representations can be reused within the batch. Karpukhin et al. [16] studied combining in-batch negatives with retrieved negatives for uni-modal queries. We further dig into this topic for multi-modal queries. We consider the following negative sampling strategies: • R-Neg: using the retrieved negative passage only.
• R-Neg+IB-Neg: using the retrieved negative, along with all other in-batch negative passages of other instances. • R-Neg+IB-Pos: using the retrieved negative, along with all other in-batch positive passages of other instances. • R-Neg+IB-All: using the retrieved negative, along with all other in-batch passages, except for + . The same query can be paired with different positive and negative passages to augment the training data as suggested in Sec. 3.1.2. Therefore, the queries within a batch can coincide even with random batching. In this case, the misuse of a positive passage as negative may hinder the learning process. We empirically examine whether this concern holds by comparing R-Neg+IB-All/Pos to R-Neg+IB-Neg. Following previous work [16,30], we use cross entropy loss to maximize the probability of the positive passage given the negatives identified above. We then average the losses for queries in the batch.

EXPERIMENTS 3.1 Experimental Setup
3.1.1 Dataset. Our retrieval dataset is based on the OK-VQA dataset [27], where all questions require outside knowledge. 2 The images in the OK-VQA dataset are from the COCO dataset [12]. We take the original training queries as our training queries and split the original validation queries into our validation and testing queries. In terms of the collection, we take the Wikipedia passage collection with 11 million passages created by previous work [30]. 3 Each passage contains at most 384 "wordpieces" [8] with intact sentence boundaries. Data statistics are presented in Tab. 1.

Data Construction for Dense Retrieval.
We create the training instances described in Sec. 2.3.3 using retrieved passages of sparse retrieval (see configuration details in Sec. 3.1.4). A passage is identified to be positive if it contains an exact match (caseinsensitive) of a ground truth answer. The other retrieved passages are considered as negatives. We take the top 5 positive passages, each repeated 5 times (for augmentation), to construct training instances with random retrieved negatives. In addition, we put together a small validation collection by taking the top 20 passages for each question. Data statistics are presented in Tab. 1.

Evaluation Metrics.
We focus on passage retrieval for multimodal information needs as the first step in the OK-VQA pipeline. The output of the retrieval process will be used by a reader model to extract the answer. Therefore, following previous work [5,16],  [4,6]. Human-annotated object names and captions are from the COCO dataset [12]. For dense retrieval, we use the HuggingFace transformers library 5 for the implementations of LXMERT and BERT. We set the maximum sequence length of the query encoder to 20 [35], that of the passage encoder to 384, the projection size ( ) for the query/passage representations to 768, the learning rate to 1e-5, the batch size to 4 per GPU, and the number of fine-tuning epochs to 2. We adopt R-Neg+IB-All as the negative sampling strategy. We save checkpoints every 5,000 steps and evaluate on the validation set to select the best model for the test set. The training time is 10 hours for Dense-BERT and 12 hours for Dense-LXMERT. All models are trained with 4 GPUs with mixed-precision training. Warm-up takes 10% of the total steps. The training instances are constructed with the top 100 retrieved passages for each question using BM25-Cap (CombSUM) with the default BM25 configuration in Anserini ( 1 = 0.9, = 0.4).

Sparse Retrieval.
We present the results for sparse retrieval in Tab. 2 to answer RQ1 and RQ2 raised in Sec. 1. First, we observe that approaches that consider visual signals outperform BM25-Orig by a large margin, verifying that visual clues are helpful in our task. We then compare BM25 with different forms of visual clues. Methods with captions (BM25-Cap/All) outperform object expansion, indicating that captions are more informative than object names. This makes sense since captions typically cover important objects descriptively. BM25-All does not benefit from incorporating both objects and captions. On the contrary, objects can be distracting and hurt the performance gain from captions. Finally, we compare different rank fusion methods. When objects are being considered (BM25-Obj/All), CombMAX yields the best performance since it is robust to potentially misleading objects by only considering objects with the best matching score. On the other hand, CombSUM and RRF work well with caption expansion. Their ability to consider the impact of all captions is desirable since captions are closely connected to the image and can be diverse and complementary. The best performing approach is BM25-Cap with CombSUM.

Dense
Retrieval. We present the dense retrieval results, along with the best sparse retrieval results in Tab. 3 to answer RQ3.
We first compare retrieval without visual clues: we observe that Dense-BERT outperforms BM25-Orig by a large margin, verifying  We further observe that Dense-LXMERT significantly outperforms BM25-Obj. Dense-LXMERT leverages both the RoI features and position features in object detection to learn object relations, which can be more effective than using ground truth object names in sparse retrieval. We then compare Dense-LXMERT with BM25-Cap/All. These sparse retrieval methods consider human-annotated image captions that are highly informative and descriptive. On the contrary, Dense-LXMERT has to learn the importance of the objects and the relation among them with object-level features. In this unfavorable situation, Dense-LXMERT still manages to match the performance of BM25-Cap/All. Although BM25-Cap is slightly better, the margins are statistically insignificant. Finally, we observe that Dense-LXMERT significantly outperforms Dense-BERT, further validating the use of a multi-modal query encoder.

Additional Results
We provide additional analysis to study the impact of the projection size and negative sampling strategies with validation performance.  3.3.1 Impact of projection size. We present the impact of the dimensionality of query/passage representations in Fig. 3a. We observe that a larger projection size always leads to better performance, although the performance gain seems to be insignificant for LXMERT after = 256. We set = 768 as reported in Sec.3.1.4 since it gives the best performance for both Dense-LXMERT and Dense-BERT. When working with a much larger collection than ours (ours has 11 million passages), one might want to use = 256 since it offers similar performance with less memory consumption.
3.3.2 Impact of negative sampling. The desirable performance of dense retrieval is contingent upon the negative sampling strategy. We present the impact of different sampling methods described in Sec. 2.3.3 in Fig. 3b. We observe that combining retrieved negatives with in-batch negatives dramatically improves the model performance, verifying the observations in Karpukhin et al. [16] for multi-modal queries. Also, different choices of in-batch negatives (R-Neg+IB-Neg/Pos/All) give a similar performance for LXMERT, indicating that coinciding questions in the same batch should not be a concern for our batch size and data size reported in Sec. 3.1. Both analyses show that Dense-BERT is more demanding on larger model capacity (larger projection size) and more negative samples. We speculate that BERT is overfitting the patterns in the training data since it lacks important visual clues for matching. In comparison, Dense-LXMERT is less sensitive to reasonably-chosen projection sizes and negative sampling strategies because it can learn matching signals from both language and vision clues.

CONCLUSIONS AND FUTURE WORK
We study passage retrieval for OK-VQA with sparse and dense retrieval and verify visual clues play an important role. We discover that captions are more informative than object names in sparse retrieval and CombMAX works well with object expansion while CombSUM and RRF are better for caption expansion. We further show a dense retriever with a multi-modal query encoder can significantly outperform sparse retrieval with object expansion and even matches the performance of that with human-generated captions. In the future, we will consider using automatic captions for sparse retrieval and study answer extraction to complete the QA pipeline.

ACKNOWLEDGMENTS
This work was supported in part by the Center for Intelligent Information Retrieval. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect those of the sponsors.