{'fraction_non_alphanumeric': 0.03908657887363407, 'fraction_numerical': 0.015060240963855422, 'mean_word_length': 4.614235155328353, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
VQD: Visual Query Detection In Natural Scenes

We propose a new visual grounding task called Visual Query Detection (VQD). In VQD, the task is to localize a variable number of objects in an image where the objects are specified in natural language. VQD is related to visual referring expression comprehension, where the task is to localize only one object. We propose the first algorithms for VQD, and we evaluate them on both visual referring expression datasets and our new VQDv1 dataset.


Introduction
In computer vision, object detection is the task of identifying all objects from a specific closed-set of pre-defined classes by putting a bounding box around each object present in an image, e.g., in the widely used COCO dataset there are 80 object categories and an algorithm must put a box around all instances of each object present in an image (Lin et al., 2014). Recent deep learning based models have significantly advanced the state-of-the-art in object detection (Ren et al., 2015b); however, many applications demand more nuanced detection of objects with specific attributes or objects in relation to each other. Here, we study goal-directed object detection, where the set of possible valid objects is far greater than in the typical object detection problem. Specifically, we introduce the Visual Query Detection (VQD) task (see Fig. 1). In VQD, a system is given a query in natural language and an image and it must produce 0-N boxes that satisfy that query. VQD has numerous applications, ranging from image retrieval to robotics.
VQD is related to the visual referring expression recognition (RER) task (Kazemzadeh et al., 2014); however, in RER every image has only a single correct box. In contrast, in VQD there could be no valid outputs for a query or multiple valid outputs, making the task both harder and more useful. As discussed later, existing RER datasets have Figure 1: Unlike VQD, object detection cannot deal with attributes and relations. In VQA, often algorithms produce the right answers due to dataset bias without 'looking' at relevant image regions. RER datasets have short and often ambiguous prompts, and by having only a single box as an output, they make it easier to exploit dataset biases. VQD requires goal-directed object detection and outputting a variable number of boxes that answer a query. multiple annotation problems and have significant language bias problems. VQD is also related to Visual Question Answering (VQA), where the task is to answer questions about images in natural language (Malinowski and Fritz, 2014;Antol et al., 2015). The key difference is that in VQD the algorithm must generate image bounding boxes that satisfy the query, making it less prone to the forms of bias that plague VQA datasets.
We make the following contributions: 1. We describe the first dataset for VQD, which will be publicly released. 2. We evaluate multiple baselines on our VQD dataset.

Related work
Over the past few years, a large amount of work has been done at the intersection of computer vision and natural language understanding, including visual madlibs (Yu et al., 2015;Tommasi et al., 2018), captioning (Farhadi et al., 2010;Kulkarni et al., 2013;Johnson et al., 2016;Liu et al., 2018), and image retrieval (Wan et al., 2014;. For VQD, the most related tasks are VQA and RER, which we review in detail.

Visual Question Answering
VQA systems take in an image and open-ended natural language question and then generate a textbased answer (Antol et al., 2015;Goyal et al., 2017;Acharya et al., 2019;Kafle et al., 2018). Many VQA datasets have been created. However, initial datasets, e.g., VQAv1 (Antol et al., 2015) and COCO-QA (Ren et al., 2015a), exhibited significant language bias in which many questions could be answered correctly without looking at the image, e.g., for VQAv1 it was possible to achieve 50% accuracy using language alone (Kafle and Kanan, 2016). To address the bias issue, the VQAv2 dataset was created with a more balanced distribution for each possible answer to make algorithms analyze the image (Goyal et al., 2017), but it still had bias in the kinds of questions asked, with some questions being scarce, e.g., reasoning questions. Synthetic datasets such as the CLEVR dataset (Johnson et al., 2017) addressed this by being synthetically generated to emphasize hard reasoning questions that are rare in VQAv1 and VQAv2. The TDIUC dataset addresses bias using both synthetically generated and human gathered questions about natural images, with performance evaluated for 12 kinds of questions (Kafle and Kanan, 2017a). While the state-of-the-art has rapidly increased on both synthetic and natural image VQA datasets, many models do not generalize across datasets (Shrestha et al., 2019).

Referring Expression Recognition
Unlike VQA, RER algorithms must produce evidence to justify their outputs. A RER algorithm outputs a box around the image location matching the input string, making it easier to tell if an algorithm is behaving correctly. The RefCOCO and RefCOCO+ datasets for RER were collected from the two-player 'ReferIt' Game (Kazemzadeh et al., 2014). The first player is asked to describe an out-lined object and the second player has to correctly localize it from player one's description. The test datasets are futher split into the 'testA' and 'testB' splits. The split 'testA' contains object categories sampled randomly to be close to the original data distribution, while 'testB' contains objects sampled from the most frequent object categories, excluding categories such as 'sky', 'sand', 'floor', etc. Since, there is a time limit on the game, the descriptions are short, e.g., 'guy in a yellow t-shirt,' 'pink,' etc. Instead of playing a timed game, to create the RefCOCOg dataset for RER, one set of Amazon Mechanical Turk (AMT) users were asked to generate a description for a marked object in an image and other users marked the region corresponding to the description (Mao et al., 2016). This resulted in more descriptive prompts compared to RefCOCO and RefCOCO+.
The Visual7W dataset for VQA includes a 'pointing' task that is closely related to RER (Zhu et al., 2016). Pointing questions require choosing which box of the four given boxes correctly answered a query. Systems did not generate their own boxes, and there is always one correct box. Cirik et al. (2018) showed that RER datasets suffer from biases caused by their dataset collection procedure. For RefCOCOg, they found that randomly permuting the word in the referring expression caused only about a 5% drop in performance, suggesting that instead of relying on language structure, systems may be using some hidden correlations in language. They further showed that an image only model that ignores the referring expression yielded a precision of 71.2% for top-2 best predictions. They also found that predicting the object category given the image region produced an accuracy of 84.2% for top-2 best predictions. By having 0-N boxes, VQD is harder for an imageonly model to perform well.
3 The VQD 1.0 Dataset (VQDv1) We created VQDv1, the first dataset for VQD. VQDv1 is created synthetically using annotations from Visual Genome (VG), COCO, and COCO Panoptic. While this limits variety, it helps combat some kinds of bias and serves as an initial version of the dataset. VQDv1 has three distinct query categories: 1. Object Presence (e.g., 'Show the dog in the image') 2. Color Reasoning (e.g., 'Which plate is white   in color?') 3. Positional Reasoning (e.g., 'Show the cylinder behind the girl in the picture') The number of queries per type are given in the Table 1. The dataset statistics and example images and are shown in Fig. 2 and Fig. 3, respectively. We show statistics for VQDv1 compared to RER datasets in Table 2.
All images in VQDv1 are from COCO. The ground truth bounding box annotations are derived from the COCO Panoptic annotations dataset (Kirillov et al., 2018). The questions are generated using multiple templates for each question type, which is an approach that has been used in earlier work for VQA (Kafle and Kanan, 2017a;Kafle et al., 2017). The query objects and their attributes are extracted by integrating the annotations from images that have both COCO and VG annotations. COCO annotations are focused on objects, while VG also has attribute and relationship information, e.g., size, color, and actions for scene objects.

Object Presence
Object presence questions require an algorithm to determine all instances of an object in an image without any relationship or positional attributes, for example, 'Show me the horse in the image' or 'Where is the chair?' We use all of the COCO 'things' labels and half of the COCO 'stuff' labels to generate these questions, making this task test the same capabilities as conventional object detection. We filter some 'stuff' categories that do not have well defined bounding boxes such as 'waterother', 'floor-stone', etc. We use multiple templates to create variety, e.g., 'Show the <object> in the image', 'Where are the <object> in the picture?' etc.

Color Reasoning
Color questions test the presence of objects modified by color attributes, e.g., 'Show me the cat which is grey in color' or 'Which blanket is blue in color?' Since, COCO has only object annotations, color attributes are derived from VG's attribute annotations. We align every VG image annotation with COCO annotations to obtain (object, color) annotations for each bounding box. When multiple color attributes for an object are present, the object is assigned a single color from that attribute set.

Positional Reasoning
Positional reasoning questions test the location of objects with respect to other objects, e.g., 'Show the building behind horses', 'Which people are in front of the lighthouse?', and 'Show the rhino behind elephant.' We again use VG's relationship and attribute annotations to create these questions.

Generating Counter-Concept Questions
Counter-concept questions have no valid boxes as outputs, and we endeavor to create hard counterconcept questions for each category. We ask 'Show me the zebra' only if there is a similar animal present (e.g., a cow), which was done by using COCO's super-categories. Likewise, 'Show me the donut that is brown in color' is only asked if a brown donut does not exist in the image.

Experiments
Our experiments are designed to probe the behavior of models on VQD compared to RER datasets. To facilitate this, we created a variant of our VQDv1 dataset that had only a single correct bounding box.
To evaluate performance for the RER and '1 Obj' version of the VQDv1 dataset, systems only output a single bounding box during test time, so the Precision@1 metric is used. For the '0-N Obj' version of the VQDv1 dataset, we use the standard PASCAL VOC metric AP IoU =.50 from object detection, which calculates the average precision across the dataset using an intersection over union (IoU) greater than 0.5 criteria for matching with the ground truth boxes.

Models Evaluated
We implemented and evaluated four models for VQD. All models are built on top of Faster R-CNN with a ResNet-101 backbone whose output bounding boxes pass through Non-Maximal Suppression  Table 3: Results on RER datasets and two versions of our VQD dataset. The '1 Obj' version is trained and evaluated on queries with only a single box, analogous to RER, and the 0-N version contains the entire VQD dataset. All models use the same object proposals and visual features.
with a threshold of 0.7. This acts as a region proposal generator that provides CNN features for each region.
The four models we evaluate are: The primary reason for providing VQDv1 (1 obj.) and the RER results is to put the benefits of the VQD task in context. To aid in this endeavor, we also include comparison results directly from the SLR models (Yu et al., 2017) for RER, which is a recent system for that task.

Training Details
The Query-Blind and Vision+Query models are trained with binary cross-entropy loss. We use a learning rate of 0.0004, and perform learning rate decay of 0.8 when the training loss plateaus continuously for five epochs. The best model is selected based on the validation loss after training for 50 epochs.

Results
Our main results are given in Table 3. Although simple, our Vision+Query model performs well across RER datasets, but it can also be applied to VQD tasks. As expected, RANDOM performs poorly on both VQDv1 datasets. DETECT beats RANDOM in the single object VQD setting by a large margin. Since, most of the questions in the RER datasets ask about common COCO categories, choosing one of those objects might be enough to get decent performance; however, DETECT performs poorly when evaluated under 0-N object settings in VQDv1. To handle queries in VQD, models must be able to understand the context and comprehend multiple objects in isolation.

Conclusion
In this paper, we described our VQDv1 dataset as a test for visual grounding via goal-directed object detection. VQDv1 has both simple object presence and complex questions with 0-N bounding boxes. While VQDv1 contains only synthetically generated questions, this can help mitigate some forms of bias present in other VQA and RER datasets (Cirik et al., 2018;Kafle and Kanan, 2017b). While it would be expensive, a large, carefully filtered, and well designed human annotated VQD dataset is the next step toward advancing visual grounding research.
Compared to VQA, we argue that it is harder to be right for the wrong reasons in VQD because methods must generate bounding boxes. Compared to RER, we argue that it is harder to exploit bias in VQD since there are a variable number of boxes per image, making it considerably more difficult, as demonstrated by our experiments. We believe the VQD approach has considerable value and can be used to advance visual grounding research.  (f) Where is the sink in the picture? Where is the toaster in the image? Figure 3: Example query-detection pairs from the VQD dataset. Counter context questions that do not have a bounding box as an answer are generated in such a way that they are still relevant to the scene context. For example, in Fig. [3f] both questions pertain to the context 'kitchen'.