task_categories:text-retrieval annotations_creators:wikipedia-sourced source_datasets:original language:en license:apache-2.0 size_categories:1K<n<10K arxiv:2305.11694 region:us
---
configs:
- config_name: main
  data_files:
  - split: train
    path: train.jsonl
  - split: test
    path: test.jsonl
  - split: validation
    path: val.jsonl
license: apache-2.0
task_categories:
- text-retrieval
language:
- en
source_datasets:
- original
pretty_name: QUEST
annotations_creators:
- wikipedia-sourced
size_categories:
- 1K<n<10K
---

# Dataset Card for QUEST

## Dataset Description

- **Repository: https://github.com/google-research/language/tree/master/language/quest** 
- **Paper: https://arxiv.org/abs/2305.11694** 
- **Point of Contact: chaitanyamalaviya@gmail.com** 

### Dataset Summary

We provide here the data accompanying the paper: [QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
](https://arxiv.org/abs/2305.11694).

## Dataset Structure

### Data Instances

QUEST contains 6307 training queries, 323 examples for development, and 1727 examples for testing.

### Data Fields

Each examples file contains newline-separated json dictionaries with the following fields:

* `query` - Paraphrased query written by annotators.
* `docs` - List of relevant document titles.
* `original_query` - The original query which was paraphrased. Atomic queries are
  enclosed by `<mark></mark>`. Augmented queries do not have this field populated.
* `scores` - This field is not populated and only used when producing predictions to enable sharing the same data structure.
* `metadata` - A dictionary with the following fields:
    * `template` - The template used to create the query.
    * `domain` - The domain to which the query belongs.
    * `fluency` - List of fluency ratings for the query.
    * `meaning` - List of ratings for whether the paraphrased query meaning is the
      same as the original query.
    * `naturalness` - List of naturalness ratings for the query.
    * `relevance_ratings` - Dictionary mapping document titles to relevance ratings
      for the document.
    * `evidence_ratings` - Dictionary mapping document titles to evidence ratings
      for the document.
    * `attributions` - Dictionary mapping a document title to its attributions
      attributions are a list of dictionaries mapping a query substring to a
      document substring.

The document corpus is at https://storage.googleapis.com/gresearch/quest/documents.jsonl. Note that this file is quite large
(899MB). The format is newline separated json dicts containing `title` and
`text`.

### Citation Information

```
@inproceedings{malaviya23expertqa,
    title = {QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations},
    author = {Chaitanya Malaviya and Peter Shaw and Ming-Wei Chang and Kenton Lee and Kristina Toutanova},
    booktitle = {ACL},
    year = {2023},
    url = "https://arxiv.org/abs/2305.11694"
}
```
