license:mit region:us
---
license: mit
---

# Synthetic Passages for MS MARCO Queries

This dataset has been generated as part of a master's thesis. The goal was to investigate how Large Language Models can be used to generate synthetic data for the MS MARCO queries and how the would compare to the original MS MARCO dataset. The code for the data generation as well as the experiments can be found in [this repository](https://github.com/malteos/thesis-jan/).

The data generation follows an approach where queries are sampled from the MS Marco training set and then used to generate passages with a Large Language Model. The generated dataset always contains 4 passages, that represent different levels of difficulty. The levels of difficulty are defined as follows:

- **Easy positive**: A text passage that is highly relevant and directly addresses the given query.
- **Hard positive**: A text passage that has the same necessary information as the easy positive passage but contains more noise or irrelevant information, making it harder to identify as relevant.
- **Hard negative**: A text passage that shares some topical or semantic similarities with the given query but is not a direct match.
- **Very hard negative**: A text passage that is similar to the easy positive passage but contains subtle differences or nuances that make it irrelevant to answering the query.

The model used for the generation is the `upstage/SOLAR-10.7B-Instruct-v1.0` model.

### Prompt Methods

Four different prompt methods haven been investigated:

1. **individual**: This prompt generates each passage individually.
2. **one-step**: This prompt generates all passages in one step and outputs them in JSON format.
3. **chain-of-thought**: This prompt first generates some rationale for the passages and then generates the passages based on the rationale.
4. **text-then-json**: This prompt first generates the text for all passages and then converts them to JSON format in the second step.

### Data Format

The format of the dataset consists of two files. One that contains all the passages and one that contains the query and the references to the passages. The format is like this:

**Queries File**
```json
{
    "query_id": "108689",
    "query": "cost subdividing",
    "msmarco_positive": {"doc_id": "0"},
    "positives": {"doc_id": ["1", "2"]},
    "negatives": {"doc_id": ["3", "4"]}
}
```

**Passages File**
```json
{
    "0": {
        "text": "..."
    },
    "1": {
        "text": "..."
    },
    "2": {
        "text": "..."
    }, ...
}
```