task_categories:text-retrieval language:no license:mit size_categories:10K<n<100K format:parquet modality:text library:datasets library:pandas library:mlcroissant library:polars arxiv:2401.00368 region:us
---
dataset_info:
  features:
  - name: response
    dtype: string
  - name: model
    dtype: string
  - name: prompt
    list:
    - name: content
      dtype: string
    - name: role
      dtype: string
  splits:
  - name: train
    num_bytes: 156271618
    num_examples: 50000
  download_size: 53985283
  dataset_size: 156271618
configs:
- config_name: default
  data_files:
  - split: train
    path: data/train-*
license: mit
task_categories:
- text-retrieval
language:
- 'no'
---
# Thanks to Arrow Denmark and Nvidia for sponsoring the compute used to generate this dataset
The purpose of this dataset is to pre- or post-train embedding models for retrieval tasks.

The dataset consists of 100,000 samples generated with gemma-2-27b-it.

The column "prompt" shows the prompt given to the LLM and "response" shows the LLM output.

Each sample in the dataset was generated from a seed task randomly sampled from https://huggingface.co/datasets/ThatsGroes/retrieval-tasks-processed

The data generation process described in this paper was followed:

https://arxiv.org/pdf/2401.00368

Compute sponsored by Arrow Denmark and Nvidia through Danish Data Science Community.