size_categories:n<1K format:parquet modality:tabular modality:text library:datasets library:pandas library:mlcroissant library:polars library:distilabel arxiv:2310.01377 region:us synthetic distilabel rlaif
---
size_categories: n<1K
dataset_info:
  features:
  - name: instruction
    dtype: string
  - name: generations
    sequence: string
  - name: model_names
    sequence: string
  - name: distilabel_metadata
    struct:
    - name: raw_input_ultra_feedback_0
      list:
      - name: content
        dtype: string
      - name: role
        dtype: string
    - name: raw_output_ultra_feedback_0
      dtype: string
  - name: types
    sequence: 'null'
  - name: rationales
    sequence: string
  - name: ratings
    sequence: int64
  - name: rationales-for-ratings
    sequence: string
  - name: model_name
    dtype: string
  - name: prompt
    dtype: string
  - name: prompt_id
    dtype: string
  - name: chosen
    list:
    - name: content
      dtype: string
    - name: role
      dtype: string
  - name: chosen_rating
    dtype: int64
  - name: rejected
    list:
    - name: content
      dtype: string
    - name: role
      dtype: string
  - name: rejected_rating
    dtype: int64
  splits:
  - name: train
    num_bytes: 7920
    num_examples: 2
  download_size: 28465
  dataset_size: 7920
configs:
- config_name: default
  data_files:
  - split: train
    path: data/train-*
tags:
- synthetic
- distilabel
- rlaif
---

<p align="left">
  <a href="https://github.com/argilla-io/distilabel">
    <img src="https://raw.githubusercontent.com/argilla-io/distilabel/main/docs/assets/distilabel-badge-light.png" alt="Built with Distilabel" width="200" height="32"/>
  </a>
</p>

# Dataset Card for example-preference-dataset2_work5

This dataset has been created with [distilabel](https://distilabel.argilla.io/).



## Dataset Summary

This dataset contains a `pipeline.yaml` which can be used to reproduce the pipeline that generated it in distilabel using the `distilabel` CLI:

```console
distilabel pipeline run --config "https://huggingface.co/datasets/ashercn97/example-preference-dataset2_work5/raw/main/pipeline.yaml"
```

or explore the configuration:

```console
distilabel pipeline info --config "https://huggingface.co/datasets/ashercn97/example-preference-dataset2_work5/raw/main/pipeline.yaml"
```

## Dataset structure

The examples have the following structure per configuration:


<details><summary> Configuration: default </summary><hr>

```json
{
    "chosen": [
        {
            "content": "it means like its like everywhere but like at the same time",
            "role": "user"
        },
        {
            "content": "[\u0027ubiquitous\u0027]",
            "role": "assistant"
        }
    ],
    "chosen_rating": 5,
    "distilabel_metadata": {
        "raw_input_ultra_feedback_0": [
            {
                "content": "Your role is to evaluate text quality based on given criteria.\nYou\u0027ll receive an instructional description (\"Instruction\") and 2 text outputs (\"Text\").\nUnderstand and interpret instructions to evaluate effectively.\nProvide annotations for each text with a rating and rationale.\nThe 2 texts given are independent, and should be evaluated separately.\n",
                "role": "system"
            },
            {
                "content": "# Truthfulness and Hallucination Assessment\n\nEvaluate the model\u0027s accuracy in providing information without introducing misleading or fabricated details. \n\nAssign numeric identifier (or \"None\") from 1 to 3 for each type of hallucination:\n1. **Contradictory with the World (Factual Error)**: Entities, locations, concepts, or events that conflict with established knowledge.\n2. **Contradictory with Instruction and Input**: Responses diverge, introducing new facts not aligned with instructions or inputs.\n3. **Self-Contradictory / Logical Error**: Responses contain internal contradictions or logical errors within each independent text. \n\n**Scoring**: Rate outputs 1 to 5 based on extent of hallucination:\n1. **Completely Hallucinated**: Entirely unreliable due to hallucinations.\n2. **Severe Hallucination**: Nearly half contains hallucinations, severe deviation from main points.\n3. **Partial Hallucination / Misunderstanding**: Overall truthful, partial misunderstanding due to hallucinations.\n4. **Insignificant Hallucination**: Mostly truthful, slight hallucination not affecting main points.\n5. **No Hallucination**: Free of hallucinations.\n\n---\n\n## Format\n\n### Input\nInstruction: [Specify task goal and restrictions]\n\nTexts:\n\u003ctext 1\u003e [Text 1]\n\u003ctext 2\u003e [Text 2]\n\n### Output\n#### Output for Text 1\nType: [List of numeric identifiers (or \"None\" if no hallucination observed) of hallucination types, separated by commas]\nRationale: [Rationale for the identification in short sentences]\nRating: [Rating for text 1]\nRationale: [Rationale for the rating in short sentences]\n\n#### Output for Text 2\nType: [List of types]\nRationale: [Rationale]\nRating: [Rating]\nRationale: [Rationale]\n\n---\n\n## Annotation\n\n### Input\nInstruction: it means like its like everywhere but like at the same time\n\nTexts:\n\u003ctext 1\u003e [\u0027ubiquitous\u0027]\n\u003ctext 2\u003e [\u0027everywhere\u0027]\n\n### Output\n",
                "role": "user"
            }
        ],
        "raw_output_ultra_feedback_0": "#### Output for Text 1\nType: None\nRationale: The response \"ubiquitous\" accurately reflects the meaning of \"like its like everywhere but like at the same time\" without introducing any misleading or fabricated details.\nRating: 5\nRationale: The text is entirely accurate and free of hallucinations.\n\n#### Output for Text 2\nType: None\nRationale: The word \"everywhere\" directly corresponds to the instruction, conveying the essence of being present in all places simultaneously without any inaccuracies.\nRating: 5\nRationale: The text is completely truthful and does not contain any hallucinations."
    },
    "generations": [
        "[\u0027ubiquitous\u0027]",
        "[\u0027everywhere\u0027]"
    ],
    "instruction": "it means like its like everywhere but like at the same time",
    "model_name": "gpt-4o-mini",
    "model_names": [
        "https://jvo0ibb76593l20m.us-east-1.aws.endpoints.huggingface.cloud",
        "https://jvo0ibb76593l20m.us-east-1.aws.endpoints.huggingface.cloud"
    ],
    "prompt": "it means like its like everywhere but like at the same time",
    "prompt_id": "bb6a2f897a05d2149f0198baadc0e8b809eb3650d9ffec753972cec7cdc483f3",
    "ratings": [
        5,
        5
    ],
    "rationales": [
        "The response \"ubiquitous\" accurately reflects the meaning of \"like its like everywhere but like at the same time\" without introducing any misleading or fabricated details.",
        "The word \"everywhere\" directly corresponds to the instruction, conveying the essence of being present in all places simultaneously without any inaccuracies."
    ],
    "rationales-for-ratings": [
        "The text is entirely accurate and free of hallucinations.",
        "The text is completely truthful and does not contain any hallucinations."
    ],
    "rejected": [
        {
            "content": "it means like its like everywhere but like at the same time",
            "role": "user"
        },
        {
            "content": "[\u0027ubiquitous\u0027]",
            "role": "assistant"
        }
    ],
    "rejected_rating": 5,
    "types": [
        null,
        null
    ]
}
```

This subset can be loaded as:

```python
from datasets import load_dataset

ds = load_dataset("ashercn97/example-preference-dataset2_work5", "default")
```

Or simply as it follows, since there's only one configuration and is named `default`: 

```python
from datasets import load_dataset

ds = load_dataset("ashercn97/example-preference-dataset2_work5")
```


</details>





## References


```
@misc{cui2024ultrafeedbackboostinglanguagemodels,
    title={UltraFeedback: Boosting Language Models with Scaled AI Feedback},
    author={Ganqu Cui and Lifan Yuan and Ning Ding and Guanming Yao and Bingxiang He and Wei Zhu and Yuan Ni and Guotong Xie and Ruobing Xie and Yankai Lin and Zhiyuan Liu and Maosong Sun},
    year={2024},
    eprint={2310.01377},
    archivePrefix={arXiv},
    primaryClass={cs.CL},
    url={https://arxiv.org/abs/2310.01377},
}
```


