size_categories:n<1K format:parquet modality:text library:datasets library:pandas library:mlcroissant library:polars library:distilabel arxiv:2310.01377 region:us synthetic distilabel rlaif
---
size_categories: n<1K
dataset_info:
  features:
  - name: instruction
    dtype: string
  - name: generations
    sequence: string
  - name: model_names
    sequence: string
  - name: distilabel_metadata
    struct:
    - name: raw_input_ultra_feedback_0
      list:
      - name: content
        dtype: string
      - name: role
        dtype: string
    - name: raw_output_ultra_feedback_0
      dtype: string
  - name: types
    sequence: 'null'
  - name: rationales
    sequence: string
  - name: ratings
    sequence: int64
  - name: rationales-for-ratings
    sequence: string
  - name: model_name
    dtype: string
  - name: prompt
    dtype: 'null'
  - name: prompt_id
    dtype: 'null'
  - name: chosen
    dtype: 'null'
  - name: chosen_model
    dtype: 'null'
  - name: chosen_rating
    dtype: 'null'
  - name: rejected
    dtype: 'null'
  - name: rejected_model
    dtype: 'null'
  - name: rejected_rating
    dtype: 'null'
  splits:
  - name: train
    num_bytes: 7627
    num_examples: 2
  download_size: 29028
  dataset_size: 7627
configs:
- config_name: default
  data_files:
  - split: train
    path: data/train-*
tags:
- synthetic
- distilabel
- rlaif
---

<p align="left">
  <a href="https://github.com/argilla-io/distilabel">
    <img src="https://raw.githubusercontent.com/argilla-io/distilabel/main/docs/assets/distilabel-badge-light.png" alt="Built with Distilabel" width="200" height="32"/>
  </a>
</p>

# Dataset Card for example-preference-dataset2_work2

This dataset has been created with [distilabel](https://distilabel.argilla.io/).



## Dataset Summary

This dataset contains a `pipeline.yaml` which can be used to reproduce the pipeline that generated it in distilabel using the `distilabel` CLI:

```console
distilabel pipeline run --config "https://huggingface.co/datasets/ashercn97/example-preference-dataset2_work2/raw/main/pipeline.yaml"
```

or explore the configuration:

```console
distilabel pipeline info --config "https://huggingface.co/datasets/ashercn97/example-preference-dataset2_work2/raw/main/pipeline.yaml"
```

## Dataset structure

The examples have the following structure per configuration:


<details><summary> Configuration: default </summary><hr>

```json
{
    "chosen": null,
    "chosen_model": null,
    "chosen_rating": null,
    "distilabel_metadata": {
        "raw_input_ultra_feedback_0": [
            {
                "content": "Your role is to evaluate text quality based on given criteria.\nYou\u0027ll receive an instructional description (\"Instruction\") and 2 text outputs (\"Text\").\nUnderstand and interpret instructions to evaluate effectively.\nProvide annotations for each text with a rating and rationale.\nThe 2 texts given are independent, and should be evaluated separately.\n",
                "role": "system"
            },
            {
                "content": "# Truthfulness and Hallucination Assessment\n\nEvaluate the model\u0027s accuracy in providing information without introducing misleading or fabricated details. \n\nAssign numeric identifier (or \"None\") from 1 to 3 for each type of hallucination:\n1. **Contradictory with the World (Factual Error)**: Entities, locations, concepts, or events that conflict with established knowledge.\n2. **Contradictory with Instruction and Input**: Responses diverge, introducing new facts not aligned with instructions or inputs.\n3. **Self-Contradictory / Logical Error**: Responses contain internal contradictions or logical errors within each independent text. \n\n**Scoring**: Rate outputs 1 to 5 based on extent of hallucination:\n1. **Completely Hallucinated**: Entirely unreliable due to hallucinations.\n2. **Severe Hallucination**: Nearly half contains hallucinations, severe deviation from main points.\n3. **Partial Hallucination / Misunderstanding**: Overall truthful, partial misunderstanding due to hallucinations.\n4. **Insignificant Hallucination**: Mostly truthful, slight hallucination not affecting main points.\n5. **No Hallucination**: Free of hallucinations.\n\n---\n\n## Format\n\n### Input\nInstruction: [Specify task goal and restrictions]\n\nTexts:\n\u003ctext 1\u003e [Text 1]\n\u003ctext 2\u003e [Text 2]\n\n### Output\n#### Output for Text 1\nType: [List of numeric identifiers (or \"None\" if no hallucination observed) of hallucination types, separated by commas]\nRationale: [Rationale for the identification in short sentences]\nRating: [Rating for text 1]\nRationale: [Rationale for the rating in short sentences]\n\n#### Output for Text 2\nType: [List of types]\nRationale: [Rationale]\nRating: [Rating]\nRationale: [Rationale]\n\n---\n\n## Annotation\n\n### Input\nInstruction: it means like its like everywhere but like at the same time\n\nTexts:\n\u003ctext 1\u003e It sounds like you might be describing a concept that is both ubiquitous and simultaneously omnipresent. This could relate to various ideas, such as collective consciousness, global connectivity, or even something like a vibe or trend that permeates many aspects of life. Could you provide a bit more context or clarify what you are referring to?\n\u003ctext 2\u003e [\u0027everywhere_at_once\u0027]\n\n### Output\n",
                "role": "user"
            }
        ],
        "raw_output_ultra_feedback_0": "#### Output for Text 1\nType: [None]\nRationale: The response accurately interprets the vague instruction and suggests plausible concepts that align well with the described idea, such as \"collective consciousness\" and \"global connectivity.\" There are no evident factual errors, contradictions with the instruction, or internal logical errors.\nRating: 5\nRationale: The response is free from hallucinations, providing a coherent interpretation without introducing any misleading information or fabrications.\n\n#### Output for Text 2\nType: [None]\nRationale: The response \"[\u0027everywhere_at_once\u0027]\" matches the vague and abstract nature of the instruction without"
    },
    "generations": [
        "It sounds like you might be describing a concept that is both ubiquitous and simultaneously omnipresent. This could relate to various ideas, such as collective consciousness, global connectivity, or even something like a vibe or trend that permeates many aspects of life. Could you provide a bit more context or clarify what you are referring to?",
        "[\u0027everywhere_at_once\u0027]"
    ],
    "instruction": "it means like its like everywhere but like at the same time",
    "model_name": "gpt-4o",
    "model_names": [
        "gpt-4o-mini",
        "https://jvo0ibb76593l20m.us-east-1.aws.endpoints.huggingface.cloud"
    ],
    "prompt": null,
    "prompt_id": null,
    "ratings": null,
    "rationales": null,
    "rationales-for-ratings": null,
    "rejected": null,
    "rejected_model": null,
    "rejected_rating": null,
    "types": null
}
```

This subset can be loaded as:

```python
from datasets import load_dataset

ds = load_dataset("ashercn97/example-preference-dataset2_work2", "default")
```

Or simply as it follows, since there's only one configuration and is named `default`: 

```python
from datasets import load_dataset

ds = load_dataset("ashercn97/example-preference-dataset2_work2")
```


</details>





## References


```
@misc{cui2024ultrafeedbackboostinglanguagemodels,
    title={UltraFeedback: Boosting Language Models with Scaled AI Feedback},
    author={Ganqu Cui and Lifan Yuan and Ning Ding and Guanming Yao and Bingxiang He and Wei Zhu and Yuan Ni and Guotong Xie and Ruobing Xie and Yankai Lin and Zhiyuan Liu and Maosong Sun},
    year={2024},
    eprint={2310.01377},
    archivePrefix={arXiv},
    primaryClass={cs.CL},
    url={https://arxiv.org/abs/2310.01377},
}
```


