size_categories:n<1K format:parquet modality:tabular modality:text library:datasets library:pandas library:mlcroissant library:polars library:distilabel arxiv:2310.01377 region:us synthetic distilabel rlaif
---
size_categories: n<1K
dataset_info:
  features:
  - name: instruction
    dtype: string
  - name: generations
    sequence: string
  - name: model_names
    sequence: string
  - name: distilabel_metadata
    struct:
    - name: raw_input_ultra_feedback_0
      list:
      - name: content
        dtype: string
      - name: role
        dtype: string
    - name: raw_output_ultra_feedback_0
      dtype: string
  - name: ratings
    sequence: int64
  - name: rationales
    sequence: string
  - name: model_name
    dtype: string
  - name: prompt
    dtype: string
  - name: prompt_id
    dtype: string
  - name: chosen
    list:
    - name: content
      dtype: string
    - name: role
      dtype: string
  - name: chosen_rating
    dtype: int64
  - name: rejected
    list:
    - name: content
      dtype: string
    - name: role
      dtype: string
  - name: rejected_rating
    dtype: int64
  splits:
  - name: train
    num_bytes: 7401
    num_examples: 2
  download_size: 26008
  dataset_size: 7401
configs:
- config_name: default
  data_files:
  - split: train
    path: data/train-*
tags:
- synthetic
- distilabel
- rlaif
---

<p align="left">
  <a href="https://github.com/argilla-io/distilabel">
    <img src="https://raw.githubusercontent.com/argilla-io/distilabel/main/docs/assets/distilabel-badge-light.png" alt="Built with Distilabel" width="200" height="32"/>
  </a>
</p>

# Dataset Card for example-preference-dataset2_work6

This dataset has been created with [distilabel](https://distilabel.argilla.io/).



## Dataset Summary

This dataset contains a `pipeline.yaml` which can be used to reproduce the pipeline that generated it in distilabel using the `distilabel` CLI:

```console
distilabel pipeline run --config "https://huggingface.co/datasets/ashercn97/example-preference-dataset2_work6/raw/main/pipeline.yaml"
```

or explore the configuration:

```console
distilabel pipeline info --config "https://huggingface.co/datasets/ashercn97/example-preference-dataset2_work6/raw/main/pipeline.yaml"
```

## Dataset structure

The examples have the following structure per configuration:


<details><summary> Configuration: default </summary><hr>

```json
{
    "chosen": [
        {
            "content": "it means like its like everywhere but like at the same time",
            "role": "user"
        },
        {
            "content": "[\u0027ubiquitous\u0027]",
            "role": "assistant"
        }
    ],
    "chosen_rating": 5,
    "distilabel_metadata": {
        "raw_input_ultra_feedback_0": [
            {
                "content": "Your role is to evaluate text quality based on given criteria.\nYou\u0027ll receive an instructional description (\"Instruction\") and 2 text outputs (\"Text\").\nUnderstand and interpret instructions to evaluate effectively.\nProvide annotations for each text with a rating and rationale.\nThe 2 texts given are independent, and should be evaluated separately.\n",
                "role": "system"
            },
            {
                "content": "# General Text Quality Assessment\n\nEvaluate the model\u0027s outputs based on various criteria:\n\n1. **Correctness \u0026 Informativeness**: Does the output provide accurate and helpful information?\n2. **Honesty \u0026 Uncertainty**: How confidently does the model convey its information, and does it express uncertainty appropriately?\n3. **Truthfulness \u0026 Hallucination**: Does the model introduce misleading or fabricated details?\n4. **Instruction Following**: Does the model\u0027s output align with given instructions and the user\u0027s intent?\n\nYour role is to provide a holistic assessment considering all the above factors.\n\n**Scoring**: Rate outputs 1 to 5 based on the overall quality, considering all aspects:\n1. **Low Quality**: Contains inaccuracies, may be entirely wrong or has severe hallucinations.\n2. **Moderate Quality**: Addresses some aspects, but has errors or is partially aligned with instructions.\n3. **Good**: Generally accurate but may contain minor errors or slight deviations.\n4. **Very Good**: Near perfect, with minor issues in terms of alignment or confidence.\n5, **Excellent**: Accurate, confident, aligned with instructions, and free of hallucinations.\n\n## Format:\n\n### Input\nInstruction: [Clearly specify the task goal and restrictions]\n\nTexts:\n\u003ctext 1\u003e [Text 1]\n\u003ctext 2\u003e [Text 2]\n\n### Output\n#### Output for Text 1\nRating: [Rating for text 1]\nRationale: [Rationale for the rating in short sentences]\n\n#### Output for Text 2\nRating: [Rating]\nRationale: [Rationale]\n\n---\n\n## Annotation\n\n### Input\nInstruction: it means like its like everywhere but like at the same time\n\nTexts:\n\u003ctext 1\u003e [\u0027ubiquitous\u0027]\n\u003ctext 2\u003e [\u0027 omnipresent\u0027]\n\n### Output\n",
                "role": "user"
            }
        ],
        "raw_output_ultra_feedback_0": "#### Output for Text 1\nRating: 5  \nRationale: The term \u0027ubiquitous\u0027 accurately reflects the meaning of being everywhere, aligning perfectly with the instruction. The answer is concise and conveys the intended definition without any uncertainty or hallucination.\n\n#### Output for Text 2\nRating: 5  \nRationale: The word \u0027omnipresent\u0027 is a synonym for ubiquitous and fits the instruction well, meaning present everywhere at the same time. The response is accurate and reflects the user\u0027s intent without any misleading details."
    },
    "generations": [
        "[\u0027ubiquitous\u0027]",
        "[\u0027 omnipresent\u0027]"
    ],
    "instruction": "it means like its like everywhere but like at the same time",
    "model_name": "gpt-4o-mini",
    "model_names": [
        "https://jvo0ibb76593l20m.us-east-1.aws.endpoints.huggingface.cloud",
        "https://jvo0ibb76593l20m.us-east-1.aws.endpoints.huggingface.cloud"
    ],
    "prompt": "it means like its like everywhere but like at the same time",
    "prompt_id": "bb6a2f897a05d2149f0198baadc0e8b809eb3650d9ffec753972cec7cdc483f3",
    "ratings": [
        5,
        5
    ],
    "rationales": [
        "The term \u0027ubiquitous\u0027 accurately reflects the meaning of being everywhere, aligning perfectly with the instruction. The answer is concise and conveys the intended definition without any uncertainty or hallucination.",
        "The word \u0027omnipresent\u0027 is a synonym for ubiquitous and fits the instruction well, meaning present everywhere at the same time. The response is accurate and reflects the user\u0027s intent without any misleading details."
    ],
    "rejected": [
        {
            "content": "it means like its like everywhere but like at the same time",
            "role": "user"
        },
        {
            "content": "[\u0027ubiquitous\u0027]",
            "role": "assistant"
        }
    ],
    "rejected_rating": 5
}
```

This subset can be loaded as:

```python
from datasets import load_dataset

ds = load_dataset("ashercn97/example-preference-dataset2_work6", "default")
```

Or simply as it follows, since there's only one configuration and is named `default`: 

```python
from datasets import load_dataset

ds = load_dataset("ashercn97/example-preference-dataset2_work6")
```


</details>





## References


```
@misc{cui2024ultrafeedbackboostinglanguagemodels,
    title={UltraFeedback: Boosting Language Models with Scaled AI Feedback},
    author={Ganqu Cui and Lifan Yuan and Ning Ding and Guanming Yao and Bingxiang He and Wei Zhu and Yuan Ni and Guotong Xie and Ruobing Xie and Yankai Lin and Zhiyuan Liu and Maosong Sun},
    year={2024},
    eprint={2310.01377},
    archivePrefix={arXiv},
    primaryClass={cs.CL},
    url={https://arxiv.org/abs/2310.01377},
}
```


