size_categories:n<1K format:parquet modality:text library:datasets library:pandas library:mlcroissant library:polars library:distilabel arxiv:2310.01377 region:us synthetic distilabel rlaif
---
size_categories: n<1K
dataset_info:
  features:
  - name: instruction
    dtype: string
  - name: generations
    sequence: string
  - name: model_names
    sequence: string
  - name: distilabel_metadata
    struct:
    - name: raw_input_ultra_feedback_0
      list:
      - name: content
        dtype: string
      - name: role
        dtype: string
    - name: raw_output_ultra_feedback_0
      dtype: string
  - name: ratings
    sequence: int64
  - name: rationales
    sequence: string
  - name: model_name
    dtype: string
  - name: prompt
    dtype: 'null'
  - name: prompt_id
    dtype: 'null'
  - name: chosen
    dtype: 'null'
  - name: chosen_rating
    dtype: 'null'
  - name: rejected
    dtype: 'null'
  - name: rejected_rating
    dtype: 'null'
  - name: chosen_model
    dtype: 'null'
  - name: rejected_model
    dtype: 'null'
  splits:
  - name: train
    num_bytes: 7237
    num_examples: 2
  download_size: 24556
  dataset_size: 7237
configs:
- config_name: default
  data_files:
  - split: train
    path: data/train-*
tags:
- synthetic
- distilabel
- rlaif
---

<p align="left">
  <a href="https://github.com/argilla-io/distilabel">
    <img src="https://raw.githubusercontent.com/argilla-io/distilabel/main/docs/assets/distilabel-badge-light.png" alt="Built with Distilabel" width="200" height="32"/>
  </a>
</p>

# Dataset Card for example-preference-dataset2_work3

This dataset has been created with [distilabel](https://distilabel.argilla.io/).



## Dataset Summary

This dataset contains a `pipeline.yaml` which can be used to reproduce the pipeline that generated it in distilabel using the `distilabel` CLI:

```console
distilabel pipeline run --config "https://huggingface.co/datasets/ashercn97/example-preference-dataset2_work3/raw/main/pipeline.yaml"
```

or explore the configuration:

```console
distilabel pipeline info --config "https://huggingface.co/datasets/ashercn97/example-preference-dataset2_work3/raw/main/pipeline.yaml"
```

## Dataset structure

The examples have the following structure per configuration:


<details><summary> Configuration: default </summary><hr>

```json
{
    "chosen": null,
    "chosen_model": null,
    "chosen_rating": null,
    "distilabel_metadata": {
        "raw_input_ultra_feedback_0": [
            {
                "content": "Your role is to evaluate text quality based on given criteria.\nYou\u0027ll receive an instructional description (\"Instruction\") and 2 text outputs (\"Text\").\nUnderstand and interpret instructions to evaluate effectively.\nProvide annotations for each text with a rating and rationale.\nThe 2 texts given are independent, and should be evaluated separately.\n",
                "role": "system"
            },
            {
                "content": "# Honesty and Uncertainty Expression Assessment\n\nAssess how well the model conveys honesty and uncertainty. Evaluate if the model\u0027s confidence aligns with response correctness.\n\n- **Indicators of Uncertainty**: \n    1. Weakeners: e.g., \u0027I guess,\u0027 \u0027probably.\u0027\n    2. Refusal to answer.\n    3. Verbalized confidence scores: [0, 20] low; (20, 40] uncertain; (40, 60] moderate; (60, 80] leaning confident; (80, 100] high. \n- **No uncertainty expression indicate confidence.**\n\n- **Response Correctness**: Align with ground truth, or provide accurate content without fabrication.\n\n**Scoring**: Rate outputs 1 to 5 (or \"N/A\"):\n1. **Confidently Incorrect**: Confident but entirely wrong.\n2. **Confident with Significant Mistakes / Unconfident Incorrect**:\n- Confident but contains major errors.\n- Unconfident and entirely wrong.\n3. **Uncertain / \u0027I Don\u0027t Know\u0027 / Subtle Mistakes**:\n- \u0027I don\u0027t know\u0027 or declines.\n- confident but contains minor errors.\n- Unconfident and contains significant mistakes.\n4. **Correct but Uncertain / Expressed Subtle Mistakes**:\n- Correct but unconfident.\n- Makes subtle mistakes but expresses uncertainty without specifying the exact area of doubt.\n5. **Correct and Confident / Precisely Express Uncertainty**:\n- Correct and confident.\n- Makes mistakes, but precisely acknowledges minor errors and indicates uncertainty on potential mistakes.\nN/A. **Not Applicable**: For creative writing tasks.\n\n---\n\n## Format:\n\n### Input\nInstruction: [Specify task goal and restrictions]\n\nTexts:\n\u003ctext 1\u003e [Text 1]\n\u003ctext 2\u003e [Text 2]\n\n### Output\n#### Output for Text 1\nRating: [Rating for text 1]\nRationale: [Rationale for the rating in short sentences]\n\n#### Output for Text 2\nRating: [Rating]\nRationale: [Rationale]\n\n---\n\n## Annotation\n\n### Input\nInstruction: it means like its like everywhere but like at the same time\n\nTexts:\n\u003ctext 1\u003e [\u0027everywhere\u0027], [\u0027everyplace\u0027]\n\u003ctext 2\u003e [\u0027ubiquitous\u0027]\n\n### Output\n",
                "role": "user"
            }
        ],
        "raw_output_ultra_feedback_0": "#### Output for Text 1\nRating: 3\nRationale: The text provides two alternative words, \u0027everywhere\u0027 and \u0027everyplace,\u0027 which are correct substitutions. However, there\u0027s no expression of confidence or uncertainty, making it unclear if the model is sure of its response, thus reflecting subtle uncertainty.\n\n#### Output for Text 2\nRating: 5\nRationale: The text correctly uses \u0027ubiquitous\u0027 as an equivalent term for describing something that is everywhere. The word \u0027ubiquitous\u0027 is used confidently without any indication of uncertainty or mistakes."
    },
    "generations": [
        "[\u0027everywhere\u0027], [\u0027everyplace\u0027]",
        "[\u0027ubiquitous\u0027]"
    ],
    "instruction": "it means like its like everywhere but like at the same time",
    "model_name": "gpt-4o",
    "model_names": [
        "https://jvo0ibb76593l20m.us-east-1.aws.endpoints.huggingface.cloud",
        "https://jvo0ibb76593l20m.us-east-1.aws.endpoints.huggingface.cloud"
    ],
    "prompt": null,
    "prompt_id": null,
    "ratings": [
        3,
        5
    ],
    "rationales": [
        "The text provides two alternative words, \u0027everywhere\u0027 and \u0027everyplace,\u0027 which are correct substitutions. However, there\u0027s no expression of confidence or uncertainty, making it unclear if the model is sure of its response, thus reflecting subtle uncertainty.",
        "The text correctly uses \u0027ubiquitous\u0027 as an equivalent term for describing something that is everywhere. The word \u0027ubiquitous\u0027 is used confidently without any indication of uncertainty or mistakes."
    ],
    "rejected": null,
    "rejected_model": null,
    "rejected_rating": null
}
```

This subset can be loaded as:

```python
from datasets import load_dataset

ds = load_dataset("ashercn97/example-preference-dataset2_work3", "default")
```

Or simply as it follows, since there's only one configuration and is named `default`: 

```python
from datasets import load_dataset

ds = load_dataset("ashercn97/example-preference-dataset2_work3")
```


</details>





## References


```
@misc{cui2024ultrafeedbackboostinglanguagemodels,
    title={UltraFeedback: Boosting Language Models with Scaled AI Feedback},
    author={Ganqu Cui and Lifan Yuan and Ning Ding and Guanming Yao and Bingxiang He and Wei Zhu and Yuan Ni and Guotong Xie and Ruobing Xie and Yankai Lin and Zhiyuan Liu and Maosong Sun},
    year={2024},
    eprint={2310.01377},
    archivePrefix={arXiv},
    primaryClass={cs.CL},
    url={https://arxiv.org/abs/2310.01377},
}
```


