size_categories:n<1K format:parquet modality:text library:datasets library:pandas library:mlcroissant library:polars library:distilabel arxiv:2310.01377 region:us synthetic distilabel rlaif
---
size_categories: n<1K
dataset_info:
  features:
  - name: instruction
    dtype: string
  - name: topic
    dtype: string
  - name: generations
    sequence: string
  - name: model_names
    sequence: string
  - name: distilabel_metadata
    struct:
    - name: raw_input_ultra_feedback_0
      list:
      - name: content
        dtype: string
      - name: role
        dtype: string
    - name: raw_output_ultra_feedback_0
      dtype: string
  - name: ratings
    sequence: int64
  - name: rationales
    sequence: string
  - name: model_name
    dtype: string
  - name: prompt
    dtype: 'null'
  - name: prompt_id
    dtype: 'null'
  - name: chosen
    dtype: 'null'
  - name: chosen_rating
    dtype: 'null'
  - name: rejected
    dtype: 'null'
  - name: rejected_rating
    dtype: 'null'
  - name: chosen_model
    dtype: 'null'
  - name: rejected_model
    dtype: 'null'
  splits:
  - name: train
    num_bytes: 131123
    num_examples: 20
  download_size: 84762
  dataset_size: 131123
configs:
- config_name: default
  data_files:
  - split: train
    path: data/train-*
tags:
- synthetic
- distilabel
- rlaif
---

<p align="left">
  <a href="https://github.com/argilla-io/distilabel">
    <img src="https://raw.githubusercontent.com/argilla-io/distilabel/main/docs/assets/distilabel-badge-light.png" alt="Built with Distilabel" width="200" height="32"/>
  </a>
</p>

# Dataset Card for example-preference-dataset2

This dataset has been created with [distilabel](https://distilabel.argilla.io/).



## Dataset Summary

This dataset contains a `pipeline.yaml` which can be used to reproduce the pipeline that generated it in distilabel using the `distilabel` CLI:

```console
distilabel pipeline run --config "https://huggingface.co/datasets/ashercn97/example-preference-dataset2/raw/main/pipeline.yaml"
```

or explore the configuration:

```console
distilabel pipeline info --config "https://huggingface.co/datasets/ashercn97/example-preference-dataset2/raw/main/pipeline.yaml"
```

## Dataset structure

The examples have the following structure per configuration:


<details><summary> Configuration: default </summary><hr>

```json
{
    "chosen": null,
    "chosen_model": null,
    "chosen_rating": null,
    "distilabel_metadata": {
        "raw_input_ultra_feedback_0": [
            {
                "content": "Your role is to evaluate text quality based on given criteria.\nYou\u0027ll receive an instructional description (\"Instruction\") and 2 text outputs (\"Text\").\nUnderstand and interpret instructions to evaluate effectively.\nProvide annotations for each text with a rating and rationale.\nThe 2 texts given are independent, and should be evaluated separately.\n",
                "role": "system"
            },
            {
                "content": "# General Text Quality Assessment\n\nEvaluate the model\u0027s outputs based on various criteria:\n\n1. **Correctness \u0026 Informativeness**: Does the output provide accurate and helpful information?\n2. **Honesty \u0026 Uncertainty**: How confidently does the model convey its information, and does it express uncertainty appropriately?\n3. **Truthfulness \u0026 Hallucination**: Does the model introduce misleading or fabricated details?\n4. **Instruction Following**: Does the model\u0027s output align with given instructions and the user\u0027s intent?\n\nYour role is to provide a holistic assessment considering all the above factors.\n\n**Scoring**: Rate outputs 1 to 5 based on the overall quality, considering all aspects:\n1. **Low Quality**: Contains inaccuracies, may be entirely wrong or has severe hallucinations.\n2. **Moderate Quality**: Addresses some aspects, but has errors or is partially aligned with instructions.\n3. **Good**: Generally accurate but may contain minor errors or slight deviations.\n4. **Very Good**: Near perfect, with minor issues in terms of alignment or confidence.\n5, **Excellent**: Accurate, confident, aligned with instructions, and free of hallucinations.\n\n## Format:\n\n### Input\nInstruction: [Clearly specify the task goal and restrictions]\n\nTexts:\n\u003ctext 1\u003e [Text 1]\n\u003ctext 2\u003e [Text 2]\n\n### Output\n#### Output for Text 1\nRating: [Rating for text 1]\nRationale: [Rationale for the rating in short sentences]\n\n#### Output for Text 2\nRating: [Rating]\nRationale: [Rationale]\n\n---\n\n## Annotation\n\n### Input\nInstruction: How can I create an efficient and robust workflow that utilizes advanced automation techniques to extract targeted data, including customer information, from diverse PDF documents and effortlessly integrate it into a designated Google Sheet? Furthermore, I am interested in establishing a comprehensive and seamless system that promptly activates an SMS notification on my mobile device whenever a new PDF document is uploaded to the Google Sheet, ensuring real-time updates and enhanced accessibility.\n\nTexts:\n\u003ctext 1\u003e Creating an efficient and robust workflow that automates the extraction of targeted data from diverse PDF documents, integrates this data into Google Sheets, and sends SMS notifications when new PDFs are uploaded involves a few steps and the use of multiple tools. Below is a detailed outline of how you can set this up:\n\n### Step-by-Step Workflow\n\n1. **Setup Your Environment**:\n   - Google Account (for Google Sheets and Google Drive).\n   - Access to a cloud service (like Google Drive) to upload your PDFs.\n   - SMS service (like Twilio) to send notifications to your mobile device.\n   - Automation tool (like Zapier,\n\u003ctext 2\u003e To create an efficient and robust workflow for extracting data from diverse PDF documents, integrating it into Google Sheets, and setting up an SMS notification system, follow these steps:\n\n### 1. Extract Data from PDFs:\n\n**Tools Required:**\n- **Python:** Use libraries like PyPDF2, PDFMiner, or Tabula to extract text and tables from PDFs.\n- **OCR Tools:** For scanned PDFs, use OCR libraries like Tesseract.\n\n**Steps:**\n1. **Set Up Python Environment:** Install necessary libraries using `pip install PyPDF2 PDFMiner Tabula-py pytesseract`.\n2. **Create Script for Data\n\n### Output\n",
                "role": "user"
            }
        ],
        "raw_output_ultra_feedback_0": "#### Output for Text 1\nRating: 3\nRationale: Text 1 provides a general outline for creating a workflow as per the instruction but lacks detailed effectiveness. It mentions tools like Google Drive and Twilio, aligning partially with the instruction. However, it is cut off and doesn\u0027t provide clear details about integrating data into Google Sheets or setting up SMS notifications, leaving the explanation incomplete and less informative.\n\n#### Output for Text 2\nRating: 4\nRationale: Text 2 offers a more structured and detailed approach. It specifies using Python libraries for data extraction, aligning well with the task. The text conveys information accurately"
    },
    "generations": [
        "Creating an efficient and robust workflow that automates the extraction of targeted data from diverse PDF documents, integrates this data into Google Sheets, and sends SMS notifications when new PDFs are uploaded involves a few steps and the use of multiple tools. Below is a detailed outline of how you can set this up:\n\n### Step-by-Step Workflow\n\n1. **Setup Your Environment**:\n   - Google Account (for Google Sheets and Google Drive).\n   - Access to a cloud service (like Google Drive) to upload your PDFs.\n   - SMS service (like Twilio) to send notifications to your mobile device.\n   - Automation tool (like Zapier,",
        "To create an efficient and robust workflow for extracting data from diverse PDF documents, integrating it into Google Sheets, and setting up an SMS notification system, follow these steps:\n\n### 1. Extract Data from PDFs:\n\n**Tools Required:**\n- **Python:** Use libraries like PyPDF2, PDFMiner, or Tabula to extract text and tables from PDFs.\n- **OCR Tools:** For scanned PDFs, use OCR libraries like Tesseract.\n\n**Steps:**\n1. **Set Up Python Environment:** Install necessary libraries using `pip install PyPDF2 PDFMiner Tabula-py pytesseract`.\n2. **Create Script for Data"
    ],
    "instruction": "How can I create an efficient and robust workflow that utilizes advanced automation techniques to extract targeted data, including customer information, from diverse PDF documents and effortlessly integrate it into a designated Google Sheet? Furthermore, I am interested in establishing a comprehensive and seamless system that promptly activates an SMS notification on my mobile device whenever a new PDF document is uploaded to the Google Sheet, ensuring real-time updates and enhanced accessibility.",
    "model_name": "gpt-4o",
    "model_names": [
        "gpt-4o-mini",
        "gpt-4o"
    ],
    "prompt": null,
    "prompt_id": null,
    "ratings": [
        3,
        4
    ],
    "rationales": [
        "Text 1 provides a general outline for creating a workflow as per the instruction but lacks detailed effectiveness. It mentions tools like Google Drive and Twilio, aligning partially with the instruction. However, it is cut off and doesn\u0027t provide clear details about integrating data into Google Sheets or setting up SMS notifications, leaving the explanation incomplete and less informative.",
        "Text 2 offers a more structured and detailed approach. It specifies using Python libraries for data extraction, aligning well with the task. The text conveys information accurately"
    ],
    "rejected": null,
    "rejected_model": null,
    "rejected_rating": null,
    "topic": "Software Development"
}
```

This subset can be loaded as:

```python
from datasets import load_dataset

ds = load_dataset("ashercn97/example-preference-dataset2", "default")
```

Or simply as it follows, since there's only one configuration and is named `default`: 

```python
from datasets import load_dataset

ds = load_dataset("ashercn97/example-preference-dataset2")
```


</details>





## References


```
@misc{cui2024ultrafeedbackboostinglanguagemodels,
    title={UltraFeedback: Boosting Language Models with Scaled AI Feedback},
    author={Ganqu Cui and Lifan Yuan and Ning Ding and Guanming Yao and Bingxiang He and Wei Zhu and Yuan Ni and Guotong Xie and Ruobing Xie and Yankai Lin and Zhiyuan Liu and Maosong Sun},
    year={2024},
    eprint={2310.01377},
    archivePrefix={arXiv},
    primaryClass={cs.CL},
    url={https://arxiv.org/abs/2310.01377},
}
```


