size_categories:10K<n<100K format:parquet modality:tabular modality:text library:datasets library:pandas library:mlcroissant library:polars arxiv:2310.01377 region:us
---
dataset_info:
  config_name: main
  features:
  - name: source
    dtype: string
  - name: prompt
    dtype: string
  - name: chosen
    list:
    - name: content
      dtype: string
    - name: role
      dtype: string
  - name: chosen-rating
    dtype: float64
  - name: chosen-model
    dtype: string
  - name: rejected
    list:
    - name: content
      dtype: string
    - name: role
      dtype: string
  - name: rejected-rating
    dtype: float64
  - name: rejected-model
    dtype: string
  splits:
  - name: train
    num_bytes: 274978608
    num_examples: 60917
  download_size: 137967608
  dataset_size: 274978608
configs:
- config_name: main
  data_files:
  - split: train
    path: main/train-*
---
# Dataset Description

`ultrafeedback_binarised_max_min` is a pairwise preference dataset designed for training models that require binary preference labels. It has been derived from the UltraFeedback dataset, which provides high-quality feedback for improving language models. The dataset is useful for tasks involving learning from preferences, such as reinforcement learning from human feedback (RLHF) and preference-based ranking.

## Source

This dataset is based on two existing resources:

- **[UltraFeedback Dataset](https://arxiv.org/abs/2310.01377)**  
  ```bibtex
  @misc{cui2023ultrafeedback,
      title={UltraFeedback: Boosting Language Models with High-quality Feedback}, 
      author={Ganqu Cui and Lifan Yuan and Ning Ding and Guanming Yao and Wei Zhu and Yuan Ni and Guotong Xie and Zhiyuan Liu and Maosong Sun},
      year={2023},
      eprint={2310.01377},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
  }
  ```

- **[A cleaned version of UltraFeedback](https://huggingface.co/datasets/argilla/ultrafeedback-binarized-preferences-cleaned)**  
  ```bibtex
  @misc{notus2023,
        author = {Alvaro Bartolome and Gabriel Martin and Daniel Vila},
        title = {Notus},
        year = {2023},
        publisher = {GitHub},
        journal = {GitHub Repository},
        howpublished = {\url{https://github.com/argilla-io/notus}}
  }
  ```

## Preprocessing

To construct `ultrafeedback_binarised_max_min`, the dataset was processed as follows:

- **Computing Average Ratings**: For each response, the average rating across all available dimensions was computed.
- **Pairwise Labeling**: The response with the highest average rating was labeled as the chosen answer, while the response with the lowest average rating was labeled as the rejected answer.
- **Binarization**: The dataset is structured for use in pairwise preference learning, where each data point consists of a preferred response and a less preferred response.

## Usage

This dataset is ideal for training models that leverage pairwise preference comparisons, such as:

- Reinforcement Learning from Human Feedback (RLHF)
- Preference-based ranking models
- Contrastive learning for response quality estimation
