size_categories:1K<n<10K format:parquet modality:tabular modality:text library:datasets library:pandas library:mlcroissant library:polars arxiv:2308.01862 arxiv:2310.07641 arxiv:2411.04424 region:us
---
dataset_info:
  features:
  - name: task
    dtype: string
  - name: worker
    dtype: string
  - name: human_label
    dtype: int64
  - name: llm_label
    dtype: int64
  - name: generator_1
    dtype: string
  - name: generator_2
    dtype: string
  - name: instruction
    dtype: string
  - name: output_1
    dtype: string
  - name: output_2
    dtype: string
  - name: sub_dataset
    dtype: string
  - name: swap_equal
    dtype: bool
  splits:
  - name: train
    num_bytes: 2400464
    num_examples: 1600
  download_size: 193743
  dataset_size: 2400464
configs:
- config_name: default
  data_files:
  - split: train
    path: data/train-*
---
# LLMEval²-Select Dataset

## Introduction

The **LLMEval²-Select** dataset is a curated subset of the original **LLMEval²** dataset introduced by [Zhang et al. (2023)](https://arxiv.org/abs/2308.01862). The original LLMEval² dataset comprises **2,553 question-answering instances**, each annotated with human preferences. Each instance consists of a question paired with two answers.

To construct LLMEval²-Select, [Zeng et al. (2024)](https://arxiv.org/abs/2310.07641) followed these steps:
1. Labelled each instance with the human-preferred answer.
2. Removed all instances with "ties."
3. Randomly sampled **200 instances** from the refined dataset. Each instance is defined as a unique `(instruction, output_1, output_2)` tuple.

### Evaluation Overview

The LLMEval²-Select dataset has been evaluated using **LLM evaluators** by the authors of the [LLMBar paper](https://arxiv.org/abs/2310.07641). The evaluation setup:
- **LLMs**: GPT-4-0613, GPT-3.5-turbo-0613 (ChatGPT), Llama-2-70B-Chat, and PaLM 2.
- **Prompting Strategies**: Vanilla and Rules+Metrics+Reference.

Each unique combination of an LLM and a prompting strategy is referred to as an **evaluator mode**. For detailed descriptions of these prompting strategies, refer to the LLMBar paper.

### Dataset Characteristics

- For each evaluator mode, the **evaluation accuracy** (i.e., the proportion of tasks where the LLM evaluator's choice aligns with the human-picked better output) is above **59%**.
- The dataset includes **8 evaluator modes** (4 LLMs × 2 prompting strategies) and **200 comparison tasks**, resulting in **1,600 rows** (200 × 8).

This dataset accompanies the paper [**Gao et al. (2024). _Bayesian Calibration of Win Rate Estimation with LLM Evaluators_**](https://arxiv.org/abs/2411.04424). If you use this dataset, please cite:
- Gao et al. (2024),
- Zhang et al. (2023),
- and Zeng et al. (2024).

For details on the original LLMEval² dataset, refer to the [LLMEval² paper](https://arxiv.org/abs/2308.01862). The curated LLMEval² subset prepared by the LLMBar authors is available on [GitHub](https://github.com/princeton-nlp/LLMBar).

## Dataset Details

### Columns

- **task**: Unique identifier for each comparison task. Format: `t_{task ID}`. Tasks with identical instructions and outputs share the same task ID (starting from 0).
- **worker**: The evaluator mode, formatted as `w_{model name}@{prompting strategy}`.
- **human_label**: Always `0`, indicating `output_1` is the human-picked better response.
- **llm_label**:
  - `0`: The LLM evaluator considers `output_1` better.
  - `1`: The LLM evaluator considers `output_2` better.
- **generator_1**: Always `"correct"`, signifying `output_1` is the superior response by human evaluator.
- **generator_2**: Always `"incorrect"`, signifying `output_2` is the inferior response by human evaluator.
- **instruction**: The question or instruction for the task.
- **output_1**: The first answer to the instruction.
- **output_2**: The second answer to the instruction.
- **sub_dataset**: Always labeled as `"LLMEval^2"`.
- **swap_equal**: Boolean field indicating whether the LLM evaluator gives the same judgment when the output order is swapped. If `false`, the `llm_label` is randomly assigned from {0, 1}.

## Usage

This dataset is suitable for various tasks, including:

- **LLM Evaluation Analysis**: Measuring LLM performance with different prompting strategies in instruction-following tasks.
- **LLM Bias Analysis**: Exploring model biases and tendencies in LLM-based evaluations.

## Citation

If you use the LLMEval²-Select dataset, please cite:

- **Gao et al. (2024)**:  
  [*Bayesian Calibration of Win Rate Estimation with LLM Evaluators*](https://arxiv.org/abs/2411.04424).

- **Zeng et al. (2024)**:  
  [*Evaluating Large Language Models at Evaluating Instruction Following*](https://arxiv.org/abs/2310.07641).

- **Zhang et al. (2023)**:  
  [*Wider and Deeper LLM Networks are Fairer LLM Evaluators*](https://arxiv.org/abs/2308.01862).
