size_categories:1K<n<10K format:parquet modality:tabular modality:text library:datasets library:pandas library:mlcroissant library:polars arxiv:2411.04424 arxiv:2007.12626 arxiv:1506.03340 region:us
---
dataset_info:
  features:
  - name: task
    dtype: string
  - name: worker
    dtype: string
  - name: human_label
    dtype: int64
  - name: llm_label
    dtype: int64
  - name: generator_1
    dtype: string
  - name: generator_2
    dtype: string
  - name: premise
    dtype: string
  - name: __index_level_0__
    dtype: float64
  splits:
  - name: train
    num_bytes: 19773135
    num_examples: 9000
  download_size: 329063
  dataset_size: 19773135
configs:
- config_name: default
  data_files:
  - split: train
    path: data/train-*
---
# SummEval-LLMEval Dataset

## Overview

The original **SummEval** dataset (Fabbri et al., 2021) consists of 1,600 summaries annotated by human expert evaluators using a 5-point Likert scale across 4 criteria: **coherence**, **consistency**, **fluency**, and **relevance**. These 1,600 summaries are based on 100 source articles from the **CNN/DailyMail** dataset (Hermann et al., 2015). For each source article, **SummEval** collects 16 summaries generated by 16 different automatic summarization systems. Each summary is evaluated by three human expert annotators.

The **SummEval-LLMEval** dataset extends this framework by incorporating evaluations from large language models (LLMs) on pairs of summaries generated by different text generation models for the same source article. This dataset accompanies the paper [**Gao et al. (2024). _Bayesian Calibration of Win Rate Estimation with LLM Evaluators_**](https://arxiv.org/abs/2411.04424). Please cite this paper if you use this dataset in your work.

For more information about the original **SummEval** dataset, please refer to the [SummEval paper](https://arxiv.org/abs/2007.12626). The original **SummEval** dataset is available on [GitHub](https://github.com/Yale-LILY/SummEval).

## Dataset Columns

- **task**: A unique identifier for each comparison task. Each task corresponds to a unique combination of premise, generator_1, and generator_2. Task labels are in the format "t_{task_id}". Tasks with the same premise, generator_1, and generator_2 will share the same task ID. Task IDs start from 0.
  
- **worker**: Identifies the evaluator mode used to assess the comparison task. The format is "w_{model_name}-{prompting_strategy}".
  
- **human_label**: 
  - `0`: Generator_1 is considered to produce a better summary than Generator_2 by human evaluators.
  - `1`: Generator_2 is considered to produce a better summary than Generator_1 by human evaluators.
  The label is determined by summing the scores from all human evaluators involved.
  
- **llm_label**:
  - `0`: Generator_1 is considered to produce a better summary than Generator_2 by the LLM evaluator (worker).
  - `1`: Generator_2 is considered to produce a better summary than Generator_1 by the LLM evaluator (worker).

- **generator_1**: The first text generator for comparison.

- **generator_2**: The second text generator for comparison.
  
- **premise**: The source article based on which the text generators are asked to generate the summaries.

- **__index_level_0__**: A column that is not useful and should be disregarded.

## Dataset Extensions

The original **SummEval** dataset includes 16 text generators across 100 source articles, resulting in 120 distinct generator pairs for comparison.

In the **SummEval-LLMEval** dataset, to reduce computational overhead, we evaluate only those comparison tasks where the first text generator is **GPT-2**. This results in a total of 100 * 15 = 1,500 unique comparison tasks. We include evaluations from two LLMs: **GPT-3.5-turbo 0125** and **Gemini-1.0-Pro**, each using three distinct prompting strategies: **Score-only**, **Rate-explain**, and **Analyze-rate**. Therefore, there are 6 evaluator modes in total, resulting in 1,500 * 6 = 9,000 rows in the dataset.

Each comparison task is evaluated twice per evaluator mode, with the order of the summaries switched in each trial. The scores from both evaluations are then summed across the four evaluation criteria (coherence, consistency, fluency, and relevance) to determine the final score for each summary. The summary with the higher final score is considered the "winner" of the comparison.


## Citation

- Gao et al. (2024). [*Bayesian Calibration of Win Rate Estimation with LLM Evaluators*.](https://arxiv.org/abs/2411.04424)
- Fabbri et al. (2021). [*SummEval: Re-evaluating Summarization Evaluation*.](https://arxiv.org/abs/2007.12626)
- Hermann et al. (2015). [*Teaching Machines to Read and Comprehend.*](https://arxiv.org/abs/1506.03340)