size_categories:10K<n<100K format:parquet modality:tabular modality:text library:datasets library:pandas library:mlcroissant library:polars arxiv:2411.04424 arxiv:2105.08920 arxiv:1805.04833 region:us
---
dataset_info:
  features:
  - name: index
    dtype: int64
  - name: task
    dtype: string
  - name: worker
    dtype: string
  - name: human_label
    dtype: int64
  - name: llm_label
    dtype: int64
  - name: generator_1
    dtype: string
  - name: generator_2
    dtype: string
  - name: premise
    dtype: string
  splits:
  - name: train
    num_bytes: 2380840
    num_examples: 12000
  download_size: 123678
  dataset_size: 2380840
configs:
- config_name: default
  data_files:
  - split: train
    path: data/train-*
---
# OpenMEVA-MANS-LLMEval Dataset

## Overview

The original **OpenMEVA-MANS** dataset (Guan et al., 2021) contains 1,000 stories generated by 5 different text generation models based on 200 prompts from the **WritingPrompts** dataset (Fan et al., 2018). Each story is rated for overall quality by five human evaluators on a 5-point Likert scale.

This **OpenMEVA-MANS-LLMEval** dataset builds upon this framework by adding LLM-based evaluations on pairs of stories generated by different text generators for the same writing prompt. This dataset accompanies the paper [**Gao et al. (2024). _Bayesian Calibration of Win Rate Estimation with LLM Evaluators_**](https://arxiv.org/abs/2411.04424). Please cite this paper if you use this dataset in your work.

For more details on the original **OpenMEVA-MANS** dataset, please refer to this [paper](https://arxiv.org/abs/2105.08920). The original **OpenMEVA-MANS** dataset is available on [GitHub](https://github.com/thu-coai/OpenMEVA).

## Dataset Columns

- **task**: A unique identifier for each comparison task. Each task corresponds to a unique combination of premise, generator_1, and generator_2. Task labels are in the format "t_{task_id}". Tasks with the same premise, generator_1, and generator_2 will share the same task ID. Task IDs start from 0.
  
- **worker**: Identifies the evaluator mode used to assess the comparison task. The format is "w_{model_name}-{prompting_strategy}".
  
- **human_label**: 
  - `0`: Generator_1 is considered to produce a better story than Generator_2 by human evaluators.
  - `1`: Generator_2 is considered to produce a better story than Generator_1 by human evaluators.
  The label is determined by summing the scores from all human evaluators involved.
  
- **llm_label**:
  - `0`: Generator_1 is considered to produce a better story than Generator_2 by the LLM evaluator (worker).
  - `1`: Generator_2 is considered to produce a better story than Generator_1 by the LLM evaluator (worker).

- **generator_1**: The first text generator for comparison.

- **generator_2**: The second text generator for comparison.
  
- **premise**: The writing prompt based on which the text generators are asked to generate the stories.

## Dataset Extensions

The original **OpenMEVA-MANS** dataset includes 5 text generators across 200 story prompts, resulting in 10 distinct generator pairs for comparison. This leads to a total of 200 * 10 = 2,000 unique comparison tasks.

In the **OpenMEVA-MANS-LLMEval** dataset, we extend this by including evaluations from two LLMs: **GPT-3.5-turbo 0125** and **Gemini-1.0-Pro**, each using three distinct prompting strategies: **Score-only**, **Rate-explain**, and **Analyze-rate**. Therefore, there are 6 evaluator modes in total, resulting in 2,000 * 6 = 12,000 rows in the dataset.

Each comparison task is evaluated twice per evaluator mode, with the order of the stories switched in each trial. The scores from both evaluations trials are then summed to determine the final score for each story. The story with the higher final score is deemed the "winner" of the comparison. If the two stories happen to have the same final score, the winner is selected randomly.

## Usage

You can access and use this dataset for tasks such as:

- Evaluating the performance of different text generation models.
- Investigating LLM-based story evaluation and ranking.
- Exploring model biases and tendencies across various evaluation criteria.

## Citation

- Gao et al. (2024). [*Bayesian Calibration of Win Rate Estimation with LLM Evaluators*](https://arxiv.org/abs/2411.04424)
- Guan et al. (2021). [*OpenMEVA: A Benchmark for Evaluating Open-ended Story Generation*](https://arxiv.org/abs/2105.08920)
- Fan et al. (2018). [*Hierarchical Neural Story Generation*.](https://arxiv.org/abs/1805.04833)
