license:mit size_categories:1K<n<10K format:parquet modality:text library:datasets library:pandas library:mlcroissant library:polars region:us
---
license: mit
dataset_info:
  features:
  - name: query
    dtype: string
  - name: choices
    sequence: string
  - name: gold
    sequence: int64
  - name: source
    dtype: string
  splits:
  - name: test
    num_bytes: 1578275
    num_examples: 2154
  download_size: 802713
  dataset_size: 1578275
configs:
- config_name: default
  data_files:
  - split: test
    path: data/test-*
---

## 🧙MAGI: A hard subset of MMLU and AGIEval✨

[Click for the long version](https://sampaech.substack.com/p/creating-magi-a-hard-subset-of-mmlu).

LLM Benchmarks are chasing a moving target and fast running out of headroom. They are struggling to effectively separate SOTA models from leaderboard optimisers. Can we salvage these old dinosaurs for scrap and make a better benchmark?

I created two subsets of MMLU + AGIEval:

MAGI-Hard: 3203 questions, 4x more discriminative between top models (as measured by std. dev.) This subset is brutal to 7b models and useful for exposing differences between high ability models. Downside: a reference model (Deepseek-67b) is “burned” and cannot be scored fairly by this subset.

MAGI-IRT: 2154 questions, 2x more discriminative. This subset is more balanced and retains discriminative power for low + mid ability models. It uses Item Response Theory (IRT) to model question difficulty, and can score all models fairly.

You can find the MAGI subsets [here](https://huggingface.co/sam-paech) and use them with [this fork of the Eleuther eval harness](https://github.com/sqrkl/lm-evaluation-harness).

MAGI has been added as a metric on the [EQ-Bench leaderboard](https://eqbench.com/).