license:mit region:us
---
license: mit
---

# ModelClash: Dynamic LLM Evaluation Duel

## Dataset Summary

ModelClash is a dynamic benchmark dataset for evaluating and comparing Large Language Models (LLMs) in a competitive environment. It consists of AI-generated challenges and solutions, pitting models against each other in creative problem-solving duels. This adaptive approach to benchmarking evolves with AI advancements, offering a novel way to assess model capabilities.

## Supported Tasks and Leaderboards

The dataset supports the following tasks:
- **Text Generation**: Models create challenges and implement solutions.
- **Code Generation**: Many challenges involve writing code to solve problems.
- **Problem Solving**: Models must understand and solve diverse challenges.

A leaderboard is maintained based on the scoring system described in the dataset.

## Dataset Structure

The dataset is structured as a collection of duels, where each duel contains:
- Challenge description
- Visible test cases
- Hidden test cases
- Solutions attempted by different models
- Scores for each model

### Data Instances

A typical data instance looks like this:

```
{
  "challenge_id": "001",
  "creator_model": "gpt-4o",
  "solver_model": "claude-3-5-sonnet-20240620",
  "challenge_description": "Implement a function that sorts and deduplicates characters in a string.",
  "visible_test_cases": [
    {"input": "banana", "expected_output": "abn"},
    {"input": "apple", "expected_output": "aelp"}
  ],
  "hidden_test_cases": [
    {"input": "racecar", "expected_output": "acer"}
  ],
  "creator_solution": "def X(s): return ''.join(sorted(set(s)))",
  "solver_solution": "def X(s): return ''.join(sorted(set(s.lower())))",
  "creator_score": 1,
  "solver_score": 2
}
```

### Data Fields

- `challenge_id`: Unique identifier for the challenge
- `creator_model`: The model that created the challenge
- `solver_model`: The model attempting to solve the challenge
- `challenge_description`: Description of the task
- `visible_test_cases`: List of test cases visible to the solver
- `hidden_test_cases`: List of test cases hidden from the solver
- `creator_solution`: Solution provided by the challenge creator
- `solver_solution`: Solution attempted by the solver
- `creator_score`: Points earned by the creator for this challenge
- `solver_score`: Points earned by the solver for this challenge

### Data Splits

The dataset is not split into traditional train/validation/test sets. Instead, it's a continuously growing collection of duels between different model pairs.

## Dataset Creation

### Curation Rationale

ModelClash was created to address limitations in traditional benchmarks:
1. Automatic challenge generation
2. Infinite scalability
3. Prevention of memorization and data leakage
4. Comprehensive evaluation of both problem creation and solving
5. Extreme diversity in problem types and domains

### Source Data

The challenges and solutions are generated in real-time by AI models, including:
- Claude-3-5-sonnet-20240620
- GPT-3.5-turbo
- GPT-4o
- GPT-4o-mini

### Annotations

The dataset does not contain human annotations. Scoring is done automatically based on whether solutions pass the test cases.

## Considerations for Using the Data

### Social Impact of Dataset

This dataset aims to drive advancement in AI capabilities by providing a dynamic, evolving benchmark. It may influence the development and evaluation of future AI models.

### Discussion of Biases

The dataset may inherit biases present in the AI models used to generate challenges and solutions. Users should be aware that these biases could influence the types of problems created and the approaches to solving them.

### Other Known Limitations

- The quality and difficulty of challenges can vary significantly.
- The dataset's effectiveness depends on the capabilities of the models used to generate content.

## Additional Information

### Dataset Curators

The dataset was curated by Rasmus Lindahl.

### Licensing Information

The dataset is distributed under the MIT License.

### Citation Information

If you use this dataset in your research, please cite:

```
@misc{modelclash2024,
  author = {Lindahl, Rasmus},
  title = {ModelClash: Dynamic LLM Evaluation Duel},
  year = {2024},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/mrconter1/ModelClash}}
}
```

### Contributions

Contributions to the dataset are welcome. Please refer to the GitHub repository for details on how to contribute.

For more information or to contribute, visit the [ModelClash GitHub repository](https://github.com/mrconter1/ModelClash).
