size_categories:1K<n<10K format:parquet modality:text library:datasets library:pandas library:mlcroissant library:polars region:us
---
dataset_info:
  features:
  - name: prompt
    dtype: string
  - name: category
    dtype: string
  splits:
  - name: cs_subfields
    num_bytes: 261751
    num_examples: 2500
  - name: major_subjects
    num_bytes: 416570
    num_examples: 4000
  download_size: 208357
  dataset_size: 678321
configs:
- config_name: default
  data_files:
  - split: cs_subfields
    path: data/cs_subfields-*
  - split: major_subjects
    path: data/major_subjects-*
---
### Dataset construction:

Our benchmark consists of two main components. The first part includes 4,000 prompts across eight major subject categories on arXiv: Math, Computer Science (CS), Quantitative Biology (QB), Physics, Quantitative Finance (QF), Statistics, Electrical Engineering and Systems Science (EESS), and Economics.

The second part of arXivBench includes 2,500 prompts from five subfields within computer science, one of the most popular fields among all the categories: Artificial Intelligence (AI), Cryptography and Security (C\&S), Data Structures and Algorithms (DS\&AL), Operating Systems (OS), and Programming Languages (PL).

## Plot of Mistral and Gemma model performance:

<img src="radar_charts.jpg" alt="Image 2" style="width: 100%;"/>

## Overall Workflow

Below is the overall workflow of our pipeline.
<img src="process_fig.jpg" alt="Image 1" style="width: 100%;"/>
