task_categories:multiple-choice task_categories:question-answering language:zh license:cc-by-nc-sa-4.0 size_categories:1K<n<10K arxiv:2403.06574 region:us ancient Chinese llm evaluation
---
license: cc-by-nc-sa-4.0
task_categories:
- multiple-choice
- question-answering
language:
- zh
tags:
- ancient Chinese
- llm
- evaluation
pretty_name: AC-EVAL
size_categories:
- 1K<n<10K
---

# AC-EVAL: Evaluating Ancient Chinese Language Understanding in Large Language Models

- **Homepage:** [https://github.com/yuting-wei/AC-EVAL](https://github.com/yuting-wei/AC-EVAL)
- **Repository:** [https://huggingface.co/datasets/yuting-wei/aceval](https://huggingface.co/datasets/yuting-wei/aceval)
- **Paper:** [AC-EVAL: Evaluating Ancient Chinese Language Understanding in Large Language Models](http://arxiv.org/abs/2403.06574).

## Introduction

AC-EVAL presents a thorough evaluation suite for Large Language Models (LLMs) focusing on ancient Chinese, covering eras from the Pre-Qin period to the Qing dynasty. This suite includes 3245 multi-choice questions across 3 levels of difficulty and 13 diverse tasks, as shown below. Please check our [paper](http://arxiv.org/abs/2403.06574) for more details.

## Leaderboard

Latest leaderboard is in our [github](https://github.com/yuting-wei/AC-EVAL).

## Data 

Each subject consists of two splits: dev and test.  The dev set per subject consists of five exemplars with explanations for few-shot evaluation. The test set is for model evaluation. Labels on the test split are not released, users are required to submit their results to automatically obtain test accuracy. [How to submit?](https://github.com/yuting-wei/AC-EVAL/tree/main#how-to-submit)

Below is a dev example from art and cultural heritage:
| Question | A | B | C | D | Answer | Explanation |
|----------|---|---|---|---|--------|-------------|
| 五代南唐时期著名画家顾闳中的绘画名作是？(The famous painting masterpiece of Gu Hongzhong, a famous painter in the Southern Tang Dynasty during the Five Dynasties, is?) |《女史箴图》(Admonitions of the Instructress to the Court Ladies) | 《五牛图》(Five Buffaloes) | 《簪花仕女图》(Ladies with Flowers) | 《韩熙载夜宴图》(Han Xizai Giving a Night Banquet) | D | 让我们逐步分析。顾闳中的绘画名作是《韩熙载夜宴图》。《五牛图》是韩滉的作品，《簪花仕女图》是周昉的作品，《女史箴图》是顾恺之的作品。 (Let‘s analyze step by step. The famous painting by Gu Hongzhong is 'Han Xizai Giving a Night Banquet.' 'Five Buffaloes' is a work by Han Huang, 'Ladies with Flowers' is by Zhou Fang, and 'Admonitions of the Instructress to the Court Ladies' is by Gu Kaizhi.) |

#### Load data

```python
from datasets import load_dataset
aceval=load_dataset(r"yuting-wei/aceval", 'art_and_cultural_heritage')
print(aceval['test'][0])
```
#### Load all data at once
```python
task_list = ['historical_facts', 'geography', 'social_customs', 'art_and_cultural_heritage', 'philosophy_and_religion', 'lexical_pragmatics_analysis', 'allusions_and_idioms', 'word_sense_disambiguation', 'translation', 'event_extraction', 'sentence_pauses', 'summarization_and_analysis', 'poetry_appreciation']
from datasets import load_dataset
aceval = {k: load_dataset(r"yuting-wei/aceval", k) for k in task_list}
```

## Citation

Please cite our paper if you use our dataset.
```
@misc{wei2024aceval,
      title={AC-EVAL: Evaluating Ancient Chinese Language Understanding in Large Language Models}, 
      author={Yuting Wei and Yuanxing Xu and Xinru Wei and Simin Yang and Yangfu Zhu and Yuqing Li and Di Liu and Bin Wu},
      year={2024},
      eprint={2403.06574},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

```

## Licenses

The AC-EVAL dataset is licensed under a [Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License](http://creativecommons.org/licenses/by-nc-sa/4.0/).
