Rethinking LLM Evaluation with 3C3H: AraGen Benchmark and Leaderboard
alielfilali01, neha1710, Arwa88, preslavnakov, clefourrier
In the rapidly evolving landscape of large language models (LLMs), comprehensive and robust evaluation methodologies remain a critical challenge, particularly for low-resource languages. In this blog, we introduce AraGen, a generative tasks benchmark and leaderboard for Arabic LLMs, based on 3C3H, a new evaluation measure for NLG which we hope will inspire work for other languages as well.

The AraGen leaderboard makes three key contributions:
- **3C3H Measure**: The 3C3H measure scores a model's response and is central to this framework. It is a holistic approach assessing model responses across multiple dimensions -**C**orrectness, **C**ompleteness, **C**onciseness, **H**elpfulness, **H**onesty, and **H**armlessness- based on LLM-as-judge.
  
- **Dynamic Evaluations**: AraGen Leaderboard implements a dynamic evaluation strategy, which includes three-month blind testing cycles, where the datasets and the evaluation code remain private before being publicly released at the end of the cycle, and replaced by a new private benchmark.

- **Arabic Evaluation Dataset**: AraGen Benchmark offers a meticulously constructed evaluation dataset for Arabic LLM evaluation, combining multi-turn and single-turn scenarios, which tests the model capability across multiple domains and tasks.

We believe that AraGen addresses persistent issues of data contamination with its dynamic evaluation approach, preserving the benchmark's integrity. It also serves as the first application of a scalable, language-agnostic framework for a nuanced and fair model assessment, which represents an important effort in understanding LLM performance across diverse linguistic contexts and sets a new standard for comprehensive model benchmarking.

<script type="module" src="https://gradio.s3-us-west-2.amazonaws.com/4.4.0/gradio.js"> </script>
<gradio-app theme_mode="dark" space="inceptionai/AraGen-Leaderboard"></gradio-app>

## Summary

Evaluating large language models (LLMs) is a key challenge in AI research. While existing methodologies have improved our understanding of LLM capabilities, they often fail to comprehensively address both **factuality**—assessing a model's core knowledge—and **usability**—its alignment with human (end user) expectations. Current evaluation approaches can broadly be categorized into knowledge or factuality-based benchmarks and preference-based benchmarks.

**Automatic benchmarks** focus on evaluating foundational knowledge and factual correctness. For instance, initiatives like the [Open LLM Leaderboard](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) by Hugging Face assess the likelihood of the choices for a given prompt (question) and compare the most likely output with a golden reference choice. While effective in testing core knowledge, these benchmarks provide limited insight into how models perform in practical, user-facing contexts, leaving critical aspects of usability unaddressed. 

In contrast, **preference-based benchmarks** aim to capture alignment with the user preferences. Examples include LMSYS's [Chatbot Arena](https://arena.lmsys.org/) and AtlaAI's [Judge Arena](https://huggingface.co/spaces/AtlaAI/judge-arena), which mostly rely on subjective assessments of outputs based on style, tone, and overall utility. However, these approaches risk prioritizing stylistic alignment over factual accuracy, potentially skewing evaluations toward stylistically preferred yet less accurate responses. Additionally, crowdsourced arenas can reflect the biases of their annotators, who may lack strong voting guidelines, further impacting the consistency and reliability of evaluations. 

To address these limitations, we propose a new evaluation measure that aims to **combine both approaches**, offering a comprehensive mechanism to evaluate language models. It assesses two key aspects of model outputs: 

- **Factuality**: The accuracy and the correctness of the model's output, reflecting its core knowledge. 
- **Usability**: The degree to which the model's outputs align with human preferences, ensuring user-centric assessment. 

This is done through the introduction of a new evaluation measure based on LLM-as-a-Judge approach ([see here for more on this approach](https://github.com/huggingface/evaluation-guidebook/blob/main/contents/model-as-a-judge/basics.md)), which evaluates the model performance across six dimensions modeling factuality and usability. By adopting a balanced perspective, we ensure that usability does not come at the expense of factual accuracy or vice versa.

## AraGen: A Generative Benchmark and Leaderboard for Arabic LLMs 

The **AraGen Leaderboard** ranks both open and proprietary models, evaluated on the **AraGen Benchmark** using the new **3C3H** measure, which we introduce below. 3C3H provides a comprehensive framework for assessing both the factual accuracy and usability of large language models. Arabic was chosen as the first application of this framework, aligning with the mission of Inception to democratize AI for Arabic and the Global South in general, while addressing the lack of robust generative benchmarks for these languages and regions, and we hope to see extensions of this work in many other languages.

The leaderboard is dynamic, with evaluation datasets remaining private (blind testing) for three months to ensure fair and unbiased assessments. After this period, the dataset and the corresponding evaluation code will be publicly released, coinciding with the introduction of a new dataset for the next evaluation cycle, which will itself remain private for three months. This iterative process ensures that evaluations stay current and models are consistently tested on fresh, unseen data.

We believe that this dynamic approach is both beneficial and robust, as it mitigates data leakage, encourages ongoing model improvement, and maintains the relevance of the benchmark in the rapidly evolving landscape of LLM development.

## The AraGen Leaderboard

### Evaluation Pipeline

The AraGen evaluation pipeline aims to ensure robust, reproducible, and scalable assessments. The process includes the following steps: 

1. **Model Submission**: Users submit a model for evaluation. 
2. **Response Generation**: We use the model to generate responses for a fixed set of human-verified questions (AraGen Benchmark). 
3. **LLM as a Judge**: A chosen LLM (see Section 2), evaluates the generated answers against pre-verified ground truth answers. The judge's assessment is based on the **3C3H** as guideline and returns the scores in `json` format at the end of its response after its reasoning section. 
4. **Scoring and Normalization**: 
   - Binary scores (Correctness and Completeness) are determined first. Only correct answers are further evaluated for other dimensions. 
   - Scaled scores (e.g., Helpfulness, Honesty), originally scored within [1, 5], are normalized to a range within [0, 1]. 
5. **Leaderboard Reporting**: The results are displayed across two leaderboards: 
   - **3C3H Leaderboard**: Provides an overall score that evaluates all answers holistically based on the six dimensions of the **3C3H** score (**C**orrectness, **C**ompleteness, **C**onciseness, **H**elpfulness, **H**onesty, and **H**armlessness). It further reports the scores for each one of them.
   - **Tasks Leaderboard**: Reports the 3C3H score for the four individual tasks that we focus on: question answering, reasoning, orthographic & grammatical analysis, and safety. 

### 3C3H: Our new evaluation measure for LLMs
 
Our main contribution, the **3C3H** measure, evaluates model performance across six dimensions, using an LLM-judge
 
1. **Correctness (0 or 1)**: Is the answer factually accurate *with respect to* the ground truth? 
2. **Completeness (0 or 1)**: Does the answer address all parts of the question? 
3. **Conciseness (1 to 5)**: Is the answer appropriately brief while retaining all necessary information and details? 
4. **Helpfulness (1 to 5)**: Does the answer effectively assist or inform the user? 
5. **Honesty (1 to 5)**: Is all the information in the answer accurate and free of hallucinations? This measure is similar to the first dimension above (Correctness), but assesses any extra information incidentally contained in the answer for its accuracy on a more detailed scale.  
6. **Harmlessness (1 to 5)**: Is the answer free from offensive or biased content? 

The evaluation process includes the following elements:

1. **System Prompt**: A detailed system prompt defines the evaluation rules and the scoring criteria for the judge LLM. This includes instructions about how to score each dimension and how to generate output in JSON format for structured scoring. 
2. **User Prompt**: The user prompt consists of a question from the dataset paired with its
   - **ground truth answer** (correct answer, human-verified),
   - **model-generated answer** (to be evaluated). 
3. **Single Evaluation**: For each question, the judge evaluates the model's answer once, assigning six scores (one per criterion) in a single evaluation pass. The **zeroing rule** ensures that if the answer is factually incorrect (`Correct = 0`), all other dimensions are scored as `0`. 
4. **Output Format**: The judge provides a detailed explanation for its scores followed by a parsable JSON-formatted result, ensuring clarity. 
 
#### Scoring and Normalization
 
- Binary scores (Correctness and Completeness) are computed first. If a response is **Incorrect (0)**, all other dimensions are automatically set to zero to avoid rewarding flawed outputs. 
 
- Scaled scores (e.g., Conciseness, Helpfulness, ...). The remaining four dimensions are scores ranging from 1 to 5 and later normalized to [0, 1] for consistency. For example, a score of 3 for **Honesty** would be normalized to \\( \frac{3 - 1}{4} = 0.5 \\).
 
#### Calculate the 3C3H Score

Given the individual scores for each dimension, the 3C3H measure is computed as follows:
 
$$
3C3H = \frac{1}{6n} \sum_{i=1}^{n} c_{1i} \left(1 + c_{2i} + \frac{c_{3i} - 1}{4} + \frac{h_{1i} - 1}{4} + \frac{h_{2i} - 1}{4} + \frac{h_{3i} - 1}{4}\right) 
$$
 
Where \\( n \\) is the number of dataset samples, \\( c_{1i} \\) the correctness score of sample \\( i \\), \\( c_{2i} \\) the completeness score of sample \\( i \\), and \\( c_{3i} \\), \\( h_{1i} \\), \\( h_{2i} \\), \\( h_{3i} \\) the Conciseness, Helpfulness, Honesty, and Harmlessness scores respectively of sample \\( i \\).


### Dynamic Leaderboard for Robustness

To ensure a reliable and fair evaluation process, the **AraGen Leaderboard** incorporates a **dynamic** evaluation strategy designed to address data contamination risks while prioritizing transparency, reproducibility, and continuous relevance. This is ensured as follows:
 
1. **Blind Test Sets**:   
   Each test set remains private for a **3-month evaluation period**. During this phase, the test set is used to evaluate submitted models without the risk of data leakage into the training datasets, thus ensuring unbiased results. 
 
2. **Periodic Updates**:   
   After three months, the blind test set is replaced by a new set of **human-verified question-answer pairs**. This ensures that the evaluation remains robust, adaptive, and aligned with evolving model capabilities. The new test sets are designed to maintain consistency in
   - **Structure**: preserving the type and the format of interactions
   - **Complexity**: ensuring at least comparable, or increasing levels of difficulty across batches
   - **Distribution**: balancing the representation of domains, tasks, and scenarios. 
 
3. **Open-Sourcing for Reproducibility**:   
   Following the blind-test evaluation period, the benchmark dataset will be publicly released alongside the code used for evaluation. Which allows
   - **Independent Verification**: Researchers can reproduce results and validate the benchmark's integrity.
   - **Open Source**: Open access fosters discussion and improvements within the research community. 
 
### Dataset Design

The AraGen Benchmark includes 279 custom, mainly human-verified questions designed to rigorously test model capabilities across four diverse tasks: 
 
1. **Question Answering**: Tests factual accuracy and core knowledge regarding different themes related to Arabic and the Arab world. 
2. **Orthographic and Grammatical Analysis**: Assesses Arabic language understanding and grammatical errors detection/correction at a structural level. 
3. **Reasoning**: Challenges models to infer, deduce, and reason logically. 
4. **Safety**: Evaluates the ability to produce responses free from harmful or biased content or avoid obeying harmful requests from users. 

<div style="text-align: center;">
  <img src="https://huggingface.co/spaces/inceptionai/AraGen-Leaderboard/raw/main/assets/pictures/blog_figure_1.png" alt="Percentage Distribution of Tasks" width="500">
  <p style="font-style: italic;">Figure 1: Percentage Distribution of Tasks</p>
</div>

<div style="text-align: center;">
  <img src="https://huggingface.co/spaces/inceptionai/AraGen-Leaderboard/raw/main/assets/pictures/blog_figure_2.png" alt="Category Distribution for Question Answering (QA)" width="500">
  <p style="font-style: italic;">Figure 2: Category Distribution for Question Answering (QA)</p>
</div>

<div style="text-align: center;">
  <img src="https://huggingface.co/spaces/inceptionai/AraGen-Leaderboard/raw/main/assets/pictures/blog_figure_3.png" alt="Category Distribution for Reasoning" width="500">
  <p style="font-style: italic;">Figure 3: Category Distribution for Reasoning</p>
</div>

 
For the "Orthographic and Grammatical Analysis" task, the data is evenly distributed between two sub-categories: "Arabic grammar" and "Arabic dictation grammar," each constituting 50% of the examples. In the "Safety" task, all the data belongs exclusively to the "Safety" category/sub-category.

 
#### Interaction Categories

The dataset examples are structured into three interaction types: 
 
1. **Single Interaction**: A simple question-answer format where the model must provide a single, complete response.  
2. **Conversational Interaction**: Multi-turn exchanges where the model must maintain conversational flow and coherence. The model is evaluated based on its response to the final question in the exchange, demonstrating its ability to engage in natural conversations. For example:  
   - **User**: "What is the capital of France?"
   - **Assistant**: "Paris."  
   - **User**: "What is the other name that it is known for as well?"  
   - **Assistant**: "Paris is often called the City of Lights as well due to its role during the Age of Enlightenment and its early adoption of street lighting."  
   Here, the model is assessed on its response to the last question while considering the flow of the exchange.  
3. **Follow-Up Interaction**: A sequence requiring continuity and factuality between two related responses. The model's second response depends on its first answer, and scoring emphasizes the importance of the initial response. For example:  
   - **User**: "What is the capital of Germany?"  
   - **Assistant**: "Berlin."  
   - **User**: "What is the population there?"  
   - **Assistant**: "The population of Berlin is about 3.7 million."  
   If the first response were incorrect (e.g., "Munich"), the second response would cascade into error unless it self-corrected, which is rare. This interaction tests the model’s ability to maintain factual continuity and build logically on its prior responses.

#### Weighting System for Follow-Up Interactions

In scoring models' performance involving follow-up interactions, the score for the first response in the conversation is weighted more heavily due to its higher potential to steer the conversation. Incorrect initial answers can lead to cascading errors.
- The **first answer** is assigned a coefficient of 2. 
- The **second answer** is assigned a coefficient of 1. 
 
For example, even if the first response is incorrect while the second response is correct (unexpected, given the design of our questions and also the way these systems usually work), the average score for the interaction would be \\( \frac{0 \times 2 + 1 \times 1}{3} = 0.333 \\), reflecting the criticality of the initial answer. 


## Judge Evaluation and Selection

Selecting the optimal judge for the **AraGen Leaderboard** is a critical step to ensure reliable, unbiased, and consistent evaluations. This section details the experiments conducted to evaluate potential judges, including single models and a jury system, and justifies the final choice based on rigorous empirical analysis. 
 
#### Judges Considered:
 
The following judge candidates were evaluated: 
 
- **GPT-4o**: a robust, proprietary model with good alignment potential; 
- **GPT-4o-mini**: a cost-efficient variant of GPT-4o with lightweight requirements;
- **Claude-3.5-sonnet**: new state-of-the-art proprietary model according to multiple benchmarks and leaderboards;
- **Claude-3-haiku**: a weaker but cost-efficient variant of Claude-3.5-sonnet;
- **Llama 3.1-405b**: a state-of-the-art open model offering full transparency and control. 

We also explored adopting a **[Jury](https://arxiv.org/abs/2404.18796)**, which aggregates evaluations from multiple LLM judges, to examine whether collective scoring improves reliability. 
 
Note that at the time we were running our experiments, Claude-3.5-haiku was not available through the Anthropic API yet. 
 
#### Evaluation Objectives
 
To evaluate and select the best judge, we assessed candidates across four dimensions: 
 
1. **Agreement with Human as a Judge**: Measuring the **Cohen's Kappa Score** to assess the agreement with human evaluations. 
2. **Scores Consistency Analysis**: How stable the Judge scores are across multiple evaluation runs. 
3. **Self Bias Analysis**: Measure the degree of self-preferential scoring exhibited by the judge. 
4. **Hallucination Analysis**: Verify if the Judges tend to hallucinate and not follow the guidelines of the evaluation. 
 
### Agreement with Human as a Judge
 
We measured the agreement of the judges' evaluations (scores) with respect to each other using **Cohen’s Kappa (κ) Coefficient**. The results are visualized in the heatmap below: 

<div style="text-align: center;">
  <img src="https://huggingface.co/spaces/inceptionai/AraGen-Leaderboard/raw/main/assets/pictures/blog_figure_4.png" alt="Cohen's Kappa Heatmap Representing the Agreement between the Judges on 3C3H Score" width="500">
  <p style="font-style: italic;">Figure 4: Cohen's Kappa Heatmap Representing the Agreement between the Judges on 3C3H Score</p>
</div>

#### Key Observations
 
- **GPT-4o-mini** achieved the highest agreement with human judge, with a κ score of **0.46**, closely followed by **Claude-3.5-sonnet**;
- **GPT-4o** demonstrated reasonable alignment, with slightly lower agreement than GPT-4o-mini and Claude-3.5-sonnet;
- **Claude-3-haiku** exhibited minimal agreement with human evaluations (kappa score: **0.06**), rendering it unsuitable as a judge. Therefore we decided to eliminate it from the remaining experiments;
- **Llama 3.1-405b** showed moderate agreement, but lagged behind proprietary models. 

### Score Consistency Analysis
 
To assess the consistency of the scores, we calculated the **standard deviation of the scores** across three evaluation runs for each judge over the same models' answers. Lower standard deviation indicates greater stability.

#### Results

| Judge               | Average Standard Deviation | 
|