language:en license:apache-2.0 size_categories:n<1K modality:text arxiv:2412.12832 region:us Grammatical Error Correction
---
license: apache-2.0
language:
- en
tags:
- Grammatical Error Correction
---
### Dataset Card for DSGram Datasets

#### Dataset Summary
The DSGram datasets are designed for the evaluation and development of Grammatical Error Correction (GEC) models in the era of large language models (LLMs). These datasets address key evaluation challenges by incorporating human annotations and LLM-generated scores. Two subsets are provided:

1. **DSGram-LLMs**: A simulated dataset containing GPT-4-annotated sentence pairs, enabling fine-tuning and cost-effective evaluation of GEC models.
2. **DSGram-Eval**: A manually annotated dataset providing high-quality, human-scored examples to benchmark the DSGram framework.

The datasets facilitate the evaluation of corrections based on three sub-metrics:
- **Semantic Coherence**: Preservation of original meaning.
- **Edit Level**: Appropriateness of modifications.
- **Fluency**: Grammatical correctness and natural flow.

#### Dataset Structure
##### DSGram-LLMs
- **Input**: Original and corrected sentences from CoNLL-2014 and BEA-2019 test sets.
- **Annotations**: Scores generated by GPT-4 using prompt engineering techniques (Chain-of-Thought, few-shot prompting).
- **Size**: ~2,500 entries.

##### DSGram-Eval
- **Input**: Sentences from CoNLL-2014.
- **Annotations**: Human-scored sentence pairs evaluated based on the three sub-metrics.
- **Size**: ~200 entries with multiple annotators for consistency.

#### Dataset Usage
##### Intended Use
- Fine-tuning open-source LLMs for GEC evaluation.
- Benchmarking GEC models with robust and context-sensitive metrics.
- Research on evaluation frameworks for text correction tasks.

#### Citation
If you use these datasets, please cite our paper.
```
@misc{xie2024dsgramdynamicweightingsubmetrics,
      title={DSGram: Dynamic Weighting Sub-Metrics for Grammatical Error Correction in the Era of Large Language Models}, 
      author={Jinxiang Xie and Yilin Li and Xunjian Yin and Xiaojun Wan},
      year={2024},
      eprint={2412.12832},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2412.12832}, 
}
```