task_categories:text2text-generation task_categories:text-classification language:en license:mit size_categories:100K<n<1M format:parquet modality:text library:datasets library:pandas library:mlcroissant library:polars region:us
---
license: mit
task_categories:
- text2text-generation
- text-classification
language:
- en
pretty_name: Wikipedia WikiNews Arxiv Text Revisions
size_categories:
- 100K<n<1M
---
This dataset is a refined version of the [original dataset](https://huggingface.co/datasets/FangornGuardian/raw_wikipedia_wikinews_arxiv_revisions), with revisions carefully filtered to ensure high-quality text.

Multiple LLMs were used to filter the revisions, and the dataset includes the aggregated labels from these models in a single column for easy reference.

The filtering process was performed using [this script](https://github.com/brotSchimmelt/llm_data_filter).


LLMs used:

- mistralai/Mistral-7B-Instruct-v0.2
- mistralai/Mistral-Nemo-Instruct-2407
- google/gemma-2-2b-it
- google/gemma-2-9b-it
