task_categories:text-classification task_categories:feature-extraction language:en license:cc0-1.0 size_categories:100K<n<1M format:csv modality:text library:datasets library:pandas library:mlcroissant library:polars region:us
---
license: cc0-1.0
task_categories:
- text-classification
- feature-extraction
language:
- en
size_categories:
- 100K<n<1M
---

**Description:** This dataset comprises approximately 200,000 paragraphs and readability metrics from each of four sources: 
- HuggingFace's Fineweb-Edu
- Ronen Eldan's TinyStories
- Wikipedia-2023-11-embed-multilingual-v3 (English only)
- ArXiv Abstracts-2021.

Each paragraph falls within the character range of 50 to 2000.

**Format:** JSON, with each row representing a paragraph and containing both the text and its corresponding readability grade.

**Features:**

- Text: A paragraph of text from one of the mentioned sources

- Grade: The median of the following readability metrics:
  - Flesch-Kincaid Grade Level
  - Gunning Fog Index
  - SMOG Readability Index
  - Automated Readability Index
  - Coleman-Liau Index
  - Linsear Write Formula

The grade estimates the U.S. grade level needed for a reader to understand the text.
The higher the grade, the more difficult the text is to read.

**Data Selection Criteria:** Only paragraphs with an Interquartile Range (IQR) of readability grades less than 1 were included in the dataset.

**Example entry:**
```json
{
  "text": "We studied in detail the optical spectrum of the post-AGB star HD56126\n(IRAS07134+1005). [text omitted...]",
  "grade": 13.51
}
```

**Language:** English only

This dataset is suitable for tasks such as text simplification, readability assessment, and language understanding, allowing researchers to analyze and develop models for diverse texts and difficulties.