task_categories:question-answering language:en license:cc-by-sa-4.0 size_categories:10K<n<100K format:json modality:text library:datasets library:pandas library:mlcroissant library:polars region:us question-answering wikipedia education
---
language:
- en
pretty_name: "Wikipedia and FineWebEdu QA Dataset"
tags:
- question-answering
- wikipedia
- education
license: "cc-by-sa-4.0"
task_categories:
- question-answering
size_categories:
- 10K<n<100K
---

# Text Questions and Answers Dataset

This dataset is a combination of multiple datasets.
The text has been cleaned and formatted into a consistent JSONL structure, containing questions and answers derived from the text.

## Dataset Summary

- **Total entries:** 41&thinsp;977
  - **From [agentlans/wikipedia-paragraph-sft](https://huggingface.co/datasets/agentlans/wikipedia-paragraph-sft):** 21&thinsp;810
  - **From [agentlans/finewebedu-sft](https://huggingface.co/datasets/agentlans/finewebedu-sft):** 10&thinsp;168
  - **From [Cosmopedia](https://huggingface.co/datasets/HuggingFaceTB/cosmopedia/viewer/web_samples_v2):** 9&thinsp;999

The Cosmopedia dataset's web_samples_v2 train split was processed using [agentlans/Qwen2.5-1.5B-Instruct-QAGenerator](https://huggingface.co/agentlans/Qwen2.5-1.5B-Instruct-QAGenerator). 
Text was tokenized with the Llama 3 tokenizer, dividing it into chunks of 400 tokens each, with a 50-token overlap between adjacent chunks.

## Data Structure

Each entry is structured as follows:

```json
{
  "text": "string",
  "qa": [
    {
      "question": "string",
      "answer": "string"
    },
    ...
  ],
  "source": "string"
}
```

- **`text`:** A passage from either English Wikipedia or FineWeb-Edu.
- **`qa`:** A list of questions and answers generated from the text. This is a JSON list encoded as a string.
- **`source`:** Indicates the original dataset (either "wikipedia-paragraph-sft" or "finewebedu-sft").

## Example Entry

```json
{
  "text": "In the fall of 1939 and the winter of 1940, Corbin was closely involved in the Anglo-French discussions about war aims...",
  "qa": [
    {
      "question": "What were the positions of the French and British regarding the restoration of Austria?",
      "answer": "The French wanted to undo the Anschluss and restore Austria, while the British accepted Austria as part of Germany."
    },
    {
      "question": "What was agreed upon regarding Czechoslovakia?",
      "answer": "Both agreed to restore Czechoslovakia to its pre-Munich frontiers."
    }
  ],
  "source": "wikipedia-paragraph-sft"
}
```

## Usage

This dataset is suitable for natural language processing tasks, particularly in question-answering and information retrieval. It offers a diverse set of texts from encyclopedic and educational sources.

## Limitations

- The quality of questions and answers may vary.
- Different writing styles may be present due to the combination of sources.

## Citation

If you use this dataset, please cite the original sources:

```
@dataset{wikipedia-paragraph-sft,
  author = {agentlans},
  title = {Wikipedia Paragraph SFT},
  year = {2024},
  publisher = {Hugging Face},
  journal = {Hugging Face Hub},
  howpublished = {\url{https://huggingface.co/datasets/agentlans/wikipedia-paragraph-sft}},
}

@dataset{finewebedu-sft,
  author = {agentlans},
  title = {FineWebEdu SFT},
  year = {2024},
  publisher = {Hugging Face},
  journal = {Hugging Face Hub},
  howpublished = {\url{https://huggingface.co/datasets/agentlans/finewebedu-sft}},
}
```

## Licensing Information

This dataset is licensed under CC-BY-SA 4.0. Users should verify licensing details for specific usage requirements related to finewebedu-sft.