license:mit size_categories:10K<n<100K format:parquet modality:text library:datasets library:dask library:mlcroissant library:polars region:us
---
license: mit
dataset_info:
- config_name: 100M
  features:
  - name: text
    dtype: string
  - name: title
    dtype: string
  - name: author
    dtype: string
  - name: author year of birth
    dtype: string
  - name: author year of death
    dtype: string
  - name: language
    dtype: string
  - name: downloads
    dtype: string
  - name: subjects
    dtype: string
  - name: document id
    dtype: string
  - name: type
    dtype: string
  splits:
  - name: train
    num_bytes: 598954146
    num_examples: 1510
  - name: validation
    num_bytes: 35382485
    num_examples: 100
  download_size: 1127443305
  dataset_size: 634336631
- config_name: 10M
  features:
  - name: text
    dtype: string
  - name: title
    dtype: string
  - name: author
    dtype: string
  - name: author year of birth
    dtype: string
  - name: author year of death
    dtype: string
  - name: language
    dtype: string
  - name: downloads
    dtype: string
  - name: subjects
    dtype: string
  - name: document id
    dtype: string
  - name: type
    dtype: string
  splits:
  - name: train
    num_bytes: 58261901
    num_examples: 175
  - name: validation
    num_bytes: 31719358
    num_examples: 100
  download_size: 191136320
  dataset_size: 89981259
- config_name: all
  features:
  - name: text
    dtype: string
  - name: title
    dtype: string
  - name: author
    dtype: string
  - name: author year of birth
    dtype: string
  - name: author year of death
    dtype: string
  - name: language
    dtype: string
  - name: downloads
    dtype: string
  - name: subjects
    dtype: string
  - name: document id
    dtype: string
  - name: type
    dtype: string
  splits:
  - name: train
    num_bytes: 21653290326
    num_examples: 58223
  download_size: 13170583099
  dataset_size: 21653290326
configs:
- config_name: 100M
  data_files:
  - split: train
    path: 100M/train-*
  - split: validation
    path: 100M/validation-*
- config_name: 10M
  data_files:
  - split: train
    path: 10M/train-*
  - split: validation
    path: 10M/validation-*
- config_name: all
  data_files:
  - split: train
    path: all/train-*
---
This repository contains a snapshot of the English subset of the Project Gutenberg corpus (containing ~59K documents in total), 
downloaded in September 2024, preprocessed with code from [this repository](https://github.com/eminorhan/gutenberg), and reformatted to make it easier to use with
the `load_dataset` interface of Hugging Face's `datasets` library. 

In addition to the full dataset (under the directory `eminorhan/gutenberg_en_sep24/all`), 
I also provide two random subsets of the data containing roughly 10M and 100M words (corresponding to 275 and ~1.6K documents, respectively). 
These subsets are intended to be used for the BabyLM challenge. 

You can load these datasets as follows:
```python
from datasets import load_dataset

ds_10M = load_dataset("eminorhan/gutenberg_en_sep24", "10M")  # 10M word subset (275 documents)

ds_100M = load_dataset("eminorhan/gutenberg_en_sep24", "100M")  # 100M word subset (~1.6K documents)

ds_all = load_dataset("eminorhan/gutenberg_en_sep24", "all")  # the full data (~59K documents)
```
Both subsets come with `train`/`validation` splits, whereas the full data only has a `train` split.