license:cdla-sharing-1.0 size_categories:1M<n<10M format:parquet library:datasets library:dask library:mlcroissant library:polars region:us
---
license: cdla-sharing-1.0
dataset_info:
  features:
  - name: tokens
    sequence: int64
  splits:
  - name: train
    num_bytes: 4467790964
    num_examples: 1087583
  - name: validation
    num_bytes: 45114056
    num_examples: 10982
  download_size: 794050292
  dataset_size: 4512905020
---

v0 refers to our tokenization process, v2 refers to stories generated by GPT4 only