task_categories:text-generation language:en license:apache-2.0 size_categories:100K<n<1M region:us data-juicer pretraining
---
license: apache-2.0
task_categories:
- text-generation
language:
- en
tags:
- data-juicer
- pretraining
size_categories:
- 100K<n<1M
---

# RedPajama -- Book (refined by Data-Juicer)

A refined version of Book dataset in RedPajama by [Data-Juicer](https://github.com/alibaba/data-juicer). Removing some "bad" samples from the original dataset to make it higher-quality.

This dataset is usually used to pretrain a Large Language Model.

**Notice**: Here is a small subset for previewing. The whole dataset is available [here](https://dail-wlcb.oss-cn-wulanchabu.aliyuncs.com/LLM_data/our_refined_datasets/pretraining/redpajama-book-refine-result.jsonl) (About 91GB).

## Dataset Information

- Number of samples: 195,983 (Keep ~95.51% from the original dataset)

## Refining Recipe
```yaml
# global parameters
project_name: 'Data-Juicer-recipes-book'
dataset_path: '/path/to/your/dataset'  # path to your dataset directory or file
export_path: '/path/to/your/dataset.jsonl'

np: 50  # number of subprocess to process your dataset
open_tracer: true

# process schedule
# a list of several process operators with their arguments
process:
  - clean_email_mapper:
  - clean_links_mapper:
  - fix_unicode_mapper:
  - punctuation_normalization_mapper:
  - whitespace_normalization_mapper:

  - alphanumeric_filter:
      tokenization: false
      min_ratio: 0.55  # <3sigma (0.697)
      max_ratio: 0.854  # 3sigma
  - average_line_length_filter:  # for code
      max_len: 500  # >3sigma (364)
  - character_repetition_filter:
      rep_len: 10
      max_ratio: 0.2  # >3sigma (0.12)
  - flagged_words_filter:
      lang: en
      tokenization: true
      max_ratio: 0.00047  # 3sigma
  - language_id_score_filter:  # remove language filter
      min_score: 0.2
  - maximum_line_length_filter:  # for code
      max_len: 13381  # 3sigma
  - perplexity_filter:
      lang: en
      max_ppl: 6000  # <3sigma (16516)
  - special_characters_filter:
      max_ratio: 0.5  # >3sigma (0.32)
  - words_num_filter:
      lang: en
      tokenization: true
      min_num: 1000
      max_num: 539754  # 3sigma
  - word_repetition_filter:
      lang: en
      tokenization: true
      rep_len: 10
      max_ratio: 0.194  # 3sigma

  - document_simhash_deduplicator:
      tokenization: space
      window_size: 6
      lowercase: true
      ignore_pattern: '\p{P}'
      num_blocks: 6
      hamming_distance: 4
```